panic due to runaway promises

[SreeniIO]

In a complex use case where there are nested promise calls, the runtime is crashing with the following panics. Do you think these can be handled without panicking? I'm trying to create a simple, reproducible code sample, but unable to get it to panic.

thread panic occurred: panicked at 'no such elem', 
  .cargo/registry/src/github.com-1ecc6299db9ec823/
  hirofa_utils-0.5.7/src/auto_id_map.rs:132:29

thread panic occurred: panicked at 'send failed: SendError { .. }', 
  .cargo/registry/src/github.com-1ecc6299db9ec823/
  hirofa_utils-0.5.7/src/eventloop.rs:297:3

[SreeniIO]

Full trace

2023-03-24T01:09:21.470046Z ERROR ThreadId(15) *****: 117: thread panic occurred: panicked at 'no such elem', /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/auto_id_map.rs:132:29
backtrace:    0: backtrace::backtrace::libunwind::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/libunwind.rs:93:5
      backtrace::backtrace::trace_unsynchronized
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:53:14
   2: backtrace::capture::Backtrace::create
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:176:9
   3: backtrace::capture::Backtrace::new
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:140:22
   4: *****::main::{{closure}}::{{closure}}
             at io-server/src/main.rs:115:25
   5: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:692:13
   6: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:579:13
   7: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:137:18
   8: rust_begin_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:575:5
   9: core::panicking::panic_fmt
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panicking.rs:64:14
  10: core::panicking::panic_display
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panicking.rs:147:5
  11: core::panicking::panic_str
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panicking.rs:131:5
  12: core::option::expect_failed
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/option.rs:1924:5
  13: core::option::Option<T>::expect
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/option.rs:786:21
  14: hirofa_utils::auto_id_map::AutoIdMap<T>::remove
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/auto_id_map.rs:132:9
  15: <quickjs_runtime::quickjsrealmadapter::QuickJsRealmAdapter as hirofa_utils::js_utils::adapters::JsRealmAdapter>::js_promise_cache_consume
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/quickjsrealmadapter.rs:815:9
  16: hirofa_utils::js_utils::adapters::promises::new_resolving_promise_async::{{closure}}::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/js_utils/adapters/promises.rs:124:342
  17: quickjs_runtime::quickjsruntimeadapter::QuickJsRuntimeAdapter::do_with::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/quickjsruntimeadapter.rs:605:13
  18: std::thread::local::LocalKey<T>::try_with
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/local.rs:446:16
  19: std::thread::local::LocalKey<T>::with
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/local.rs:422:9
  20: quickjs_runtime::quickjsruntimeadapter::QuickJsRuntimeAdapter::do_with
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/quickjsruntimeadapter.rs:597:19
  21: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_rt_task_to_event_loop_void::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:138:45
  22: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_task_to_event_loop_void::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:74:13
  23: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
  24: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
  25: hirofa_utils::eventloop::EventLoop::new::{{closure}}::{{closure}}::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:92:52
  26: <futures_task::future_obj::LocalFutureObj<T> as core::future::future::Future>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/futures-task-0.3.27/src/future_obj.rs:84:18
  27: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.27/src/stream/futures_unordered/mod.rs:518:17
  28: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.27/src/stream/stream/mod.rs:1632:9
  29: futures_executor::local_pool::LocalPool::poll_pool
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:279:28
  30: futures_executor::local_pool::LocalPool::run_until_stalled::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:254:33
  31: futures_executor::local_pool::run_executor::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:90:37
  32: std::thread::local::LocalKey<T>::try_with
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/local.rs:446:16
  33: std::thread::local::LocalKey<T>::with
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/local.rs:422:9
  34: futures_executor::local_pool::run_executor
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:86:5
  35: futures_executor::local_pool::LocalPool::run_until_stalled
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:254:9
  36: hirofa_utils::eventloop::EventLoop::new::{{closure}}::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:95:21
  37: std::thread::local::LocalKey<T>::try_with
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/local.rs:446:16
  38: std::thread::local::LocalKey<T>::with
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/local.rs:422:9
  39: hirofa_utils::eventloop::EventLoop::new::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:72:13
  40: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:121:18
  41: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:558:17
  42: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  43: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  44: ___rust_try
  45: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  46: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  47: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:557:30
  48: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
  49: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys/unix/thread.rs:108:17
  50: __pthread_start

2023-03-24T01:09:22.596345Z ERROR ThreadId(45) *****: 117: thread panic occurred: panicked at 'send failed: SendError { .. }', /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:297:37
backtrace:    0: backtrace::backtrace::libunwind::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/libunwind.rs:93:5
      backtrace::backtrace::trace_unsynchronized
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:53:14
   2: backtrace::capture::Backtrace::create
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:176:9
   3: backtrace::capture::Backtrace::new
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:140:22
   4: *****::main::{{closure}}::{{closure}}
             at io-server/src/main.rs:115:25
   5: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:692:13
   6: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:579:13
   7: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:137:18
   8: rust_begin_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:575:5
   9: core::panicking::panic_fmt
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panicking.rs:64:14
  10: core::result::unwrap_failed
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1790:5
  11: core::result::Result<T,E>::expect
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1069:23
  12: hirofa_utils::eventloop::EventLoop::add_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:297:13
  13: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:73:9
  14: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_rt_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:138:9
  15: <quickjs_runtime::facades::QuickjsRuntimeFacadeInner as hirofa_utils::js_utils::facades::JsRuntimeFacadeInner>::js_add_rt_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:62:9
  16: hirofa_utils::js_utils::adapters::promises::new_resolving_promise_async::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/js_utils/adapters/promises.rs:120:13
  17: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:223:17
  18: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/loom/std/unsafe_cell.rs:14:9
  19: tokio::runtime::task::core::Core<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:212:13
  20: tokio::runtime::task::harness::poll_future::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:476:19
  21: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  22: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  23: ___rust_try
  24: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  25: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  26: tokio::runtime::task::harness::poll_future
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:464:18
  27: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:198:27
  28: tokio::runtime::task::harness::Harness<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:152:15
  29: tokio::runtime::task::raw::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:255:5
  30: tokio::runtime::task::raw::RawTask::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:200:18
  31: tokio::runtime::task::LocalNotified<S>::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/mod.rs:394:9
  32: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:487:21
  33: tokio::runtime::coop::with_budget
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/coop.rs:107:5
      tokio::runtime::coop::budget
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/coop.rs:73:5
      tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  34: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:426:24
  35: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  36: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/macros/scoped_tls.rs:61:9
  37: tokio::runtime::scheduler::multi_thread::worker::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  38: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  39: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/task.rs:42:21
  40: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:223:17
  41: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/loom/std/unsafe_cell.rs:14:9
  42: tokio::runtime::task::core::Core<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:212:13
  43: tokio::runtime::task::harness::poll_future::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:476:19
  44: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  45: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  46: ___rust_try
  47: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  48: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  49: tokio::runtime::task::harness::poll_future
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:464:18
  50: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:198:27
  51: tokio::runtime::task::harness::Harness<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:152:15
  52: tokio::runtime::task::raw::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:255:5
  53: tokio::runtime::task::raw::RawTask::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:200:18
  54: tokio::runtime::task::UnownedTask<S>::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/mod.rs:431:9
  55: tokio::runtime::blocking::pool::Task::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:159:9
  56: tokio::runtime::blocking::pool::Inner::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:511:17
  57: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:469:13
  58: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:121:18
  59: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:558:17
  60: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  61: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  62: ___rust_try
  63: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  64: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  65: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:557:30
  66: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
  67: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys/unix/thread.rs:108:17
  68: __pthread_start

2023-03-24T01:09:22.756458Z ERROR ThreadId(45) *****: 117: thread panic occurred: panicked at 'send failed: SendError { .. }', /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:297:37
backtrace:    0: backtrace::backtrace::libunwind::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/libunwind.rs:93:5
      backtrace::backtrace::trace_unsynchronized
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:53:14
   2: backtrace::capture::Backtrace::create
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:176:9
   3: backtrace::capture::Backtrace::new
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:140:22
   4: *****::main::{{closure}}::{{closure}}
             at io-server/src/main.rs:115:25
   5: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:692:13
   6: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:579:13
   7: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:137:18
   8: rust_begin_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:575:5
   9: core::panicking::panic_fmt
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panicking.rs:64:14
  10: core::result::unwrap_failed
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1790:5
  11: core::result::Result<T,E>::expect
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1069:23
  12: hirofa_utils::eventloop::EventLoop::add_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:297:13
  13: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:73:9
  14: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_rt_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:138:9
  15: <quickjs_runtime::facades::QuickjsRuntimeFacadeInner as hirofa_utils::js_utils::facades::JsRuntimeFacadeInner>::js_add_rt_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:62:9
  16: hirofa_utils::js_utils::adapters::promises::new_resolving_promise_async::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/js_utils/adapters/promises.rs:120:13
  17: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:223:17
  18: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/loom/std/unsafe_cell.rs:14:9
  19: tokio::runtime::task::core::Core<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:212:13
  20: tokio::runtime::task::harness::poll_future::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:476:19
  21: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  22: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  23: ___rust_try
  24: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  25: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  26: tokio::runtime::task::harness::poll_future
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:464:18
  27: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:198:27
  28: tokio::runtime::task::harness::Harness<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:152:15
  29: tokio::runtime::task::raw::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:255:5
  30: tokio::runtime::task::raw::RawTask::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:200:18
  31: tokio::runtime::task::LocalNotified<S>::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/mod.rs:394:9
  32: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:487:21
  33: tokio::runtime::coop::with_budget
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/coop.rs:107:5
      tokio::runtime::coop::budget
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/coop.rs:73:5
      tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  34: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:426:24
  35: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  36: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/macros/scoped_tls.rs:61:9
  37: tokio::runtime::scheduler::multi_thread::worker::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  38: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  39: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/task.rs:42:21
  40: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:223:17
  41: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/loom/std/unsafe_cell.rs:14:9
  42: tokio::runtime::task::core::Core<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:212:13
  43: tokio::runtime::task::harness::poll_future::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:476:19
  44: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  45: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  46: ___rust_try
  47: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  48: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  49: tokio::runtime::task::harness::poll_future
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:464:18
  50: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:198:27
  51: tokio::runtime::task::harness::Harness<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:152:15
  52: tokio::runtime::task::raw::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:255:5
  53: tokio::runtime::task::raw::RawTask::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:200:18
  54: tokio::runtime::task::UnownedTask<S>::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/mod.rs:431:9
  55: tokio::runtime::blocking::pool::Task::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:159:9
  56: tokio::runtime::blocking::pool::Inner::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:511:17
  57: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:469:13
  58: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:121:18
  59: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:558:17
  60: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  61: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  62: ___rust_try
  63: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  64: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  65: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:557:30
  66: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
  67: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys/unix/thread.rs:108:17
  68: __pthread_start

2023-03-24T01:09:22.838077Z ERROR ThreadId(45) *****: 117: thread panic occurred: panicked at 'send failed: SendError { .. }', /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:297:37
backtrace:    0: backtrace::backtrace::libunwind::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/libunwind.rs:93:5
      backtrace::backtrace::trace_unsynchronized
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:53:14
   2: backtrace::capture::Backtrace::create
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:176:9
   3: backtrace::capture::Backtrace::new
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:140:22
   4: *****::main::{{closure}}::{{closure}}
             at io-server/src/main.rs:115:25
   5: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:692:13
   6: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:579:13
   7: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:137:18
   8: rust_begin_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:575:5
   9: core::panicking::panic_fmt
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panicking.rs:64:14
  10: core::result::unwrap_failed
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1790:5
  11: core::result::Result<T,E>::expect
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1069:23
  12: hirofa_utils::eventloop::EventLoop::add_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:297:13
  13: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:73:9
  14: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_rt_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:138:9
  15: <quickjs_runtime::facades::QuickjsRuntimeFacadeInner as hirofa_utils::js_utils::facades::JsRuntimeFacadeInner>::js_add_rt_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:62:9
  16: hirofa_utils::js_utils::adapters::promises::new_resolving_promise_async::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/js_utils/adapters/promises.rs:120:13
  17: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:223:17
  18: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/loom/std/unsafe_cell.rs:14:9
  19: tokio::runtime::task::core::Core<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:212:13
  20: tokio::runtime::task::harness::poll_future::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:476:19
  21: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  22: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  23: ___rust_try
  24: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  25: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  26: tokio::runtime::task::harness::poll_future
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:464:18
  27: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:198:27
  28: tokio::runtime::task::harness::Harness<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:152:15
  29: tokio::runtime::task::raw::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:255:5
  30: tokio::runtime::task::raw::RawTask::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:200:18
  31: tokio::runtime::task::LocalNotified<S>::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/mod.rs:394:9
  32: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:487:21
  33: tokio::runtime::coop::with_budget
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/coop.rs:107:5
      tokio::runtime::coop::budget
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/coop.rs:73:5
      tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  34: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:426:24
  35: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  36: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/macros/scoped_tls.rs:61:9
  37: tokio::runtime::scheduler::multi_thread::worker::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  38: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  39: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/task.rs:42:21
  40: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:223:17
  41: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/loom/std/unsafe_cell.rs:14:9
  42: tokio::runtime::task::core::Core<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:212:13
  43: tokio::runtime::task::harness::poll_future::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:476:19
  44: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  45: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  46: ___rust_try
  47: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  48: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  49: tokio::runtime::task::harness::poll_future
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:464:18
  50: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:198:27
  51: tokio::runtime::task::harness::Harness<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:152:15
  52: tokio::runtime::task::raw::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:255:5
  53: tokio::runtime::task::raw::RawTask::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:200:18
  54: tokio::runtime::task::UnownedTask<S>::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/mod.rs:431:9
  55: tokio::runtime::blocking::pool::Task::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:159:9
  56: tokio::runtime::blocking::pool::Inner::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:511:17
  57: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:469:13
  58: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:121:18
  59: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:558:17
  60: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  61: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  62: ___rust_try
  63: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  64: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  65: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:557:30
  66: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
  67: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys/unix/thread.rs:108:17
  68: __pthread_start

2023-03-24T01:09:22.853084Z ERROR ThreadId(45) *****: 117: thread panic occurred: panicked at 'send failed: SendError { .. }', /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:297:37
backtrace:    0: backtrace::backtrace::libunwind::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/libunwind.rs:93:5
      backtrace::backtrace::trace_unsynchronized
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:53:14
   2: backtrace::capture::Backtrace::create
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:176:9
   3: backtrace::capture::Backtrace::new
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:140:22
   4: *****::main::{{closure}}::{{closure}}
             at io-server/src/main.rs:115:25
   5: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:692:13
   6: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:579:13
   7: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:137:18
   8: rust_begin_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:575:5
   9: core::panicking::panic_fmt
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panicking.rs:64:14
  10: core::result::unwrap_failed
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1790:5
  11: core::result::Result<T,E>::expect
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1069:23
  12: hirofa_utils::eventloop::EventLoop::add_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:297:13
  13: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:73:9
  14: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_rt_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:138:9
  15: <quickjs_runtime::facades::QuickjsRuntimeFacadeInner as hirofa_utils::js_utils::facades::JsRuntimeFacadeInner>::js_add_rt_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:62:9
  16: hirofa_utils::js_utils::adapters::promises::new_resolving_promise_async::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/js_utils/adapters/promises.rs:120:13
  17: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:223:17
  18: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/loom/std/unsafe_cell.rs:14:9
  19: tokio::runtime::task::core::Core<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:212:13
  20: tokio::runtime::task::harness::poll_future::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:476:19
  21: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  22: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  23: ___rust_try
  24: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  25: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  26: tokio::runtime::task::harness::poll_future
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:464:18
  27: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:198:27
  28: tokio::runtime::task::harness::Harness<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:152:15
  29: tokio::runtime::task::raw::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:255:5
  30: tokio::runtime::task::raw::RawTask::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:200:18
  31: tokio::runtime::task::LocalNotified<S>::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/mod.rs:394:9
  32: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:487:21
  33: tokio::runtime::coop::with_budget
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/coop.rs:107:5
      tokio::runtime::coop::budget
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/coop.rs:73:5
      tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  34: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:426:24
  35: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  36: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/macros/scoped_tls.rs:61:9
  37: tokio::runtime::scheduler::multi_thread::worker::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  38: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  39: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/task.rs:42:21
  40: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:223:17
  41: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/loom/std/unsafe_cell.rs:14:9
  42: tokio::runtime::task::core::Core<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:212:13
  43: tokio::runtime::task::harness::poll_future::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:476:19
  44: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  45: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  46: ___rust_try
  47: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  48: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  49: tokio::runtime::task::harness::poll_future
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:464:18
  50: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:198:27
  51: tokio::runtime::task::harness::Harness<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:152:15
  52: tokio::runtime::task::raw::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:255:5
  53: tokio::runtime::task::raw::RawTask::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:200:18
  54: tokio::runtime::task::UnownedTask<S>::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/mod.rs:431:9
  55: tokio::runtime::blocking::pool::Task::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:159:9
  56: tokio::runtime::blocking::pool::Inner::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:511:17
  57: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:469:13
  58: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:121:18
  59: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:558:17
  60: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  61: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  62: ___rust_try
  63: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  64: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  65: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:557:30
  66: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
  67: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys/unix/thread.rs:108:17
  68: __pthread_start

2023-03-24T01:09:22.868194Z ERROR ThreadId(45) *****: 117: thread panic occurred: panicked at 'send failed: SendError { .. }', /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:297:37
backtrace:    0: backtrace::backtrace::libunwind::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/libunwind.rs:93:5
      backtrace::backtrace::trace_unsynchronized
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:53:14
   2: backtrace::capture::Backtrace::create
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:176:9
   3: backtrace::capture::Backtrace::new
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:140:22
   4: *****::main::{{closure}}::{{closure}}
             at io-server/src/main.rs:115:25
   5: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:692:13
   6: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:579:13
   7: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:137:18
   8: rust_begin_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:575:5
   9: core::panicking::panic_fmt
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panicking.rs:64:14
  10: core::result::unwrap_failed
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1790:5
  11: core::result::Result<T,E>::expect
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/result.rs:1069:23
  12: hirofa_utils::eventloop::EventLoop::add_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/eventloop.rs:297:13
  13: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:73:9
  14: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_rt_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:138:9
  15: <quickjs_runtime::facades::QuickjsRuntimeFacadeInner as hirofa_utils::js_utils::facades::JsRuntimeFacadeInner>::js_add_rt_task_to_event_loop_void
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.8.7/src/facades.rs:62:9
  16: hirofa_utils::js_utils::adapters::promises::new_resolving_promise_async::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.5.7/src/js_utils/adapters/promises.rs:120:13
  17: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:223:17
  18: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/loom/std/unsafe_cell.rs:14:9
  19: tokio::runtime::task::core::Core<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:212:13
  20: tokio::runtime::task::harness::poll_future::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:476:19
  21: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  22: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  23: ___rust_try
  24: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  25: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  26: tokio::runtime::task::harness::poll_future
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:464:18
  27: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:198:27
  28: tokio::runtime::task::harness::Harness<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:152:15
  29: tokio::runtime::task::raw::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:255:5
  30: tokio::runtime::task::raw::RawTask::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:200:18
  31: tokio::runtime::task::LocalNotified<S>::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/mod.rs:394:9
  32: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:487:21
  33: tokio::runtime::coop::with_budget
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/coop.rs:107:5
      tokio::runtime::coop::budget
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/coop.rs:73:5
      tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:463:9
  34: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:426:24
  35: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:406:17
  36: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/macros/scoped_tls.rs:61:9
  37: tokio::runtime::scheduler::multi_thread::worker::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:403:5
  38: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/scheduler/multi_thread/worker.rs:365:45
  39: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/task.rs:42:21
  40: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:223:17
  41: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/loom/std/unsafe_cell.rs:14:9
  42: tokio::runtime::task::core::Core<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/core.rs:212:13
  43: tokio::runtime::task::harness::poll_future::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:476:19
  44: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  45: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  46: ___rust_try
  47: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  48: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  49: tokio::runtime::task::harness::poll_future
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:464:18
  50: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:198:27
  51: tokio::runtime::task::harness::Harness<T,S>::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/harness.rs:152:15
  52: tokio::runtime::task::raw::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:255:5
  53: tokio::runtime::task::raw::RawTask::poll
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/raw.rs:200:18
  54: tokio::runtime::task::UnownedTask<S>::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/task/mod.rs:431:9
  55: tokio::runtime::blocking::pool::Task::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:159:9
  56: tokio::runtime::blocking::pool::Inner::run
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:511:17
  57: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.26.0/src/runtime/blocking/pool.rs:469:13
  58: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys_common/backtrace.rs:121:18
  59: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:558:17
  60: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/panic/unwind_safe.rs:271:9
  61: std::panicking::try::do_call
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:483:40
  62: ___rust_try
  63: std::panicking::try
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panicking.rs:447:19
  64: std::panic::catch_unwind
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/panic.rs:140:14
  65: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/thread/mod.rs:557:30
  66: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/core/src/ops/function.rs:250:5
  67: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/2c8cc343237b8f7d5a3c3703e3a87f2eb2c54a74/library/std/src/sys/unix/thread.rs:108:17
  68: __pthread_start

[andrieshiemstra]

Hmm.. first of all lets assume the first error (no such elem) causes the second (eventloop borked due to earlier panic)

So i'll focus on the 'no such elem' error..

you're using new_resolving_promise_async which adds a ref to a map (autoidmap) in the realm... then after going async the realm still exists (by id) but it's map (promise_cache) does not have the id anymore...

so either

• you're calling 'js_promise_cache_consume' in your code somewhere (which I'll guess for now you're not)
  or
• the realm instance is a different instance... e.g.
  • the promise is created
  • the async producer runs
  • the realm is destroyed
  • a new realm is instantiated with the same realm id
  • the async producer gets item from map which is a different map.....

could it be that your code causes the second case? meaning the realm could be destroyed while not all promises are resolved yet? e.g. async log calls being discarded

[SreeniIO]

I'm not calling js_promise_cache_consume anywhere.
Yes, it's possible that a new realm gets created with the same id (in my case, session-id)

[andrieshiemstra]

ok, it's an edge case which i need to fix, and just log but even then there are apparently promises which you have not awaited and which will not resolve...

[andrieshiemstra]

ok, I changed the AutoIdMap to have random ID's this means that the chance of your dangling Promise resolving the wrong newer promises is 1 in (usize::MAX / the number of Promises in the new realm)..
so small chance..

when a dangling promise is resolve it will log an error,

DISCLAIMER
you SHOULD take these errors seriously and update your code to prevent dangling promises after a realm is dropped..

released as utils:0.6 and qjs_runtime 0.9 due to changes in the JSRealm trait

[SreeniIO]

Thanks a lot for the quick turnaround. These errors happen in development due to missing await that will eventually be fixed.

[SreeniIO]

This happens when using Promise.all([...multiplePromises]), and if one of them throws an error leaving all the other promises to continue running until complete or rejected. In our case, the context is removed as soon as the first promise throws an error.

thread panic occurred: panicked at 'no such elem', /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.6.1/src/auto_id_map.rs:133:29
backtrace:    0: backtrace::backtrace::libunwind::trace
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/libunwind.rs:93:5
      backtrace::backtrace::trace_unsynchronized
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:53:14
   2: backtrace::capture::Backtrace::create
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:176:9
   3: backtrace::capture::Backtrace::new
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:140:22
   4: cloudio::main::{{closure}}::{{closure}}
             at io-server/src/main.rs:115:25
   5: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:692:13
   6: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:579:13
   7: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:137:18
   8: rust_begin_unwind
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
   9: core::panicking::panic_fmt
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
  10: core::panicking::panic_display
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:147:5
  11: core::panicking::panic_str
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:131:5
  12: core::option::expect_failed
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/option.rs:1924:5
  13: core::option::Option<T>::expect
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/option.rs:786:21
  14: hirofa_utils::auto_id_map::AutoIdMap<T>::remove
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.6.1/src/auto_id_map.rs:133:9
  15: quickjs_runtime::quickjsrealmadapter::QuickJsRealmAdapter::consume_cached_obj
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.9.0/src/quickjsrealmadapter.rs:297:9
  16: <quickjs_runtime::quickjsrealmadapter::QuickJsRealmAdapter as hirofa_utils::js_utils::adapters::JsRealmAdapter>::js_cache_dispose
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.9.0/src/quickjsrealmadapter.rs:823:17
  17: hirofa_utils::js_utils::facades::values::CachedJsObjectRef::new::{{closure}}::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.6.1/src/js_utils/facades/values.rs:45:25
  18: quickjs_runtime::quickjsruntimeadapter::QuickJsRuntimeAdapter::do_with::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.9.0/src/quickjsruntimeadapter.rs:604:13
  19: std::thread::local::LocalKey<T>::try_with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:446:16
  20: std::thread::local::LocalKey<T>::with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:422:9
  21: quickjs_runtime::quickjsruntimeadapter::QuickJsRuntimeAdapter::do_with
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.9.0/src/quickjsruntimeadapter.rs:596:19
  22: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_rt_task_to_event_loop_void::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.9.0/src/facades.rs:138:45
  23: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_task_to_event_loop_void::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/quickjs_runtime-0.9.0/src/facades.rs:74:13
  24: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/ops/function.rs:250:5
  25: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
  26: hirofa_utils::eventloop::EventLoop::new::{{closure}}::{{closure}}::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.6.1/src/eventloop.rs:92:52
  27: <futures_task::future_obj::LocalFutureObj<T> as core::future::future::Future>::poll
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-task-0.3.27/src/future_obj.rs:84:18
  28: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.27/src/stream/futures_unordered/mod.rs:518:17
  29: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.27/src/stream/stream/mod.rs:1632:9
  30: futures_executor::local_pool::LocalPool::poll_pool
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:279:28
  31: futures_executor::local_pool::LocalPool::run_until_stalled::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:254:33
  32: futures_executor::local_pool::run_executor::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:90:37
  33: std::thread::local::LocalKey<T>::try_with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:446:16
  34: std::thread::local::LocalKey<T>::with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:422:9
  35: futures_executor::local_pool::run_executor
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:86:5
  36: futures_executor::local_pool::LocalPool::run_until_stalled
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:254:9
  37: hirofa_utils::eventloop::EventLoop::new::{{closure}}::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.6.1/src/eventloop.rs:95:21
  38: std::thread::local::LocalKey<T>::try_with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:446:16
  39: std::thread::local::LocalKey<T>::with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:422:9
  40: hirofa_utils::eventloop::EventLoop::new::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/hirofa_utils-0.6.1/src/eventloop.rs:72:13
  41: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:121:18
  42: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/mod.rs:558:17
  43: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panic/unwind_safe.rs:271:9
  44: std::panicking::try::do_call
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:483:40
  45: ___rust_try
  46: std::panicking::try
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:447:19
  47: std::panic::catch_unwind
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panic.rs:140:14
  48: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/mod.rs:557:30
  49: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/ops/function.rs:250:5
  50: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys/unix/thread.rs:108:17
  51: __pthread_start

[SreeniIO]

I tried to fix this on my fork, but not sure if it's ok to change the return param of js_cache_consume from JSValueRef to Option; hence didn't create a PR.

[SreeniIO]

One other place

thread panic occurred: panicked at 'no such obj in cache', /dir/.cargo/git/checkouts/quickjs_es_runtime-1026d506f58488f3/d0eb3ca/src/quickjsrealmadapter.rs:308:34

[SreeniIO]

Full trace

thread panic occurred: panicked at 'no such obj in cache', /dir/.cargo/git/checkouts/quickjs_es_runtime-1026d506f58488f3/d0eb3ca/src/quickjsrealmadapter.rs:308:34
backtrace:    0: backtrace::backtrace::libunwind::trace
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/libunwind.rs:93:5
      backtrace::backtrace::trace_unsynchronized
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:53:14
   2: backtrace::capture::Backtrace::create
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:176:9
   3: backtrace::capture::Backtrace::new
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/capture.rs:140:22
   4: cloudio::main::{{closure}}::{{closure}}
             at io-server/src/main.rs:115:25
   5: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:2002:9
      std::panicking::rust_panic_with_hook
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:692:13
   6: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:579:13
   7: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:137:18
   8: rust_begin_unwind
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
   9: core::panicking::panic_fmt
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
  10: core::panicking::panic_display
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:147:5
  11: core::panicking::panic_str
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:131:5
  12: core::option::expect_failed
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/option.rs:1924:5
  13: core::option::Option<T>::expect
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/option.rs:786:21
  14: quickjs_runtime::quickjsrealmadapter::QuickJsRealmAdapter::with_cached_obj
             at /dir/.cargo/git/checkouts/quickjs_es_runtime-1026d506f58488f3/d0eb3ca/src/quickjsrealmadapter.rs:308:30
  15: <quickjs_runtime::quickjsrealmadapter::QuickJsRealmAdapter as hirofa_utils::js_utils::adapters::JsRealmAdapter>::js_cache_with
             at /dir/.cargo/git/checkouts/quickjs_es_runtime-1026d506f58488f3/d0eb3ca/src/quickjsrealmadapter.rs:830:9
  16: hirofa_utils::js_utils::facades::values::CachedJsObjectRef::js_get_object::{{closure}}::{{closure}}
             at /dir/.cargo/git/checkouts/utils-3679415642976e46/e0fa902/src/js_utils/facades/values.rs:84:31
  17: quickjs_runtime::quickjsruntimeadapter::QuickJsRuntimeAdapter::do_with::{{closure}}
             at /dir/.cargo/git/checkouts/quickjs_es_runtime-1026d506f58488f3/d0eb3ca/src/quickjsruntimeadapter.rs:604:13
  18: std::thread::local::LocalKey<T>::try_with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:446:16
  19: std::thread::local::LocalKey<T>::with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:422:9
  20: quickjs_runtime::quickjsruntimeadapter::QuickJsRuntimeAdapter::do_with
             at /dir/.cargo/git/checkouts/quickjs_es_runtime-1026d506f58488f3/d0eb3ca/src/quickjsruntimeadapter.rs:596:19
  21: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_rt_task_to_event_loop::{{closure}}
             at /dir/.cargo/git/checkouts/quickjs_es_runtime-1026d506f58488f3/d0eb3ca/src/facades.rs:131:40
  22: quickjs_runtime::facades::QuickjsRuntimeFacadeInner::add_task_to_event_loop::{{closure}}
             at /dir/.cargo/git/checkouts/quickjs_es_runtime-1026d506f58488f3/d0eb3ca/src/facades.rs:103:23
  23: hirofa_utils::eventloop::EventLoop::add::{{closure}}
             at /dir/.cargo/git/checkouts/utils-3679415642976e46/e0fa902/src/eventloop.rs:231:38
  24: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::future::future::Future>::poll
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panic/unwind_safe.rs:296:9
  25: <futures_util::future::future::catch_unwind::CatchUnwind<Fut> as core::future::future::Future>::poll::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.27/src/future/future/catch_unwind.rs:36:42
  26: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panic/unwind_safe.rs:271:9
  27: std::panicking::try::do_call
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:483:40
  28: ___rust_try
  29: std::panicking::try
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:447:19
  30: std::panic::catch_unwind
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panic.rs:140:14
  31: <futures_util::future::future::catch_unwind::CatchUnwind<Fut> as core::future::future::Future>::poll
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.27/src/future/future/catch_unwind.rs:36:9
  32: <futures_util::future::future::remote_handle::Remote<Fut> as core::future::future::Future>::poll
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.27/src/future/future/remote_handle.rs:105:29
  33: <futures_task::future_obj::LocalFutureObj<T> as core::future::future::Future>::poll
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-task-0.3.27/src/future_obj.rs:84:18
  34: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.27/src/stream/futures_unordered/mod.rs:518:17
  35: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.27/src/stream/stream/mod.rs:1632:9
  36: futures_executor::local_pool::LocalPool::poll_pool
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:279:28
  37: futures_executor::local_pool::LocalPool::run_until_stalled::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:254:33
  38: futures_executor::local_pool::run_executor::{{closure}}
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:90:37
  39: std::thread::local::LocalKey<T>::try_with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:446:16
  40: std::thread::local::LocalKey<T>::with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:422:9
  41: futures_executor::local_pool::run_executor
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:86:5
  42: futures_executor::local_pool::LocalPool::run_until_stalled
             at /dir/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.27/src/local_pool.rs:254:9
  43: hirofa_utils::eventloop::EventLoop::new::{{closure}}::{{closure}}
             at /dir/.cargo/git/checkouts/utils-3679415642976e46/e0fa902/src/eventloop.rs:95:21
  44: std::thread::local::LocalKey<T>::try_with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:446:16
  45: std::thread::local::LocalKey<T>::with
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/local.rs:422:9
  46: hirofa_utils::eventloop::EventLoop::new::{{closure}}
             at /dir/.cargo/git/checkouts/utils-3679415642976e46/e0fa902/src/eventloop.rs:72:13
  47: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys_common/backtrace.rs:121:18
  48: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/mod.rs:558:17
  49: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panic/unwind_safe.rs:271:9
  50: std::panicking::try::do_call
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:483:40
  51: ___rust_try
  52: std::panicking::try
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:447:19
  53: std::panic::catch_unwind
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panic.rs:140:14
  54: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/thread/mod.rs:557:30
  55: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/ops/function.rs:250:5
  56: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/alloc/src/boxed.rs:1988:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/sys/unix/thread.rs:108:17
  57: __pthread_start

[andrieshiemstra]

Hi,

Reverting to _opt and logging error is fine for now.. someday that's going to resolve/reject a completely different promise in your new context..

the solution is probably something like storing Weakrefs to the Realm instead of an id string.. please leave this tocke topen for that

[SreeniIO]

Oh. That would be even more dangerous. Will this be resolved if I don't reuse the context id? I was trying to avoid creation of too many contexts for a given session request.

[andrieshiemstra]

yeah if you use a different context id every time it will be safe

since resolving this definitively needs some refactoring in utils and in quickjs_runtime i'd like to finish a large refactor i'm working on first...