QUIC DATAGRAM bursts after ~30 connections on one Endpoint

[FedorKiselev76]

Hi,

We are using iroh QUIC DATAGRAM for realtime audio.

With one server-side Endpoint, everything is stable up to about 30 concurrent connections. Above that, conn.read_datagram() starts returning packets in bursts. We see 100–800 ms gaps between read_datagram() returns.

The workload is small DATAGRAM packets every 20 ms per connection. The selected path is usually direct IP, not relay. Server CPU is not saturated.

We checked our application pipeline after read_datagram():

• application queues are not full
• packet processing is not slow
• no local backlog

So the burst seems to happen before the datagram reaches our application code.

Is this expected for one Endpoint with many realtime DATAGRAM connections? Would you recommend increasing runtime worker threads, tuning datagram buffers, or sharding traffic across multiple Endpoints/UDP ports?

Thanks

[matheus23]

Would you recommend increasing runtime worker threads

I've seen this make a difference with burstiness in the past. So yes, increasing tokio worker threads is likely to help IME.

[flub]

I wonder if there's any lock-contention starting to be at play.

[FedorKiselev76]

I wonder if there's any lock-contention starting to be at play.

This is only a suspicion, but could the burstiness be caused by contention/coalescing in the noq DATAGRAM receive path? ReadDatagram::poll locks the per-connection State and reads from inner.datagrams().recv(). The ConnectionDriver also locks the same State and wakes read_datagram via shared.datagram_received Notify from forward_app_events(). With many realtime DATAGRAM-heavy connections, could this cause read_datagram wakeups to be delayed/coalesced and then delivered in bursts? More broadly, we are interested in using iroh for high-load media-switch/media-gateway workloads.

[flub]

FYI, we're unlikely to get to this soon as we're busy with plenty of other things. If you need this addressed within some timelines please reach out.