Gradual CPU drift over time on long-running session with high outbound connection failure rate (ARM64, Docker)

[zonzille]

NOTE : I did most of my debugging process using Claude. I checked the results myself every time I could, but some results like the output of the strace runs were out of my skill range. If the results are inconsistent with the conclusions I came to, please let me know. If that's not allowed, then I'm very sorry. I lack the technical ability of debugging this deeply without just hitting around blindly with my current skill level. Hope I'm not making a fuss over a false positive.

Environment

• libtorrent version: 1.2.20.0 (as shipped with qBittorrent 5.2.2, hotio.dev image)
• qBittorrent version: 5.2.2
• Platform: ARM64, Raspberry Pi 4 (4GB), Raspberry Pi OS Lite (Trixie)
• Deployment: Docker container, network interface bound to WireGuard VPN (wg0), with PIA as the VPN provider
• Torrent count: ~120 active seeding torrents, mostly private trackers (but not exclusively)
• Resume data storage: fastresume files

---

Description

Hello everyone. On long-running sessions, I noticed qBittorrent-nox exhibits gradual CPU drift on ARM64. After a clean container restart, CPU usage sits at ~3%. Over approximately 40 hours it climbs to ~30%, and extrapolating from previous observations, would reach ~80%+ within a week without intervention. A container restart always brings CPU back to baseline immediately.

The issue reproduces consistently and is not caused by any external factor — disabling DHT, PEX, and LSD made no measurable difference, nor did adjusting connection limits, disk cache, async I/O threads, or refresh interval.

Stopping all torrents solves the issue, but restarting even a single torrent for seeding, even when no peers are leeching, sends the CPU usage back to the current highest state immediately, suggesting it's not a per-torrent problem.

---

Diagnostic findings

strace analysis (two runs: fresh restart and after 40h drift)

Fresh restart (baseline ~3% CPU):

• ppoll dominant at ~90% of syscall time, ~750 calls/second — event loop already spinning faster than expected (analyzed by Claude)

After 40h drift (~30% CPU):

• futex becomes dominant at ~54% of syscall time
• ppoll at ~37%
• connect(): ~60,000 calls with ~99% failure rate (57,328 errors out of 57,395 in one run) (analyzed by Claude, worth a double check against the logs)

Latest run (after 40h drift) available here: https://logs.notifiarr.com/?c5cc9bd12c459dac#2fyEpJy6NkNfdfrpTLk6ssew3zB3dL96iWURyYNSYqpV

Claude says the pattern suggests libtorrent's peer connection retry logic is feeding a feedback loop:

1. Tracker announces return peer lists (~50 peers × 120 torrents)
2. Most outbound connection attempts fail instantly (peers behind NAT, unreachable from VPN IP)
3. Each failure generates an internal alert, waking the ppoll event loop
4. Multiple threads compete to process the alert queue, causing futex contention
5. Over time, peer candidate lists accumulate stale entries that are retried continuously without sufficient backoff

Key observations:

• ss shows zero established peer connections despite 120 active torrents — all traffic goes through the correctly forwarded VPN port, though, so maybe I just don't understand how to properly run ss in the context of a VPN within a container
• fastresume files are not bloated (6.2MB across 253 files)
• No file descriptor leak (191 fds)
• WireGuard tunnel is healthy (0% packet loss, clean handshakes)
• nftables firewall rules are correct and expected
• The drift rate and syscall pattern are consistent across multiple restart cycles

---

Hypothesis (Claude's hypothesis)

libtorrent does not appear to apply sufficient exponential backoff or hard blacklisting to peers that fail connection attempts instantly and repeatedly. On a long-running session with many torrents and a high ambient connection failure rate (typical for VPN-bound clients on private trackers), the peer retry queue grows unbounded, causing the event loop to spin progressively harder over time.

This may be specific to or exacerbated by ARM64, where thread synchronization (futex) overhead is higher than on x86.

---

Workaround

Scheduled weekly container restart. Not a fix, but prevents CPU from reaching critical levels.

[zonzille]

Follow-up : after setting async IO threads to 2 and setting global limits to the following numbers :

The issue seems to have gone away. I'm currently sitting at 4 days uptime and CPU use is hovering around 7-10%. I'm sorry if I made a fuss about a non-issue... I'll keep monitoring in case anyone wants to investigate the strace numbers, but otherwise I guess you could consider this solved.

[arvidn]

libtorrent keeps track of failcount for peers, backs off and eventually stops trying to connect. however:

• If we're successfully connected to the peer, the failcount is reset to 0.
• If we see the peer in a tracker or DHT response (i.e. some indication of the peer still being alive) we decrement the fail counter to allow one more attempt.

Is it possible this is related to #8519 ?

If a lot of peers are not seeds, but merely upload-only, we may assume that they're still interested in downloading and we connect to them.

The best way of understanding what's spending CPU would be to profile. either sample (MacOS) or perf (linux) are good options. (perf needs to be told to use a larger buffer for stack samples, otherwise the results will be very difficult to read).

[zonzille]

Thank you for replying. Actually this morning the CPU usage was back at ~35% stable, after a night of downloading ~150GB split in about 20 torrents. All from the same private tracker. I restarted the container this morning (I should have known better... sorry). Anyway I still ran those commands :

sudo perf record -F 99 -p 994268 -g --call-graph dwarf,16384 -- sleep 60 with 994268 being qbit's PID, then

sudo perf report --stdio > perf_report.txt which produced the file I'm attaching here.

perf_report.txt

I hope there's relevant info in there, otherwise I'll try and replicate the situation later, but I'm busy this week and won't necessarily have the time. Thanks for taking the time anyway, it's much appreciated.