Skip to content

Fix poll coroutine growth#1739

Merged
AlexxIT merged 3 commits into
AlexxIT:masterfrom
RonaldCoppieters:fix-poll-coroutine-growth
Mar 2, 2026
Merged

Fix poll coroutine growth#1739
AlexxIT merged 3 commits into
AlexxIT:masterfrom
RonaldCoppieters:fix-poll-coroutine-growth

Conversation

@RonaldCoppieters

Copy link
Copy Markdown
Contributor

Commits

  • fix: dedupe periodic uiActive refresh tasks per device
  • test: cover uiActive refresh task dedupe scheduling

Issue

  • custom_components/sonoff/core/ewelink/init.py:356
  • custom_components/sonoff/core/ewelink/init.py:359
  • custom_components/sonoff/core/ewelink/init.py:367
  • custom_components/sonoff/core/ewelink/init.py:377
  • custom_components/sonoff/core/ewelink/cloud.py:437
  • custom_components/sonoff/core/ewelink/cloud.py:438

XRegistry.run_forever() schedules fire-and-forget cloud uiActive updates every 30s for several UIIDs.
But XRegistryCloud.send() globally rate-limits any uiActive payload to 1 per second (last_ui_active shared across all devices).

If the site has >30 cloud-polled uiActive devices, production rate > service rate:

  • producer: N / 30 sends/sec
  • consumer cap: 1 send/sec for uiActive

That creates a queue that grows forever. Over days, thousands of pending send() coroutines accumulate and keep sleeping/waking, causes severe HA Core responsiveness degradation.

For example:

  • 42 devices that update via cloud => 42/30 = 1.4 uiActive sends/sec
  • cap = 1.0/sec
  • net backlog growth = 0.4/sec
  • that's ~24/min, ~1,440/hour, ~34,560/day pending coroutines

How to reproduce

  1. Configure >30 Sonoff devices that rely on cloud updates (uiActive) and cannot update via LAN.
  2. Leave Home Assistant running normally.

After several days of uptime, Home Assistant Core responsiveness gradually degrades (slow UI/API responses and delayed service execution) due to accumulation of pending cloud uiActive refresh tasks.

Fix

This change deduplicates the periodic cloud uiActive refresh per device, so each device can have at most one pending/in-flight uiActive refresh task at a time.

To keep the change minimal, it does not alter user-initiated command behavior. Normal cloud commands (e.g. turn_on/turn_off) will still queue indiscriminately and command rate could exceed cloud send capacity; this fix specifically prevents unbounded backlog growth from the periodic uiActive refresh scheduler. The original throttling behaviour in XRegistryCloud.send() also remains unchanged.

Profiling evidence

Production HA with 44 sonoff switches, captured for 60s with profiler.start.

Before fix (degraded state after a few days) vs post-restart baseline:

  • custom_components/sonoff/core/ewelink/cloud.py:send

    • baseline: 3,400 calls / 60s
    • degraded: 322,243 calls / 60s (~95x)
  • asyncio.sleep

    • baseline: 7,721
    • degraded: 645,058 (~84x)
  • asyncio scheduler churn:

    • asyncio.events.__lt__: 43,075 → 4,470,033 (~104x)
    • call_later: 3,866 → 322,541 (~83x)
    • call_at: 4,012 → 322,663 (~80x)
    • _call_soon: 4,562 → 323,746 (~71x)

Post-fix profiling (production HA after third day):

  • cloud.py:send: 1,694 calls / 60s (down from 322,243 degraded)
  • asyncio.sleep: 4,328 (down from 645,058 degraded)
  • scheduler churn normalized:
    • asyncio.events.__lt__: 24,132 (was 4,470,033 degraded)
    • call_later: 2,166 (was 322,541 degraded)
    • call_at: 2,283 (was 322,663 degraded)
    • _call_soon: 2,845 (was 323,746 degraded)

After applying this patch, the runaway pattern disappeared in production profiling (same 60s profiler.start capture).

Tests

  • This branch was tested for one week on our production Home Assistant environment (42 Sonoff switches updating via the cloud). The previously observed gradual server and UI degradation did not occur during testing.
  • A unit test has been added to prevent regression.
  • 71/71 tests passed; no new warnings.

@AlexxIT
AlexxIT merged commit 346102f into AlexxIT:master Mar 2, 2026
2 checks passed
@AlexxIT AlexxIT added this to the master milestone Mar 2, 2026
@AlexxIT

AlexxIT commented Mar 2, 2026

Copy link
Copy Markdown
Owner

Thanks! I simplified the code a little and removed the clearing of old tasks. They will be replaced with new ones anyway.

I think all the logic is preserved. I don't have any real devices to test it on.

PS. How you found this error is amazing!

@AlexxIT

AlexxIT commented Mar 5, 2026

Copy link
Copy Markdown
Owner

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants