fix: lock CPU power history swap - #1335
davidberenstein1957 wants to merge 3 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1335 +/- ##
==========================================
+ Coverage 91.70% 91.74% +0.04%
==========================================
Files 49 49
Lines 5157 5160 +3
==========================================
+ Hits 4729 4734 +5
+ Misses 428 426 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
d70b9ae to
ed2b386
Compare
00f2a1e to
b813d76
Compare
The 1 Hz monitor scheduler thread appends to `CPU._power_history` while the measurement scheduler thread drains it. The read-then-rebind in `total_power()` is not atomic, so any sample appended between the list comprehension and the rebinding was written to the discarded list and lost, biasing the reported `cpu_power`. Take the swap under a lock, held only for the O(1) rebinding so a slow `_get_power_from_cpus()` backend never blocks the monitor thread. Also drop the unreachable empty-history branch: a sample is always appended before the average. The test asserts the drain invariant rather than the drain shape, so it does not pin the implementation. Closes #1315 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b813d76 to
f72f432
Compare
Verdict: ✅ Approve with nitsThe race is real. The monitor scheduler can append to the old list between the list comprehension and the
56 tests pass on this PR merged with master. Nits:
|
…window Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Made the changes in 44ef635: merged master, added a test that a post-swap sample lands in the next window. Not done: updating the description's test paragraph (my PR-edit was blocked; the diff's tests are the lock-blocking one plus the new next-window one, no list subclass). |
Description
Fixes a lost-update race on
CPU._power_history, which is appended to by the 1 Hz monitor scheduler thread (CPU.monitor_power) and drained by themeasure_power_secsscheduler thread (CPU.total_power) — two distinctPeriodicSchedulertimer threads. The read-then-rebind intotal_powerwas not atomic, so any sample appended between the list comprehension andself._power_history = []was silently discarded. Added athreading.LocktoCPU, held only for the O(1) list swap;_get_power_from_cpus()still runs outside the lock in both paths so a slow backend (e.g. theIntelPowerGadgetsubprocess) never blocks the monitor thread. Also removed an unreachableif not power_history_in_W:branch, since a sample is unconditionally appended before the average is taken.Related Issue
Fixes #1315
Motivation and Context
In
cpu_loadmode the reportedcpu_poweris the mean of the buffered samples. The samples dropped in the unlocked swap window are exactly those taken while a slow measurement is in flight, so the resulting error is small but systematic and silent.How Has This Been Tested?
New test
tests/test_cpu_load.py::TestCPULoad::test_cpu_total_power_keeps_samples_added_while_drainingmakes the race deterministic (alistsubclass that firesmonitor_power()at iteration exhaustion, inside the lost-update window) and asserts the injected sample survives. It fails on master and passes with this change.uv run pytest tests/test_cpu_load.py -q→ 9 passed.Screenshots (if appropriate):
N/A
Types of changes
AI Usage Disclosure
Checklist:
Note:
uv run task formatreformats a large number of unrelated files on the current tree, so only the two touched files (codecarbon/external/hardware.py,tests/test_cpu_load.py) are included in this PR.