fix: send and store real sub-second emission durations - #1374
davidberenstein1957 wants to merge 3 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1374 +/- ##
==========================================
+ Coverage 91.70% 91.71% +0.01%
==========================================
Files 49 49
Lines 5157 5164 +7
==========================================
+ Hits 4729 4736 +7
Misses 428 428 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
dfe1c2f to
ed11471
Compare
ed11471 to
735be78
Compare
Adds an ASGI middleware that gives each HTTP request its share of a long-running tracker's energy, plus the attribution model behind it. One tracker runs for the app's lifetime. Each completed sampling window (t_prev, t_now, dE) is split across the requests in flight during it, weighted by their overlap with the window and normalised by the sum of the weights. Windows with nothing in flight are recorded as unattributed. The invariant attributed + unattributed == settled holds exactly after every window, and is what the concurrency test pins down. Why not per-request start/stop energy snapshots: with N requests in flight each request observes the whole machine's delta, so the sum overcounts by roughly N - measured up to 88x at 100 concurrent requests. Fair-share weighting is the only split that conserves the run total. A request's share is only known one or more sampling windows after its response was sent, so results are reported then, via a callback. A request that never covered a completed window reports energy_kwh=None rather than zero: there is no honest number for it. Tracker side: add_energy_window_observer / remove_energy_window_observer expose the sampling windows, and http_request_emissions() scales the run's EmissionsData down to one attributed share using the run's accumulated component ratios and carbon intensity. Depends on #1374 (duration int -> float in the emissions schemas, and dropping the duration < 1 send guard) and #1375 (scheduler pause handling around tasks). Both are carried by their own PRs rather than duplicated here, so this should merge after them. Deliberately left out, to keep the diff reviewable: hardware-tier gating of which backends can resolve a sampling window, include/exclude path filtering (endpoint labelling is two lines inline), idle-baseline subtraction, per-endpoint aggregation, routing per-request rows into the tracker's own CSV/API output handlers, a lifespan helper, and a dedicated docs page. Each is additive on top of this and can follow if there is demand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds an ASGI middleware that gives each HTTP request its share of a long-running tracker's energy, plus the attribution model behind it. One tracker runs for the app's lifetime. Each completed sampling window (t_prev, t_now, dE) is split across the requests in flight during it, weighted by their overlap with the window and normalised by the sum of the weights. Windows with nothing in flight are recorded as unattributed. The invariant attributed + unattributed == settled holds exactly after every window, and is what the concurrency test pins down. Why not per-request start/stop energy snapshots: with N requests in flight each request observes the whole machine's delta, so the sum overcounts by roughly N - measured up to 88x at 100 concurrent requests. Fair-share weighting is the only split that conserves the run total. A request's share is only known one or more sampling windows after its response was sent, so results are reported then, via a callback. A request that never covered a completed window reports energy_kwh=None rather than zero: there is no honest number for it. Tracker side: add_energy_window_observer / remove_energy_window_observer expose the sampling windows, and http_request_emissions() scales the run's EmissionsData down to one attributed share using the run's accumulated component ratios and carbon intensity. Depends on #1374 (duration int -> float in the emissions schemas, and dropping the duration < 1 send guard) and #1375 (scheduler pause handling around tasks). Both are carried by their own PRs rather than duplicated here, so this should merge after them. Deliberately left out, to keep the diff reviewable: hardware-tier gating of which backends can resolve a sampling window, include/exclude path filtering (endpoint labelling is two lines inline), idle-baseline subtraction, per-endpoint aggregation, routing per-request rows into the tracker's own CSV/API output handlers, a lifespan helper, and a dedicated docs page. Each is additive on top of this and can follow if there is demand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
735be78 to
0ef7f00
Compare
Verdict: ✅ Approve, but deploy the server before releasing the clientMoving
Deploy ordering: new client → old server loses data (medium)
Nit:
|
The `duration` field was typed `int` in the pydantic schemas while the DB column and ORM were always `Float`, so every duration sent to the API was truncated to whole seconds. `ApiClient.add_emission` compounded this by dropping any measurement shorter than one second outright, which silently discarded data from short tasks and from trackers running a small `measure_power_secs`. Type the schemas as float to match the column, and stop rejecting non-integer durations. Emissions with a non-positive duration are still skipped, since those carry no measurement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0ef7f00 to
a5c5724
Compare
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Made the changes in b71cd53: added a deploy-order and self-hoster release note to the description, reworded the test docstrings. |
…oats Servers older than this client declare duration as int and answer 422 on fractional seconds. Retry once with a rounded duration so emissions are not dropped. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Made the changes in 0ddda1a: client retries once with a rounded duration on a 422, so old servers keep receiving emissions. |
Description
durationchanges frominttofloatincodecarbon/core/schemas.pyandcarbonserver/carbonserver/api/schemas.py(EmissionBase), matching the database column and ORM, which were alreadyColumn(Float).ApiClient.add_emissionno longer refuses emissions shorter than one second and no longer truncates the duration withint(...); it still skips a non-positive duration, since the server declaresdurationasField(..., gt=0).ExperimentReport,ProjectReportandOrganizationReportwidendurationfrominttofloatbecause those endpoints sum a Float column into a field declaredint, which only validates today by luck. A later commit adds a retry that rounds the duration when an older server (stillint-typed) rejects a fractional value with a 422, so emissions are not dropped during a mixed-version rollout. Extracted from #1203, which bundled this with unrelated FastAPI middleware work.Related Issue
N/A
Motivation and Context
The pydantic schemas claimed
duration: intwhile the database column was alwaysFloat, so every duration sent to the API was truncated to whole seconds, and the client's< 1guard silently dropped measurements from any short-lived task. This also created a latent bug: the first sub-second duration stored would make a report endpoint'sSUM()non-integral and raise a validation error, a latent 500 on the experiment/project/organization report endpoints.How Has This Been Tested?
carbonserver/tests/api/test_schema_compatibility.py::test_millisecond_duration_survives_client_to_server— a client payload withduration=0.0042validates against the server schema unchanged.tests/test_api_call.py::TestApi::test_add_emission_sends_millisecond_duration_unchanged— replacestest_add_emission_skips_short_duration; asserts the POST body carries0.0042.tests/test_api_call.py::TestApi::test_add_emission_skips_zero_duration— aduration=0.0flush is dropped client-side.master(ValidationError, and "emissions not sent because of a duration smaller than 1") and pass with the fix.uv run pytest tests/ -q --ignore=tests/test_viz_data.py→ 626 passed, 21 skipped. carbonserver unit tests → 108 passed.pre-commit run --all-filesclean.Screenshots (if appropriate):
N/A
Types of changes
AI Usage Disclosure
Checklist:
Deploy order: server first
No DB migration is needed;
emissions.durationis alreadyColumn(Float), so only the pydantic layer changes. A new client sending a fractionaldurationto an old (int-typed) server gets a 422, andHTTPOutputonly logs it, so emissions would be silently dropped. Deploy api.codecarbon.io with this PR before releasing the package to PyPI. Self-hosters should upgrade their carbonserver before or together with thecodecarbonclient.