Skip to content

fix: send and store real sub-second emission durations - #1374

Open
davidberenstein1957 wants to merge 3 commits into
masterfrom
fix/subsecond-emission-duration
Open

davidberenstein1957 wants to merge 3 commits into
masterfrom
fix/subsecond-emission-duration

Conversation

@davidberenstein1957

@davidberenstein1957 davidberenstein1957 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Description

duration changes from int to float in codecarbon/core/schemas.py and carbonserver/carbonserver/api/schemas.py (EmissionBase), matching the database column and ORM, which were already Column(Float). ApiClient.add_emission no longer refuses emissions shorter than one second and no longer truncates the duration with int(...); it still skips a non-positive duration, since the server declares duration as Field(..., gt=0). ExperimentReport, ProjectReport and OrganizationReport widen duration from int to float because those endpoints sum a Float column into a field declared int, which only validates today by luck. A later commit adds a retry that rounds the duration when an older server (still int-typed) rejects a fractional value with a 422, so emissions are not dropped during a mixed-version rollout. Extracted from #1203, which bundled this with unrelated FastAPI middleware work.

Related Issue

N/A

Motivation and Context

The pydantic schemas claimed duration: int while the database column was always Float, so every duration sent to the API was truncated to whole seconds, and the client's < 1 guard silently dropped measurements from any short-lived task. This also created a latent bug: the first sub-second duration stored would make a report endpoint's SUM() non-integral and raise a validation error, a latent 500 on the experiment/project/organization report endpoints.

How Has This Been Tested?

  • carbonserver/tests/api/test_schema_compatibility.py::test_millisecond_duration_survives_client_to_server — a client payload with duration=0.0042 validates against the server schema unchanged.
  • tests/test_api_call.py::TestApi::test_add_emission_sends_millisecond_duration_unchanged — replaces test_add_emission_skips_short_duration; asserts the POST body carries 0.0042.
  • tests/test_api_call.py::TestApi::test_add_emission_skips_zero_duration — a duration=0.0 flush is dropped client-side.
  • Both fail on master (ValidationError, and "emissions not sent because of a duration smaller than 1") and pass with the fix.
  • uv run pytest tests/ -q --ignore=tests/test_viz_data.py → 626 passed, 21 skipped. carbonserver unit tests → 108 passed. pre-commit run --all-files clean.

Screenshots (if appropriate):

N/A

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

AI Usage Disclosure

  • 🟥 AI-vibecoded: You cannot explain the logic. Car analogy : the car drive by itself, you are outside it and just tell it where to go.
  • 🟠 AI-generated: Car analogy : the car drive by itself, you are inside and give instructions.
  • ⭐ AI-assisted. Car analogy : you drive the car, AI help you find your way.
  • ♻️ No AI used. Car analogy : you drive the car.

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the docs/how-to/contributing.md document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

Deploy order: server first

No DB migration is needed; emissions.duration is already Column(Float), so only the pydantic layer changes. A new client sending a fractional duration to an old (int-typed) server gets a 422, and HTTPOutput only logs it, so emissions would be silently dropped. Deploy api.codecarbon.io with this PR before releasing the package to PyPI. Self-hosters should upgrade their carbonserver before or together with the codecarbon client.

@codecov

codecov Bot commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.71%. Comparing base (e5e46ab) to head (0ddda1a).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1374      +/-   ##
==========================================
+ Coverage   91.70%   91.71%   +0.01%     
==========================================
  Files          49       49              
  Lines        5157     5164       +7     
==========================================
+ Hits         4729     4736       +7     
  Misses        428      428              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@davidberenstein1957
davidberenstein1957 marked this pull request as ready for review August 13, 2026 05:05
@davidberenstein1957
davidberenstein1957 requested a review from a team as a code owner August 13, 2026 05:05
@davidberenstein1957
davidberenstein1957 force-pushed the fix/subsecond-emission-duration branch from dfe1c2f to ed11471 Compare August 19, 2026 09:18
@davidberenstein1957
davidberenstein1957 force-pushed the fix/subsecond-emission-duration branch from ed11471 to 735be78 Compare August 19, 2026 14:12
davidberenstein1957 added a commit that referenced this pull request Aug 19, 2026
Adds an ASGI middleware that gives each HTTP request its share of a
long-running tracker's energy, plus the attribution model behind it.

One tracker runs for the app's lifetime. Each completed sampling window
(t_prev, t_now, dE) is split across the requests in flight during it,
weighted by their overlap with the window and normalised by the sum of the
weights. Windows with nothing in flight are recorded as unattributed. The
invariant attributed + unattributed == settled holds exactly after every
window, and is what the concurrency test pins down.

Why not per-request start/stop energy snapshots: with N requests in flight
each request observes the whole machine's delta, so the sum overcounts by
roughly N - measured up to 88x at 100 concurrent requests. Fair-share
weighting is the only split that conserves the run total.

A request's share is only known one or more sampling windows after its
response was sent, so results are reported then, via a callback. A request
that never covered a completed window reports energy_kwh=None rather than
zero: there is no honest number for it.

Tracker side: add_energy_window_observer / remove_energy_window_observer
expose the sampling windows, and http_request_emissions() scales the run's
EmissionsData down to one attributed share using the run's accumulated
component ratios and carbon intensity.

Depends on #1374 (duration int -> float in the emissions schemas, and
dropping the duration < 1 send guard) and #1375 (scheduler pause handling
around tasks). Both are carried by their own PRs rather than duplicated
here, so this should merge after them.

Deliberately left out, to keep the diff reviewable: hardware-tier gating of
which backends can resolve a sampling window, include/exclude path filtering
(endpoint labelling is two lines inline), idle-baseline subtraction,
per-endpoint aggregation, routing per-request rows into the tracker's own
CSV/API output handlers, a lifespan helper, and a dedicated docs page. Each
is additive on top of this and can follow if there is demand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
davidberenstein1957 added a commit that referenced this pull request Aug 19, 2026
Adds an ASGI middleware that gives each HTTP request its share of a
long-running tracker's energy, plus the attribution model behind it.

One tracker runs for the app's lifetime. Each completed sampling window
(t_prev, t_now, dE) is split across the requests in flight during it,
weighted by their overlap with the window and normalised by the sum of the
weights. Windows with nothing in flight are recorded as unattributed. The
invariant attributed + unattributed == settled holds exactly after every
window, and is what the concurrency test pins down.

Why not per-request start/stop energy snapshots: with N requests in flight
each request observes the whole machine's delta, so the sum overcounts by
roughly N - measured up to 88x at 100 concurrent requests. Fair-share
weighting is the only split that conserves the run total.

A request's share is only known one or more sampling windows after its
response was sent, so results are reported then, via a callback. A request
that never covered a completed window reports energy_kwh=None rather than
zero: there is no honest number for it.

Tracker side: add_energy_window_observer / remove_energy_window_observer
expose the sampling windows, and http_request_emissions() scales the run's
EmissionsData down to one attributed share using the run's accumulated
component ratios and carbon intensity.

Depends on #1374 (duration int -> float in the emissions schemas, and
dropping the duration < 1 send guard) and #1375 (scheduler pause handling
around tasks). Both are carried by their own PRs rather than duplicated
here, so this should merge after them.

Deliberately left out, to keep the diff reviewable: hardware-tier gating of
which backends can resolve a sampling window, include/exclude path filtering
(endpoint labelling is two lines inline), idle-baseline subtraction,
per-endpoint aggregation, routing per-request rows into the tracker's own
CSV/API output handlers, a lifespan helper, and a dedicated docs page. Each
is additive on top of this and can follow if there is demand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@davidberenstein1957
davidberenstein1957 force-pushed the fix/subsecond-emission-duration branch from 735be78 to 0ef7f00 Compare August 20, 2026 06:13
@benoit-cty

Copy link
Copy Markdown
Contributor

🤖 This review comment was written and posted by Claude Opus 5.5 (AI assistant), at the request of @benoit-cty. Findings were checked by reading the code and running tests locally (merged with current master where relevant), but please double-check before acting on them.

Verdict: ✅ Approve, but deploy the server before releasing the client

Moving duration from int to float in the client dataclass, EmissionBase and the three *Report schemas is correct.

  • No migration is needed: emissions.duration is already Column(Float) (sql_models.py:14).
  • Old client → new server: works; ints are accepted by the float field.
  • Report widening: needed, since the reports func.sum() a Float column into duration: int.
  • Webapp: uses z.number(), so it is unaffected.
  • Tests: 77 client tests and 56 carbonserver tests pass.

Deploy ordering: new client → old server loses data (medium)

  • codecarbon/core/api_client.py:201 now sends the unrounded duration (e.g. 15.0023).
  • An old server with duration: int rejects fractional floats: pydantic v2 int_from_float returns 422. HTTPOutput._emit (output_methods/http.py:66-69) only logs that, so nearly every emission is silently dropped.
  • This hits users if the package reaches PyPI before api.codecarbon.io is redeployed, and anyone on a self-hosted carbonserver older than this PR.
  • Please mention this in the PR description, and plan to deploy the API before the package release. Also add a release note for self-hosters. Optionally, the client could retry once with round(duration) on a 422 to stay compatible with old servers.

Nit:

The `duration` field was typed `int` in the pydantic schemas while the DB
column and ORM were always `Float`, so every duration sent to the API was
truncated to whole seconds. `ApiClient.add_emission` compounded this by
dropping any measurement shorter than one second outright, which silently
discarded data from short tasks and from trackers running a small
`measure_power_secs`.

Type the schemas as float to match the column, and stop rejecting
non-integer durations. Emissions with a non-positive duration are still
skipped, since those carry no measurement.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@benoit-cty
benoit-cty force-pushed the fix/subsecond-emission-duration branch from 0ef7f00 to a5c5724 Compare September 23, 2026 07:22
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@davidberenstein1957

Copy link
Copy Markdown
Collaborator Author

Made the changes in b71cd53: added a deploy-order and self-hoster release note to the description, reworded the test docstrings.
Skipped the optional retry with round(duration) on 422: deploying the server first covers it, happy to add if you want old-server compat.

…oats

Servers older than this client declare duration as int and answer 422 on
fractional seconds. Retry once with a rounded duration so emissions are
not dropped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@davidberenstein1957

Copy link
Copy Markdown
Collaborator Author

Made the changes in 0ddda1a: client retries once with a rounded duration on a 422, so old servers keep receiving emissions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants