Skip to content

dedup: don't inject a counter adjustment before any sample is emitted - #9041

Open
DeviousCardi wants to merge 2 commits into
thanos-io:mainfrom
DeviousCardi:fix/9034-dedup-spurious-initial-counter-adjust
Open

DeviousCardi wants to merge 2 commits into
thanos-io:mainfrom
DeviousCardi:fix/9034-dedup-spurious-initial-counter-adjust

Conversation

@DeviousCardi

Copy link
Copy Markdown
  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

newDedupSeriesIterator pre-seeds lastIter: a, useA: true before any sample has been emitted, while lastT is still math.MinInt64. Next() captures that as lastUseA and, if it then picks replica b, fires the counter-reset adjustment against replica a's first sample — a value that was never emitted. adjustAtValue sees a decrease, concludes "counter reset", and installs a permanent errAdjust equal to the inter-replica scrape offset.

The result is a counter flattened at the start of the window, so rate() under-reports. Query-frontend range splitting makes it visible in practice: each split gets a fresh iterator over a different first sample, and split starts are aligned to absolute interval multiples, so the dip lands at the same wall-clock boundaries every time.

This guards the adjustment on emittedAny := it.lastT != math.MinInt64, reusing the sentinel Next() already uses twice for the initial-penalty decision. Genuine mid-stream replica switches still adjust, so the #2401 mitigation is untouched.

Closes #9034

Verification

Synthetic counter increasing exactly 1/s across two replicas at 30s scrape with a 15s offset, so the true rate() is exactly 1.0:

before after
unsplit 1.000000 1.000000
split at 1800000 0.666667 1.000000
mismatching points 1 of 61 0 of 61

The corrupted sample is directly visible — t=1755000 v=1770.000 where it should be 1755.000, giving delta 30 over 45s.

Controls isolate it to this code path: a single replica, --deduplication.func=chain, and irate all return 1.000000 before and after.

New table case Regression test against 9034 in TestDedupSeriesSet, with replica b holding the earliest sample. Reverting only iter.go makes it fail with +3 on every sample — exactly 103 - 100, the inter-replica offset. TestDedupSeriesSet/Regression_test_against_2401 still passes.

go test ./pkg/dedup/... -count=1 and -race both pass; gofmt clean; no new go vet warnings.

newDedupSeriesIterator pre-seeds lastIter to replica "a" with useA=true
before anything has been emitted, while lastT is still math.MinInt64.
The first Next() therefore captures lastUseA=true and reads lastFloatVal
from replica a's first sample. If replica b owns the earliest sample of
the window, useA flips to false and this looks like a replica switch, so
counterErrAdjustSeriesIterator.adjustAtValue sees lastFloatValue > v and
records a permanent errAdjust equal to the inter-replica scrape offset.

Replica b's counter is then flattened at the start of the window and
rate() under-reports for the whole range. Query-range splitting is the
trigger rather than the defect: each split gets a fresh iterator over a
different first sample, so results alternate between correct and wrong
depending on which replica owns the first sample of each split.

No sample has been emitted at that point, so there is no previous value
to adjust against. Guard the adjustment on lastT != math.MinInt64, the
same "nothing emitted yet" sentinel the penalty logic already uses in
this function. A genuine mid-stream replica switch still adjusts, so the
mitigation for thanos-io#2401 is unaffected.

Fixes thanos-io#9034

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>
The PR template asks for `- [#<PR-id>](<PR-URL>) Thanos <Component> ...`
with the pull URL. The entry was written before the PR existed, so it
referenced the issue.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>
@codecov

codecov Bot commented Sep 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.56%. Comparing base (050d688) to head (6f41f18).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #9041   +/-   ##
=======================================
  Coverage   64.55%   64.56%           
=======================================
  Files         289      289           
  Lines       37412    37413    +1     
=======================================
+ Hits        24153    24157    +4     
+ Misses      11158    11156    -2     
+ Partials     2101     2100    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

query-frontend/query: penalty HA deduplication produces incorrect rate() values at split boundaries

1 participant