Skip to content

Enforce last-value-wins semantics in AttributesMap without performance regression#8548

Open
EvgeniiR wants to merge 4 commits into
open-telemetry:mainfrom
EvgeniiR:respect-last-value-wins-semantics-clean
Open

Enforce last-value-wins semantics in AttributesMap without performance regression#8548
EvgeniiR wants to merge 4 commits into
open-telemetry:mainfrom
EvgeniiR:respect-last-value-wins-semantics-clean

Conversation

@EvgeniiR

@EvgeniiR EvgeniiR commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Fixes #7897

Problem

AttributesMap extended HashMap<AttributeKey<?>, Object>, using AttributeKey as the map
key. Because AttributeKey.equals() includes the attribute type, two attributes with the same
name but different types (e.g. stringKey("http.method") and longKey("http.method")) were
stored as separate entries.
This violated the OpenTelemetry specification, which requires that
attribute name alone determines identity — last write wins regardless of type.

Solution

Correctness fix

Replace the HashMap<AttributeKey<?>, Object> backing store with a string-keyed map so that
put("http.method", String) followed by put("http.method", Long) results in exactly one entry
(the Long value), consuming only one capacity slot.

The first implementations were based on HashMap/LinkedHashMap (2nd performed better once forEach was included in the benchmarks), but they introduced another issue — the fixed AttributesMap performed ~40-80% worse than baseline.
Therefore, I started looking for a better solution that would preserve the required last-value-wins semantics without introducing a performance regression.

LinkedHashMap implementation can be observed in the previous commit — https://github.com/EvgeniiR/opentelemetry-java/blob/d7df58af76e693aa1fe897d2757e2bdb50ab9798/sdk/common/src/main/java/io/opentelemetry/sdk/common/internal/AttributesMap.java

Final solution is described below.

Performance optimization

Instead of LinkedHashMap<String, AttributeEntry>, use parallel arrays with an open-addressing
int[] hash table (linear probing, load factor ≤ 0.5):

int[]           hashTable   — slot → entryIndex+1 (0 = empty, JVM zero-init)
String[]        entryNames  — for hash lookup
AttributeKey[]  entryKeys   — for get() type check and forEach()
Object[]        entryValues — attribute values

forEach becomes a tight sequential array loop with no pointer chasing, directly benefiting the export.

Benchmark results (avgt ns/op, lower is better)

putThenForEach — N unique puts + 1 forEach (dominant production path)

n attrs baseline (spec-violating) LHM fix PA fix
4 36.3 51.8 34.5
16 135.6 244.5 137.5
20 166.8 == 260.2*
128 1321.4 2145.9 1968.3

* This is a worst-case for the new implementation. Performance for spans with 16–32 attributes could be improved by raising the initial array size from 16 to 32, at the cost of extra memory per map. I'm not sure where the right tradeoff sits. Currently our implementation uses less memory then the baseline:

AttributesMapBenchmark — putThenForEach memory allocation (gc.alloc.rate.norm, B/op, lower is better)

n attrs baseline PA fix delta
4 272 144 −47%
16 800 384 −52%
128 6224 5104 −18%

PA matches the baseline at n=4 and n=16 while being spec-correct. Given that, I think that further performance optimizations(besides may be changing init size) are out of scope of this PR.

EvgeniiR added 2 commits June 26, 2026 23:41
…ixes open-telemetry#7897)

AttributesMap previously extended HashMap<AttributeKey<?>, Object>, where
AttributeKey.equals() includes the AttributeType. This caused attributes with
the same string name but different types to coexist as separate entries,
violating the OTel spec last-value-wins rule.

Replace the HashMap backing with LinkedHashMap<String, AttributeEntry> keyed
by raw attribute name. Overwrites with a different type now update the existing
entry in place, so size() stays correct and capacity limits are not consumed.
Also eliminates the double hash-probe in put() (containsKey + get → single get).
@EvgeniiR EvgeniiR requested a review from a team as a code owner June 26, 2026 21:59
@codecov

codecov Bot commented Jun 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.10145% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.99%. Comparing base (8f41449) to head (7373807).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...entelemetry/sdk/common/internal/AttributesMap.java 97.10% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8548      +/-   ##
============================================
+ Coverage     90.96%   90.99%   +0.02%     
- Complexity    10206    10227      +21     
============================================
  Files          1013     1013              
  Lines         27166    27228      +62     
  Branches       3182     3191       +9     
============================================
+ Hits          24712    24776      +64     
+ Misses         1730     1729       -1     
+ Partials        724      723       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@EvgeniiR

Copy link
Copy Markdown
Contributor Author

Benchmark results ... putThenForEach — N unique puts + 1 forEach

This PR also includes benchmarks for individual read/write operations (uniqueKeys, getHit, getTypeMiss, forEachAll). The project already has SpanRecordBenchmark and FillSpanBenchmark that cover the write path at the span level. I will publish some additional benchmarks under cut, if for some reason you want to check them out too.

More benchmarks

AttributesMapBenchmark — write (avg ns/op, lower is better)

Benchmark n baseline LHM fix PA PA vs base PA vs LHM
uniqueKeys 4 41.8 48.9 38.2 −3.6 (−9%) −10.7 (−22%)
uniqueKeys 16 134.9 231.0 134.3 −0.6 (0%) −96.7 (−42%)
uniqueKeys 128 1309.7 2199.7 1950.2 +640.5 (+49%) −249.5 (−11%)
sameKeySameType 4 46.1 33.7 35.0 −11.1 (−24%) +1.3 (+4%)
sameKeySameType 16 164.5 105.4 124.6 −39.9 (−24%) +19.2 (+18%)
sameKeySameType 128 1320.7 659.1 862.5 −458.2 (−35%) +203.4 (+31%)
sameKeyDifferentType 4 32.1 42.9 42.9 +10.8 (+34%) 0.0 (0%)
sameKeyDifferentType 16 181.6 146.5 156.6 −25.0 (−14%) +10.1 (+7%)
sameKeyDifferentType 128 1407.2 1073.0 1183.7 −223.5 (−16%) +110.7 (+10%)
mixedUniqueAndOverwrite 4 32.1 43.2 44.4 +12.3 (+38%) +1.2 (+3%)
mixedUniqueAndOverwrite 16 134.0 155.7 154.5 +20.5 (+15%) −1.2 (−1%)
mixedUniqueAndOverwrite 128 1400.7 1531.9 1721.5 +320.8 (+23%) +189.6 (+12%)

AttributesMapBenchmark — read (avg ns/op, lower is better)

Benchmark n baseline LHM fix PA PA vs base PA vs LHM
getHit 4 10.9 15.8 18.1 +7.2 (+66%) +2.3 (+15%)
getHit 16 41.7 58.3 73.7 +32.0 (+77%) +15.4 (+26%)
getHit 128 350.6 476.6 916.9 +566.3 (+161%) +440.3 (+92%)
getTypeMiss 4 8.7 23.9 24.7 +16.0 (+184%) +0.8 (+3%)
getTypeMiss 16 35.0 96.9 97.3 +62.3 (+178%) +0.4 (+0%)
getTypeMiss 128 283.3 856.9 1136.7 +853.4 (+301%) +279.8 (+33%)
forEachAll 4 11.1 5.8 2.9 −8.2 (−74%) −2.9 (−50%)
forEachAll 16 26.0 25.3 8.1 −17.9 (−69%) −17.2 (−68%)
forEachAll 128 185.5 229.9 46.8 −138.7 (−75%) −183.1 (−80%)

FillSpanBenchmark (ops/ms, higher is better)

Benchmark baseline LHM fix PA PA vs base PA vs LHM
setFourAttributes 6160.0 5589.7 5197.7 −962.3 (−16%) −392.0 (−7%)

SpanRecordBenchmark (ops/s, higher is better)

threads SpanSize baseline LHM fix PA PA vs base PA vs LHM
1 SMALL 6 043 137 6 079 646 5 334 831 −708 306 (−12%) −744 815 (−12%)
1 MEDIUM 658 387 650 851 574 200 −84 187 (−13%) −76 651 (−12%)
1 LARGE 65 240 62 313 64 207 −1 033 (−2%) +1 894 (+3%)
4 SMALL 8 785 156 9 009 770 8 454 487 −330 669 (−4%) −555 283 (−6%)
4 MEDIUM 1 535 077 1 697 175 1 463 172 −71 905 (−5%) −234 003 (−14%)
4 LARGE 165 137 175 858 165 824 +687 (+0.4%) −10 034 (−6%)

@EvgeniiR EvgeniiR changed the title Enforce last value wins semantics clean in AttributesMap Enforce last-value-wins semantics in AttributesMap without performance regression Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AttributeMap does not respect last-value-wins semantics for duplicate keys (SpanBuilder.setAttribute vs Attributes.builder)

1 participant