enhancement(metrics): support for metrics v3 protocol

[tobz]

Summary

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

References

[tobz]

• enhancement(metrics): support for metrics v3 protocol #1175 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: da59d73 · Comparison: e78a605 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 40.55 MiB (baseline) vs 40.75 MiB (comparison)
Size Change: +208.42 KiB (+0.50%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
saluki_components::encoders::datadog	+115.30 KiB	428
core	+72.91 KiB	15490
figment	+58.52 KiB	645
saluki_components::common::datadog	+57.63 KiB	539
serde	-42.73 KiB	94
alloc	-35.21 KiB	3064
hashbrown	+33.45 KiB	1481
saluki_components::sources::dogstatsd	-21.29 KiB	355
anon.c631d4f28c2b2db28eff0b5e986a13c9.248.llvm.11665572444932606208	-16.70 KiB	1
anon.a14e3b46faabe6cbf5d496845f50a791.1.llvm.5370817080494531371	+16.70 KiB	1
anon.0beb901ed35d6fa3b889001aad262adc.649.llvm.782532261936460638	+15.25 KiB	1
anon.661ef608d0e9e2423c9230617f2bc873.62.llvm.10199967635082326453	-15.25 KiB	1
datadog_protos::trace_piecemeal_include::datadog	+15.18 KiB	21
piecemeal	-14.10 KiB	43
tokio	-13.62 KiB	4749
&mut serde_json	+12.61 KiB	91
hyper_util	-11.49 KiB	161
quick_cache	-8.91 KiB	166
serde_core	-8.42 KiB	992
saluki_core::topology::blueprint	-8.33 KiB	89

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.6%  +148Ki  +0.7%  +122Ki    [55159 Others]
  [NEW] +57.7Ki  [NEW] +57.6Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h43bea40865104461
  [NEW] +40.3Ki  [NEW] +40.2Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hedea565e69c57390
  [NEW] +39.1Ki  [NEW] +38.9Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h4aaa6af49c9fde5b
  [NEW] +36.2Ki  [NEW] +35.9Ki    _<saluki_components::common::datadog::obfuscation::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::obfuscation::ObfuscationConfig>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h85965b50bfe07de9
  [NEW] +30.3Ki  [NEW] +30.2Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::h3f44af8f5c14ae9f
  [NEW] +29.0Ki  [NEW] +28.8Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h5b92aa19ac8a2b33
  [NEW] +28.5Ki  [NEW] +28.4Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::he59c2dac62fdc695
  [NEW] +25.7Ki  [NEW] +25.5Ki    _<saluki_components::sources::dogstatsd::DogStatsDConfiguration as saluki_core::components::sources::builder::SourceBuilder>::build::_{{closure}}::h8b6bbe704f229baf
  [NEW] +25.1Ki  [NEW] +24.9Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::h93f3449c5a9aaa43
  [NEW] +24.8Ki  [NEW] +24.7Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::h62ba5dfe67284c63
  [NEW] +24.6Ki  [NEW] +24.4Ki    agent_data_plane::internal::remote_agent::run_remote_agent_registration_loop::_{{closure}}::h0ee20f9c16ea5c40
  [DEL] -24.5Ki  [DEL] -24.3Ki    agent_data_plane::internal::remote_agent::run_remote_agent_registration_loop::_{{closure}}::hcf083b5830259c14
  [DEL] -25.0Ki  [DEL] -24.9Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::h4618b6207a99eb2b
  [DEL] -25.1Ki  [DEL] -24.9Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::h9f679e5a6425f93c
  [DEL] -28.3Ki  [DEL] -28.2Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::h6032372c5d685c9d
  [DEL] -30.4Ki  [DEL] -30.3Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hd47e947b534c018e
  [DEL] -33.7Ki  [DEL] -33.5Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::hb4a84cbbc21a072f
  [DEL] -39.8Ki  [DEL] -39.7Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h772878347210b56d
  [DEL] -40.0Ki  [DEL] -39.8Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h622a7084ee8b9e54
  [DEL] -54.3Ki  [DEL] -54.1Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h2c37bd2f0a7c3fc7
  +0.5%  +208Ki  +0.5%  +182Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 520349ea-7d68-4e04-9933-42aa0034b0e7
Baseline: da59d73b · Comparison: e78a6054 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_idle	memory	⚪ +1.32	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ +0.87	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.68	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ +0.41	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.30	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 131 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 42.9 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 65.6 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 191 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 28.8 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[tobz]

This is temporarily blocked on there being a version of the Datadog Agent for us to test against in correctness tests that has up-to-date v3 metrics support.

Currently, we're hitting an issue related to rate intervals being delta encoded when they shouldn't be. That bug is fixed in DataDog/datadog-agent#45825 but won't be released until 7.77: roughly 2 weeks from now before an RC is available to use. We can potentially do a hacky image build or something for keep going in the meantime and then switch back to a proper Agent version once available, we'll see.

[tobz]

We've temporarily handled the issue of correctness tests by using a "dev" container image (datadoghq/agent-dev) based on the latest fix for V3 support in the Datadog Agent. With this in place, we're now currently passing for both dsd-plain (V2 payloads) and dsd-plain-v3 (new, V3 payloads only).

We can't merge this as-is: we need to wait for at least an RC build of Datadog Agent 7.77 so we can pin to a non-development image. In the meantime, I'm going to work on making sure we've integrated all of the same small fixes/changes that have been steadily being made upstream in the Datadog Agent repository for V3 support.