Skip to content

Commit 61c31fa

Browse files
rayzrayz
authored
chore(config): add v3 enablement config (#1915)
## Summary <!-- Please provide a brief summary about what this PR does. This should help the reviewers give feedback faster and with higher quality. --> This pr adds ADP Metrics V3 enablement config while mirroring Datadog Agent V3 routing semantics. Config keys added: - `use_v3_api.series.enabled` - `use_v3_api.series.endpoints` - `observability_pipelines_worker.metrics.use_v3_api.series` - `vector.metrics.use_v3_api.series` ADP keeps V3 series off by default with the ADP-only safety gate: - `data_plane.metrics.v3.series.enabled` ADP also mirrors the Agent behavior that disables Metrics V3 when the compressor kind is `zlib`. ## Change Type - [ ] Bug fix - [ ] New feature - [ ] Non-functional (chore, refactoring, docs) - [ ] Performance ## How did you test this PR? <!-- Please how you tested these changes here --> Will be doing a deploy utilizing the new config flags. ## References <!-- Please list any issues closed by this PR. --> <!-- - Closes: <issue link> --> <!-- Any other issues or PRs relevant to this PR? Feel free to list them here. -->
1 parent e85e1b8 commit 61c31fa

16 files changed

Lines changed: 1119 additions & 63 deletions

File tree

‎docs/agent-data-plane/configuration/dogstatsd.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -320,6 +320,7 @@ The following settings are specific to ADP and have no equivalent in the core ag
320320
| `apm_config.obfuscation.sql.replace_digits` | Replace digits in SQL obfuscation | |
321321
| `apm_config.obfuscation.sql.table_names` | Collect table names during obfuscation | |
322322
| `counter_expiry_seconds` | Idle counter keep-alive duration | 300 |
323+
| `data_plane.metrics.v3.series.enabled` | Enable ADP V3 series | false |
323324
| `dogstatsd_allow_context_heap_allocs` | Allow heap allocations for contexts | |
324325
| `dogstatsd_autoscale_udp_listeners` | Bind multiple UDP sockets via SO_REUSEPORT | |
325326
| `dogstatsd_buffer_count` | Number of receive buffers | |
@@ -343,6 +344,10 @@ The following settings are specific to ADP and have no equivalent in the core ag
343344
| `otlp_string_interner_size` | OTLP context interner capacity | |
344345
| `serializer_max_metrics_per_payload` | Max metrics per payload | |
345346

347+
### `data_plane.metrics.v3.series.enabled`
348+
349+
ADP requires this flag before it generates or forwards authoritative V3 series payloads. This is separate from `use_v3_api.series.*`, which matches the Core Agent V3 routing configuration.
350+
346351
### `dogstatsd_minimum_sample_rate`
347352

348353
ADP enforces a minimum sample rate on incoming metrics to prevent memory exhaustion from extremely low sample rates on histograms and sketches. Sending metrics with a very high inverse sample rate (for example `@0.0000001`) can cause unbounded memory growth in a sketch; this setting prevents that. The default is conservative enough that normal clients are unaffected.
@@ -518,6 +523,7 @@ compressed wire payload bytes.
518523
| `no_proxy_nonexact_match` | Domain/CIDR `no_proxy` matching |
519524
| `observability_pipelines_worker.metrics.enabled` | Route metrics to OPW instance |
520525
| `observability_pipelines_worker.metrics.url` | OPW metrics intake URL |
526+
| `observability_pipelines_worker.metrics.use_v3_api.series` | Use V3 series for OPW |
521527
| `origin_detection_unified` | Unified origin detection mode |
522528
| `otlp_config.logs.enabled` | otlp_config.logs.enabled |
523529
| `otlp_config.metrics.enabled` | otlp_config.metrics.enabled |
@@ -558,8 +564,11 @@ compressed wire payload bytes.
558564
| `syslog_uri` | Syslog destination URI |
559565
| `use_proxy_for_cloud_metadata` | Proxy cloud metadata endpoints |
560566
| `use_v2_api.series` | Send series via V2 protobuf endpoint |
567+
| `use_v3_api.series.enabled` | Global V3 series mode |
568+
| `use_v3_api.series.endpoints` | Per-endpoint V3 series modes |
561569
| `vector.metrics.enabled` | Route metrics to OPW (legacy alias) |
562570
| `vector.metrics.url` | OPW metrics intake URL (legacy alias) |
571+
| `vector.metrics.use_v3_api.series` | Use V3 series for Vector |
563572
| `vsock_addr` | vsock address for Agent IPC endpoint |
564573

565574

‎lib/datadog-agent/config-overlay-model/src/saluki_keys.rs‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,25 @@ pub struct SalukiKey {
2222
}
2323

2424
pub static SALUKI_KEYS: &[SalukiKey] = &[
25+
// ── data_plane.rs ───────────────────────────────────────────────────────
26+
SalukiKey {
27+
yaml_path: "data_plane.metrics.v3.series.enabled",
28+
description: "Enable ADP V3 series",
29+
default: "false",
30+
documentation: Some(
31+
"ADP requires this flag before it generates or forwards authoritative V3 series payloads. \
32+
This is separate from `use_v3_api.series.*`, which matches the Core Agent V3 routing configuration.",
33+
),
34+
value_type: "ValueType::Bool",
35+
schema_default: Some("false"),
36+
env_vars: &[],
37+
env_var_override: None,
38+
additional_yaml_paths: &[],
39+
used_by: &["DATADOG_METRICS_CONFIGURATION", "FORWARDER_CONFIGURATION"],
40+
test_json: None,
41+
pipeline_affinity: "PipelineAffinity::Pipelines(&[Pipeline::DogStatsD])",
42+
filename: "data_plane.rs",
43+
},
2544
// ── dogstatsd.rs ─────────────────────────────────────────────────────────
2645
SalukiKey {
2746
yaml_path: "dogstatsd_allow_context_heap_allocs",

‎lib/datadog-agent/config-testing/src/config_registry/data_plane.rs‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,14 @@ use super::schema;
44
#[allow(unused_imports)]
55
use super::*;
66

7+
static DATA_PLANE_METRICS_V3_SERIES_ENABLED_SCHEMA: SchemaEntry = SchemaEntry {
8+
schema: Schema::Saluki,
9+
yaml_path: "data_plane.metrics.v3.series.enabled",
10+
env_vars: &[],
11+
value_type: ValueType::Bool,
12+
default: Some("false"),
13+
};
14+
715
crate::declare_annotations! {
816
/// `data_plane.api_listen_address`-Unprivileged API listen address
917
DATA_PLANE_API_LISTEN_ADDRESS = SalukiAnnotation {
@@ -27,6 +35,17 @@ crate::declare_annotations! {
2735
test_json: None,
2836
pipeline_affinity: PipelineAffinity::CrossCutting,
2937
};
38+
/// `data_plane.metrics.v3.series.enabled`
39+
DATA_PLANE_METRICS_V3_SERIES_ENABLED = SalukiAnnotation {
40+
schema: &DATA_PLANE_METRICS_V3_SERIES_ENABLED_SCHEMA,
41+
support_level: SupportLevel::Full,
42+
additional_yaml_paths: &[],
43+
env_var_override: None,
44+
used_by: &[structs::DATADOG_METRICS_CONFIGURATION, structs::FORWARDER_CONFIGURATION],
45+
value_type_override: None,
46+
test_json: None,
47+
pipeline_affinity: PipelineAffinity::Pipelines(&[Pipeline::DogStatsD]),
48+
};
3049
/// `data_plane.otlp.proxy.logs.enabled`-Proxy OTLP logs to Core Agent
3150
DATA_PLANE_OTLP_PROXY_LOGS_ENABLED = SalukiAnnotation {
3251
schema: &schema::DATA_PLANE_OTLP_PROXY_LOGS_ENABLED,

‎lib/datadog-agent/config-testing/src/config_registry/encoders.rs‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ crate::declare_annotations! {
2727
support_level: SupportLevel::Full,
2828
additional_yaml_paths: &[],
2929
env_var_override: None,
30-
used_by: &[structs::DATADOG_EVENTS_CONFIGURATION, structs::DATADOG_LOGS_CONFIGURATION, structs::DATADOG_METRICS_CONFIGURATION, structs::DATADOG_SERVICE_CHECKS_CONFIGURATION, structs::DATADOG_TRACE_CONFIGURATION],
30+
used_by: &[structs::DATADOG_EVENTS_CONFIGURATION, structs::DATADOG_LOGS_CONFIGURATION, structs::DATADOG_METRICS_CONFIGURATION, structs::DATADOG_SERVICE_CHECKS_CONFIGURATION, structs::DATADOG_TRACE_CONFIGURATION, structs::FORWARDER_CONFIGURATION],
3131
value_type_override: None,
3232
test_json: None,
3333
pipeline_affinity: PipelineAffinity::CrossCutting,
@@ -252,4 +252,26 @@ crate::declare_annotations! {
252252
test_json: None,
253253
pipeline_affinity: PipelineAffinity::Pipelines(&[Pipeline::DogStatsD]),
254254
};
255+
/// `use_v3_api.series.enabled`-Global V3 series mode
256+
USE_V3_API_SERIES_ENABLED = SalukiAnnotation {
257+
schema: &schema::USE_V3_API_SERIES_ENABLED,
258+
support_level: SupportLevel::Full,
259+
additional_yaml_paths: &[],
260+
env_var_override: Some(&["DD_USE_V3_API_SERIES_ENABLED"]),
261+
used_by: &[structs::DATADOG_METRICS_CONFIGURATION, structs::FORWARDER_CONFIGURATION],
262+
value_type_override: None,
263+
test_json: None,
264+
pipeline_affinity: PipelineAffinity::Pipelines(&[Pipeline::DogStatsD]),
265+
};
266+
/// `use_v3_api.series.endpoints`-Per-endpoint V3 series modes
267+
USE_V3_API_SERIES_ENDPOINTS = SalukiAnnotation {
268+
schema: &schema::USE_V3_API_SERIES_ENDPOINTS,
269+
support_level: SupportLevel::Full,
270+
additional_yaml_paths: &[],
271+
env_var_override: Some(&["DD_USE_V3_API_SERIES_ENDPOINTS"]),
272+
used_by: &[structs::DATADOG_METRICS_CONFIGURATION, structs::FORWARDER_CONFIGURATION],
273+
value_type_override: None,
274+
test_json: Some(r#""{\"http://datadog.example.com\":\"false\"}""#),
275+
pipeline_affinity: PipelineAffinity::Pipelines(&[Pipeline::DogStatsD]),
276+
};
255277
}

‎lib/datadog-agent/config-testing/src/config_registry/forwarder.rs‎

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ crate::declare_annotations! {
4444
support_level: SupportLevel::Full,
4545
additional_yaml_paths: &[],
4646
env_var_override: Some(&["DD_OBSERVABILITY_PIPELINES_WORKER_METRICS_ENABLED"]),
47-
used_by: &[structs::FORWARDER_CONFIGURATION],
47+
used_by: &[structs::DATADOG_METRICS_CONFIGURATION, structs::FORWARDER_CONFIGURATION],
4848
value_type_override: None,
4949
test_json: None,
5050
pipeline_affinity: PipelineAffinity::CrossCutting,
@@ -55,7 +55,7 @@ crate::declare_annotations! {
5555
support_level: SupportLevel::Full,
5656
additional_yaml_paths: &[],
5757
env_var_override: Some(&["DD_OBSERVABILITY_PIPELINES_WORKER_METRICS_URL"]),
58-
used_by: &[structs::FORWARDER_CONFIGURATION],
58+
used_by: &[structs::DATADOG_METRICS_CONFIGURATION, structs::FORWARDER_CONFIGURATION],
5959
value_type_override: None,
6060
test_json: None,
6161
pipeline_affinity: PipelineAffinity::CrossCutting,
@@ -66,7 +66,7 @@ crate::declare_annotations! {
6666
support_level: SupportLevel::Full,
6767
additional_yaml_paths: &[],
6868
env_var_override: Some(&["DD_VECTOR_METRICS_ENABLED"]),
69-
used_by: &[structs::FORWARDER_CONFIGURATION],
69+
used_by: &[structs::DATADOG_METRICS_CONFIGURATION, structs::FORWARDER_CONFIGURATION],
7070
value_type_override: None,
7171
test_json: None,
7272
pipeline_affinity: PipelineAffinity::CrossCutting,
@@ -77,7 +77,7 @@ crate::declare_annotations! {
7777
support_level: SupportLevel::Full,
7878
additional_yaml_paths: &[],
7979
env_var_override: Some(&["DD_VECTOR_METRICS_URL"]),
80-
used_by: &[structs::FORWARDER_CONFIGURATION],
80+
used_by: &[structs::DATADOG_METRICS_CONFIGURATION, structs::FORWARDER_CONFIGURATION],
8181
value_type_override: None,
8282
test_json: None,
8383
pipeline_affinity: PipelineAffinity::CrossCutting,
@@ -88,7 +88,7 @@ crate::declare_annotations! {
8888
support_level: SupportLevel::Full,
8989
additional_yaml_paths: &[],
9090
env_var_override: None,
91-
used_by: &[structs::FORWARDER_CONFIGURATION],
91+
used_by: &[structs::DATADOG_METRICS_CONFIGURATION, structs::FORWARDER_CONFIGURATION],
9292
value_type_override: None,
9393
test_json: Some(r#"{"smoke-host-1.example.com": ["smoke-api-key"]}"#),
9494
pipeline_affinity: PipelineAffinity::CrossCutting,
@@ -324,4 +324,26 @@ crate::declare_annotations! {
324324
test_json: None,
325325
pipeline_affinity: PipelineAffinity::CrossCutting,
326326
};
327+
/// `observability_pipelines_worker.metrics.use_v3_api.series`-Use V3 series for OPW
328+
OBSERVABILITY_PIPELINES_WORKER_METRICS_USE_V3_API_SERIES = SalukiAnnotation {
329+
schema: &schema::OBSERVABILITY_PIPELINES_WORKER_METRICS_USE_V3_API_SERIES,
330+
support_level: SupportLevel::Full,
331+
additional_yaml_paths: &[],
332+
env_var_override: Some(&["DD_OBSERVABILITY_PIPELINES_WORKER_METRICS_USE_V3_API_SERIES"]),
333+
used_by: &[structs::DATADOG_METRICS_CONFIGURATION, structs::FORWARDER_CONFIGURATION],
334+
value_type_override: None,
335+
test_json: None,
336+
pipeline_affinity: PipelineAffinity::Pipelines(&[Pipeline::DogStatsD]),
337+
};
338+
/// `vector.metrics.use_v3_api.series`-Use V3 series for Vector
339+
VECTOR_METRICS_USE_V3_API_SERIES = SalukiAnnotation {
340+
schema: &schema::VECTOR_METRICS_USE_V3_API_SERIES,
341+
support_level: SupportLevel::Full,
342+
additional_yaml_paths: &[],
343+
env_var_override: Some(&["DD_VECTOR_METRICS_USE_V3_API_SERIES"]),
344+
used_by: &[structs::DATADOG_METRICS_CONFIGURATION, structs::FORWARDER_CONFIGURATION],
345+
value_type_override: None,
346+
test_json: None,
347+
pipeline_affinity: PipelineAffinity::Pipelines(&[Pipeline::DogStatsD]),
348+
};
327349
}

‎lib/datadog-agent/config/schema/core/core_schema.yaml‎

Lines changed: 62 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -449,6 +449,51 @@ properties:
449449
description: Forwarder timeout in seconds
450450
tags:
451451
- template_section:Common
452+
use_v3_api:
453+
node_type: section
454+
type: object
455+
visibility: public
456+
description: Controls the use of the v3 metrics intake.
457+
tags:
458+
- template_section:Common
459+
properties:
460+
series:
461+
node_type: section
462+
type: object
463+
visibility: public
464+
description: Controls the use of the v3 metrics intake for non-distribution
465+
metrics (series).
466+
tags:
467+
- template_section:Common
468+
properties:
469+
enabled:
470+
node_type: setting
471+
type: string
472+
default: 'true'
473+
visibility: public
474+
description: |-
475+
Global default for the non-distribution metrics intake version. Accepted values:
476+
- true: v3 for every destination except Observability Pipelines Worker.
477+
- datadog_only: v3 only when the destination URL is a Datadog endpoint.
478+
- false: v2 everywhere.
479+
tags:
480+
- template_section:Common
481+
endpoints:
482+
node_type: setting
483+
type: object
484+
default: {}
485+
additionalProperties:
486+
type: string
487+
visibility: public
488+
description: |-
489+
Per-URL override map for non-distribution metrics intake version. The URL
490+
must match exactly what was configured under `dd_url` or `additional_endpoints`.
491+
Accepted values:
492+
- true: v3 for every destination except Observability Pipelines Worker.
493+
- datadog_only: v3 only when the destination URL is a Datadog endpoint.
494+
- false: v2 everywhere.
495+
tags:
496+
- template_section:Common
452497
forwarder_retry_queue_payloads_max_size:
453498
node_type: setting
454499
type: integer
@@ -1077,6 +1122,14 @@ properties:
10771122
example: '"http://127.0.0.1:8080"'
10781123
tags:
10791124
- template_section:Common
1125+
use_v3_api:
1126+
node_type: section
1127+
type: object
1128+
properties:
1129+
series:
1130+
node_type: setting
1131+
type: boolean
1132+
default: false
10801133
logs:
10811134
node_type: section
10821135
type: object
@@ -9469,7 +9522,7 @@ properties:
94699522
shadow_sample_rate:
94709523
node_type: setting
94719524
type: number
9472-
default: 0.001
9525+
default: 0
94739526
tags:
94749527
- golang_type:float64
94759528
shadow_sites:
@@ -10005,6 +10058,14 @@ properties:
1000510058
node_type: setting
1000610059
type: string
1000710060
default: ''
10061+
use_v3_api:
10062+
node_type: section
10063+
type: object
10064+
properties:
10065+
series:
10066+
node_type: setting
10067+
type: boolean
10068+
default: false
1000810069
traces:
1000910070
node_type: section
1001010071
type: object

0 commit comments

Comments
 (0)