agent-payload

Payload format description for communication between the Agent and the Datadog backend.

This repository includes the protocol-buffer IDL used by the agent6 and agent7 to communicate with the Datadog backend. Those payloads are only supported by the V2 API endpoints. The generated Go, Java, and Rust implementations are checked into this repository and can be used directly. Other consumers may copy the .proto files into their repository and generate their own bindings.

Use

• Create a virtualenv for your python runtime
• Install invoke from the requirements.txt
• Run the codegen.all task

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
inv codegen.all

Prerequisites

You will need

• Python (3+, CI builds with 3.12)
• Go (at least the version in go.mod)
• A checkout of this repository within a GOPATH (so, at $GOPATH/src/github.com/DataDog/agent-payload)
• Rust, only if you're working on metrics/dd-metrics-v3/ — install the toolchain version pinned in rust-toolchain.toml.

Payloads

Logs

The logs payload is defined in proto/logs/agent_logs_payload.proto. The following implementations are available:

• Go (gogofast): github.com/DataDog/agent-payload/pb
• Java: java/com/dd/agent/pb/Log.java

Metrics

The metrics payload is defined in proto/metrics/agent_payload.proto. The following implementations are available:

• Go (gogofast): github.com/DataDog/agent-payload/gogen

Metrics V3

The V3 metrics payload is defined in proto/metrics/intake_v3.proto. It uses a columnar layout with dictionary-based string deduplication instead of one message per time series. The following implementations are available:

• Go (protoc-gen-go): github.com/DataDog/agent-payload/v5/metrics/intake_v3.
• Rust: dd-metrics-v3, a hand-rolled no_std encoder.

Process

The process payload is defined in proto/process/agent.proto. The following implementations are available:

• Go (gogofast): github.com/DataDog/agent-payload/process (note that this go package contains additional functionality beyond the generated PB implementation).

CWS

The CWS security dumps payload is defined in proto/cws/dumpsv1/activity_dump.proto. The following implementations are available:

• Go (vtproto): github.com/DataDog/agent-payload/v5/cws/dumpsv1
• Java: java/com/dd/cws/adv1/pb/SecDumpProto.java

Agent Health

The Agent Health payloads are defined in proto/healthplatform/healthplatform.proto. The following implementation is available:

• Go (protoc-gen-go): [github.com/DataDog/agent-payload/healthplatform]

Updating Proto Definitions

After updating the IDL you must:

• Regenerate the code: inv codegen.all, invoke will use gimme to run the task command with the current defined go version
• If you have indentation/newlines changes, run inv codegen.all with the same Go version as defined in go.mod
• If you changed proto/metrics/intake_v3.proto, also regenerate metrics/dd-metrics-v3/tests/pb/mod.rs (cargo build --features generate-protobuf from within metrics/dd-metrics-v3/) and update the hand-rolled encoder in metrics/dd-metrics-v3/src/ to match, since it doesn't use a Protocol Buffers library
• Create a new tag with the updated version of the payload

Publishing Changes

After merging changes to master create a release by:

1. Navigate to the Releases page

2. Click "Draft a new release"

3. In the "Choose a tag" drop down, type in the next version number

   Generally you can add one to the last version number. Make sure to include the v prefix. For example, if the last release was v5.0.37, your release should be v5.0.38.

4. The release title should be the same as the version tag

5. Use "Generate release notes" to fill in the release description

6. Click "Publish release"

   This will create a git tag that can now be referenced in other repos.