Investigate running TLS servers in a FIPS-compliant mode when FIPS support is enabled.

[tobz]

Context

We currently handle FIPS compliance with client TLS configurations: if FIPS is enabled, check that the crypto provider is operating in FIPS mode, and so on. However, we don't do that for server TLS configurations.

We should.

[tobz]

(Autogenerated evaluation / plan from Bits Code just to try and seed the planning / discussion)

Current state

• The process-wide Rustls provider is initialized through saluki_tls::initialize_default_crypto_provider(), called during bootstrap, so Rustls builders normally use AWS-LC before clients/servers are created: lib/saluki-app/src/tls.rs#L16-L18 and lib/saluki-tls/src/lib.rs#L375-L405.
• Regular outbound HTTP clients already flow through ClientTLSConfigBuilder::build(), which checks ClientConfig::fips() when the fips feature is enabled: lib/saluki-tls/src/lib.rs#L319-L369 and lib/saluki-io/src/net/client/http/client.rs#L303-L310.
• IPC TLS is a separate path and bypasses that client builder. build_ipc_client_ipc_tls_config() and build_ipc_server_tls_config() construct Rustls configs directly: lib/datadog-agent/commons/src/ipc/tls.rs#L76-L133.
• The privileged API endpoint uses the IPC server TLS config, then passes it into the dynamic HTTP/gRPC API server: bin/agent-data-plane/src/internal/control_plane.rs#L52-L57. This covers the standalone/privileged HTTPS server path as well as normal ADP operation.
• The generic HTTP server is the central TLS acceptor point for APIBuilder and DynamicAPIBuilder: lib/saluki-io/src/net/server/http.rs#L102-L109. Higher-level self-signed server helpers also build ServerConfig directly in lib/saluki-app/src/api.rs#L106-L119 and lib/saluki-app/src/dynamic_api.rs#L146-L157.

Proposed implementation plan

1. Add shared FIPS validation helpers in lib/saluki-tls/src/lib.rs:

   • ensure_client_config_fips_compliant(&ClientConfig) -> Result<(), GenericError> to reuse the existing client-side check outside ClientTLSConfigBuilder.
   • ensure_server_config_fips_compliant(&ServerConfig) -> Result<(), GenericError> to mirror the client check with ServerConfig::fips() under #[cfg(feature = "fips")] and no-op otherwise.
   • This keeps the policy in saluki-tls, close to provider initialization and existing FIPS behavior.

2. Apply the helper to all current direct Rustls config construction paths:

   • lib/datadog-agent/commons/src/ipc/tls.rs
     • Check the IPC client config explicitly, because it bypasses ClientTLSConfigBuilder.
     • Check the IPC server config explicitly before returning it.
     • Consider changing the IPC server builder to ServerConfig::builder_with_protocol_versions(&[&TLS13]) to match the IPC client’s TLS 1.3-only configuration. This appears behavior-preserving for current IPC clients and makes the intended FIPS-friendly protocol surface explicit.
   • lib/saluki-app/src/api.rs and lib/saluki-app/src/dynamic_api.rs
     • Route with_self_signed_tls() through the shared server check, or add fallible try_with_self_signed_tls() variants and keep the existing methods as convenience wrappers.
   • lib/saluki-io/src/net/server/http.rs
     • Add a central guard before constructing TlsAcceptor, after the server mutates ALPN protocols. This protects any future caller that passes a ServerConfig into the lower-level HTTP server without using the higher-level helpers.

3. Keep API/auth semantics unchanged.

   • This work should only validate/com constrain TLS configuration. It should not alter the privileged API authentication model or the existing CLI behavior that connects to the privileged API with danger_accept_invalid_certs(): bin/agent-data-plane/src/cli/utils.rs#L50-L92.
   • One thing surfaced during review: IPC comments describe mutual TLS, but the server builder currently uses with_no_client_auth() (lib/datadog-agent/commons/src/ipc/tls.rs#L124-L126). I would treat that as a separate follow-up unless this issue intentionally includes changing client-auth behavior.

Feasibility and risks

• Feasibility: high. The Cargo feature chain already enables Rustls FIPS support for the binary via agent-data-plane/fips -> saluki-app/tls-fips -> saluki-tls/fips -> rustls/fips: bin/agent-data-plane/Cargo.toml#L11-L14, lib/saluki-app/Cargo.toml#L11-L12, lib/saluki-tls/Cargo.toml#L11-L13.
• Runtime overhead should be negligible: validation happens once per config/server start, not per handshake.
• Main risk: the self-signed helpers use rcgen defaults. If those generated cert/key parameters do not satisfy Rustls FIPS checks, we should either select an explicit FIPS-acceptable algorithm/key type or make the failure message clear. This path is mostly development/test-oriented.
• Secondary risk: making IPC server TLS 1.3-only should be checked against any non-ADP consumers, even though the in-repo IPC client is already TLS 1.3-only.

Testing plan

• Add unit tests in saluki-tls for the new validation helpers. In FIPS builds, assert known-good client/server configs pass; if Rustls gives us an easy non-compliant construction path, assert it fails with an actionable error.
• Add/update tests around build_ipc_client_ipc_tls_config() and build_ipc_server_tls_config() using a temporary PEM cert/key fixture so both direct IPC config paths are covered.
• Add a targeted HTTP server test, likely at lib/saluki-io/src/net/server/http.rs or via DynamicAPIBuilder, to ensure TLS-enabled serving still works after the central validation guard.
• Run targeted checks first:
  • cargo check -p saluki-tls --features fips
  • cargo check -p datadog-agent-commons
  • cargo check -p agent-data-plane --features fips
  • targeted cargo nextest run for saluki-tls / datadog-agent-commons / relevant HTTP server tests
• Finish with repository-standard validation for Rust changes: make fmt, then make check-all if time permits.