Skip to content

feat(metrics): add opt-in TLS and mTLS support - #3877

Open
gmenher wants to merge 10 commits into
NVIDIA:mainfrom
gmenher:codex/secure-metrics-runtime
Open

gmenher wants to merge 10 commits into
NVIDIA:mainfrom
gmenher:codex/secure-metrics-runtime

Conversation

@gmenher

@gmenher gmenher commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Add opt-in TLS and mutual TLS support for OpenShell's dedicated metrics listener while preserving plaintext metrics as the default for compatibility.

This is the first implementation PR for #3664. It intentionally focuses on the runtime listener, operator-provided Secrets, and the metrics trust boundary; chart-managed metrics PKI, monitoring resources, and metrics ingress policy configuration remain follow-up work.

Related Issue

Refs #3664

Changes

  • Add dedicated metrics TLS configuration through gateway TOML, CLI flags, and environment variables.
  • Serve /metrics over HTTPS when metrics TLS is configured, with optional client-certificate authentication.
  • Keep the metrics client CA independent from the gateway client CA, and reject Helm configurations that reuse the effective gateway client CA Secret.
  • Mount operator-provided server TLS and metrics client CA Secrets read-only through the Helm chart.
  • Reuse the existing TLS reload machinery while identifying metrics-specific reload logs and OCSF events with listener=metrics.
  • Add actionable structured warning logs for rejected metrics TLS handshakes without logging certificate contents or subjects.
  • Document the runtime configuration, Helm values, trust boundary, and secure scraper requirements.
  • Add runtime, configuration, Helm rendering, trust-boundary, and tracing-log coverage.

Testing

  • cargo fmt --check
  • cargo test -p openshell-core metrics_tls --lib
  • cargo test -p openshell-server metrics_tls --lib
  • cargo test -p openshell-server tls_metrics_listener_serves_https_and_rejects_plaintext --lib
  • cargo test -p openshell-server metrics_mtls_trusts_only_the_metrics_client_ca --lib
  • cargo test -p openshell-server metrics_tls_handshake_rejection_logs_actionable_context --lib
  • helm unittest deploy/helm/openshell -f tests/gateway_config_test.yaml (107 tests)
  • helm lint deploy/helm/openshell --set agentSandbox.preflight.enabled=false
  • mise run helm:docs:check
  • Manual isolated Kubernetes validation: HTTP, anonymous HTTPS, gateway-client certificate, and wrong CA were rejected; a dedicated metrics client certificate returned 200 OK.
  • mise -j 1 run ci completed all relevant checks but failed in the unrelated openshell-supervisor-network test proxy::tests::mediated_connect_keeps_workload_bytes_read_with_the_synthesized_header, which times out waiting for its upstream. The failure reproduces in isolation; this PR does not modify that crate or its proxy path.

Checklist

  • Follows Conventional Commits.
  • Commits are signed off (DCO).
  • Architecture and deployment documentation updated.

@copy-pr-bot

copy-pr-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@gmenher
gmenher force-pushed the codex/secure-metrics-runtime branch from 42b63fb to e0485f1 Compare September 29, 2026 18:27
gmenher added 10 commits October 1, 2026 10:37
Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>
Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>
Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>
Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>
Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>
Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>
Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>
Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>
Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>
Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>
@gmenher
gmenher force-pushed the codex/secure-metrics-runtime branch from e0485f1 to b4b7be9 Compare October 1, 2026 09:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant