Meter Network Services edge-proxy usage into the billing pipeline

[kevwilliams]

Tracking enhancement — contributes the Network Services slice of the "Service instrumentation" gate in #681.

Builds on the service-catalog registration completed in network-services-operator #155 (design brief #156, Phase 1 implementation #165).

High-Level Summary

Network Services runs an Envoy Gateway–based edge proxy that routes HTTP traffic, terminates TLS, and enforces WAF/rate-limit policies for customers. As of PR #165 it is registered in the service catalog — a Service and a ServiceConfiguration declare HTTPRoute as the billable resource and four meters under the networking.datumapis.com/gateway/* family (request count, egress bytes, ingress bytes, connection seconds). That is declaration only; no usage is being measured yet.

This enhancement is Phase 2 of that design brief: wire the edge proxy's telemetry into the billing usage pipeline so real consumption flows proxy → Vector → Ingestion Gateway → Billing Consumer → Amberflo, attributed to a BillingAccount. It closes the loop between "we declared meters for Network Services" and "Network Services usage shows up in Amberflo and the portals" — exactly the end-to-end metering proof #681 is still missing.

This phase is measurement only, consistent with #681: no rates, no invoices, no charging.

Motivation

• Billing and Metering for Datum Cloud using Integration of 3rd Party Metering Service #681 is blocked on real service instrumentation. Its "End-to-end metering proof" and "Service instrumentation" gates need actual services emitting CloudEvents. Compute and AI Assistant are wired; Network Services (which underpins AI Edge, DNS, Tunnels, gVPCs ingress) is declared but silent.
• Catalog declarations are inert without an emission path. PR View Datum Cloud Users #165 shipped meters in Draft phase. Meter meterName and measurement.unit become immutable at Published — we want to validate them against real traffic before promotion, which requires emitting events now.
• Retrofitting metering is expensive. The four billing signals exist today only as operational Prometheus counters with no billing attribution. Getting the collection path right while the meters are still Draft costs a few PRs; getting it wrong after promotion costs a new meter family + SDK change + customer migration.
• The design investigation is already done. PR Project Settings Page #156 evaluated three collection approaches and landed on a recommendation (Vector access-log scraping + a controller-side connection emitter). This enhancement is about executing that recommendation and resolving its remaining open decisions — not re-opening the design.

[kevwilliams]

Current status of this is in design with P1 awaiting approval. I would like to deliver this in the next 2 weeks.

Design: datum-cloud/network-services-operator#156
Phase 1 PR: datum-cloud/network-services-operator#165

[JoseSzycho]

@kevwilliams I can go ahead and make the work for start metering the usage from the Envoy Gateway if you are okay with

[kevwilliams]

Yes @JoseSzycho feel free - You can take the two PRs I linked above and run with them or do whatever you like! Let me know if I can help.

[JoseSzycho]

@kevwilliams Great! Thank you, will ping you if I have any questions.

[github-actions]

Issues must have exactly one assignee, but 2 were assigned (@kevwilliams, @JoseSzycho). All assignees have been removed. Please assign a single owner.

[JoseSzycho]

@kevwilliams @scotwells as per the metering design, seems that is is the chosen option for extracting the billing signals:

• https://github.com/datum-cloud/network-services-operator/blob/feat/service-catalog-registration-design/docs/enhancements/service-catalog-registration.md#option-a-access-log-scraping-via-vector-agent-recommended

Are we certain that this is the approach we intend to adopt? I'm really worried about the huge quantity/volume of logs we will need to process. And probably, we will probably need to add an enrichment layer for that.

I'll wait for your response before proceeding with the implementation investigation. In the mean time, I will continue my studies on understanding how envoy exposes the information we need.

[JoseSzycho]

Being able to meter:

• networking.datumapis.com/gateway/requests
• networking.datumapis.com/gateway/egress-bytes
• networking.datumapis.com/gateway/ingress-bytes
• networking.datumapis.com/gateway/connection-seconds

Metering these signal have been tested in a local environment:

• Feat/envoy logs network-services-operator#188

Right now, waiting for the enhancement documentation to be approved so we can proceed with an actual implementation for staging, and then production.

• docs: refactor service registration brief to HTTP traffic metering network-services-operator#181