Evaluate the pros and cons of using metrics from kube-state-metrics

[naved001]

Before doing any other work related to kube-state-metrics, let's evaluate it first.

Why are we looking at kube-state-metrics?

• The current metrics kube_pod_resource_request emitted by kube-scheduler does not reliably capture short lived pods (less than 30 seconds).

• Since we rely on the timestamps in our query output to determine start and end time of pods, when we collect metrics at 1 minute interval, the shortest time we can bill for 1 is minute. It's possible to to capture metrics at 30 seconds but then it uses even more data.

Pros of kube-state-metrics

• Captures all pods - including short lived pods.
• Gives us an actual start time timestamp.
• Give us pod UIDs (currently kube-scheduler metrics only use the pod name as a unique identifier).

Cons of kube-state-metrics

No effective pod resource request

Resource requests are provided for each container and initcontainer instead of at the pod level.

The Pod's effective request/limit for a resource is the higher of:

• the sum of all app containers request/limit for a resource
• the effective init request/limit for a resource

If we were to use kube-state-metrics for billing:

• We will need to capture resource requests for all containers and initcontainers
• Based on the pod UIDs, calculate the effective resource request for a pod from all the captured containers and initcontainers.
• Get the timestamp for all the containers/initcontainers and get the min/max of the timestamps to get the start/end for the pod.

Getting pod/container end time is unclear

• @computate has looked into this, but it's unclear why there's no straightforward way to get the pod completion time.

[naved001]

I am biased but I like the custom scripts I wrote which I think we can use specifically for billing short-lived pods (or for everything).

The current mechanism for billing by using the metric from kube-scheduler is still pretty solid for most of the workloads we see except for the short lived pods.

Looking for your thoughts @knikolla @schwesig @computate @larsks

[larsks]

I don't really know enough about KSM to have an opinion. Certainly your custom scripts are relatively straightforward and thus easy to reason about. On the other hand, there are advantages to building on top of something that appears to have an active development community behind it.

And just a note on your bullet list...

• We will need to capture resource requests for all containers and initcontainers
• Based on the pod UIDs, calculate the effective resource request for a pod from all the captured containers and initcontainers.
• Get the timestamp for all the containers/initcontainers and get the min/max of the timestamps to get the start/end for the pod.

...those all seem like they should be relatively simple SQL queries.

[naved001]

...those all seem like they should be relatively simple SQL queries.

PromQL Queries actually. Not saying that it's necessarily difficult, it's additional work (compared to the current metrics we collect) that the billing code will have to do and I just wanted to lay it out.

[morantara]

@naved001 do you think you can get the input you need before the end of this sprint? should it be pushed to a future sprint?

[schwesig]

@schwesig @computate

[computate]

@naved001 If you think it would be useful to produce our own new persistent metrics to replace metrics with your service like when a pod is completed, I am building a persistent metrics collector with zookeeper here:
https://github.com/nerc-images/persistent-metrics-zookeeper

[schwesig]

Short version of my opinion

• Using kube-state-metrics (k-s-m) as the only source for billing is maybe not enough,
  but
• as an additional source (a hybrid approach) it is really very useful — specially for short-living pods and also for correct pod identities (UIDs).
• In the long run we should take the requests/limits always from the Kubernetes state, but not trust it alone for the runtime informations (start/end/duration).
• two measures to compare and detect errors and missing parts

1. Big YES for your k-s-m approach
2. needed, even if not all details are figured out yet
3. in additon with the kube_pod_resource_request, etc and our other metrics, we can create a stable system?!
   1. @larsks queries combines these, yes
   2. @naved001 if the extra work is justified, depends on the amolunt of pods we may miss (not to be billed). Do I remember right you calculated that at some point?

---

some thoughts

• k-s-m helps a lot with short-living pods

  • batch / CI / functions / GPU jobs are often missed by kube_pod_resource_request
  • billing depends strong on catching these
  • k-s-m gives:
    • real Pod UID
    • start timestamp
    • very short running pods
    • full requests/limits (even per container)

• but k-s-m has also some blind spots for billing

  • no real runtime information
  • no actual container start/stop
  • no crash or retry details
  • no init-container duration
  • end timestamp is not reliable (must be guessed from status/absence)

• therefore a hybrid approach looks more correct

  • Requests/Limits → from kube-state-metrics
  • Start/End/Runtime → from events or kubelet/container-runtime metrics

• so: k-s-m for the state, and runtime comes better from the lower-level sources for proper accounting

---

?

Information	Best Source
CPU/GPU/Mem Requests	kube-state-metrics
CPU/GPU Limits	kube-state-metrics
Pod UID	kube-state-metrics
Pod Start Time	events or k8s API
Pod End Time	events or kubelet metrics
Runtime	kubelet (container) or events

[naved001]

@schwesig thanks for your comments! Just some feedback/comments.

if the extra work is justified, depends on the amolunt of pods we may miss (not to be billed). Do I remember right you calculated that at some point?

Correct, right now while we are missing some short lived pods (at least on the academic cluster), the overall impact is negligible. This will change if the number of such short lived pods went up significantly . So, for now I've put any more development on hold for tackling this problem of short-lived pods.

batch / CI / functions / GPU jobs are often missed by kube_pod_resource_request

Just to be clear, there's nothing special about the GPU jobs. Just any pod that runs for less than 30 seconds may be missed.

full requests/limits (even per container)

Just to be clear, this is a bit of drawback as it requires us to compute the effective resource request. kube_pod_resource_request gives us that effective number and we don't have to worry about individual containers runtime and request, we just look at the pod as a whole.

no real runtime information

What do you mean by that? It's not clear to me

no crash or retry details

I guess we don't care about that as long as the pod is scheduled?

[schwesig]

@naved001 thanks for the feedback! 🫶

1. Ack — fully agree, second that.
2. Ack (this one is on me) — you’re right: I didn't fully think about that/I wasn’t clear enough there.
3. I think we should have a quick call to go through the details together, if possible?
4. (this one is on me, not clear enough in text or in my head yet 🫤) — Trying to clarify what I meant (call follow up?): I was thinking about clearly separating timing (start/stop, queue time vs GPU time, dev pod hand over to work pod timing, in general traces) and being able to close that flow back to the original dev pod. Sorry if that came across confusing — I was loosely referring to this discussion/solution:

• feat(OPE): Deep dive, status, and design of batch job timing metrics for fine-grained GPU course workloads nerc-project/operations#1336 (comment)

5. Good point on failures — I’m not fully sure yet. If a pod crashes due to infra issues, whether that counts toward a job/customer feels more like a billing/policy decision than a technical one.

Happy to sync live where needed and update the issue after.