Elasticsearch timestamps do not match exactely the poll interval

[bzizou]

Hello,

I have some trouble making histogram graphs of the cpu data uploaded by the oar plugin to the elasticsearch plugin (via an alumet relay) using the timestamp values. For example, if we want the load of a given node, we need to group data of a given interval as a sum of the values in this time interval, and then, we need the values of a given interval to have the same timestamp.

Given this configuration:

[plugins.oar]        
oar_version = "oar3" 
jobs_only = true     
poll_interval = "5s" 

The nodes upload metrics every 5s, but the timestamps show that we have more intervals

Furthermore, it seems that we have 2 values for the same job, with 0.2 to 0.5s delay, one with the slice ans one with the scope

Is the @timestamp value set by alumet when it collects the data or is it the automatic timestamp given by elasticsearch when recording the data? If it is not the case, I think that alumet should set the @timestamp value to the date of the start of the interval of the record

[TheElectronWill]

I think that the @timestamp sent to elastic is the one obtained when the measurement source produces a data point (see code). Each cgroup gets a separate source (created when the cgroup is detected) that gets triggered at poll_interval.

If I look at one particular cgroup, let's say the slice for job 607303, I have:

The interval seems ok?

[bzizou]

Yes, okay... Well, so, playing a bit more with grafana, I finally found that using back cpu_time_delta is more reliable than cpu_percent because (I think) it's normal to sum such data : the sum in an interval, for a fully loaded node should give 1s and it is actualy the case. But the sum of cpu_percent is not 100%... Maybe this is not related to timestamps, but if each cgroup has it's own interval, it seems logical that we can't get a sum that is 100%, as the measurement is not made at the same time for each cgroup of a node

[TheElectronWill]

That's strange, because cpu_time_delta and cpu_percent are computed side by side, at the same time (so it's still not synchronized between cgroups) 🤔
But I see that it would be better to synchronize that, taking note for a future version :)