Monitor every backup deployed in the same namespace

[lenglet-k]

Summary

As an SRE,
I want to “monitor every backup deployed in the same namespace”
in order to “create a Prometheus rule that detects failed backups.”

Background

Our environment deploys multiple schedules in the same namespace, and each schedule backs up a PostgreSQL database.

In Prometheus, we cannot differentiate between each backup task because the instance label is mapped to the namespace where the task is located. As a result, all our K8up metrics look the same, and worse still, the latest metric overwrites the previous ones, and we cannot add additional labels. It seems sensible to add a schedule or backup task name to the metrics label in order to quickly identify the backup that has failed.

Furthermore, the operator has no metrics indicating that the backup has failed.

Implementation ideas

For restic, grouping metrics by schedule or Backup name or both.

Exemple:

k8up_backup_restic_last_errors{namespace="namespace", cluster="mycluster", backup="backupJob"} 1 

For operator add new metrics:

k8up_backup_failed{namespace="namespace", cluster="mycluster",  schedule="schedule", backup="backupJob"} 1
k8up_backup_succeeded{namespace="namespace", cluster="mycluster", schedule="schedule", backup="backupJob"} 1

[arska]

Thanks for the suggestion! Monitoring all backups in a namespace is currently possible via Prometheus metrics (k8up_jobs_* counters with namespace label). A dedicated monitoring CRD is a bigger scope change. Keeping open.

[lenglet-k]

Thanks for the suggestion! Monitoring all backups in a namespace is currently possible via Prometheus metrics (k8up_jobs_* counters with namespace label). A dedicated monitoring CRD is a bigger scope change. Keeping open.

Hello, except that the metric you're referring to doesn't clearly identify which job is failing. When we have multiple different backups, we need to be able to easily identify the job that's failing.