Evaluate concept for valkey in clustered configuration

[Gerrit91]

Description

We require a Redis-compatible key-value datastore for our metal-apiserver, which is worth to give the backup-restore functionality (it stores API tokens, invites and more). We started with keydb but want to move it to valkey.

The implementation already works in standalone configuration, however supporting a clustered configuration would be beneficial, too, as it provides the opportunity to update without zero-downtime.

With #67 it became possible to at least backup a clustered configuration by detecting the master in the cluster, but the restore mechanism would lead to all replicas restoring at the same time, leading to an uncontrolled state of the database.

Let's approach the issue the following way:

• Add a test for valkey with the current implementation (this also starts to make the valkey helm-chart obsolete, which is currently undergoing some changes due to Bitnami, see Add valkey chart to remove dependency from bitnami helm-charts#131)
• Build up a clustered configuration of valkey for the test and check the backup mechanism is working
• Figure out an approach to complete the implementation
  • One idea that comes to my mind would be to add leader election to the sidecar (https://pkg.go.dev/k8s.io/client-go/tools/leaderelection), then the leader could take over the backup-restore tasks, however the details have to be thought through (e.g. the leader election should consider that the database it runs along with is healthy or leader, in restore phase the members of the quorum need to be certain that they are all running in the init phase, ...)
• Implement the proposal and provide an integration test
• Add this solution to metal-roles

[Gerrit91]

/cc @BotondGalxc
I guess this contradicts a bit with what you are currently doing in the helm-chart repository, even though your work can be utilized for building the manifests for the backup-restore-sidecar.