feat: Add Valkey Sentinel & HAProxy support in High Availability setup. by khtee · Pull Request #137 · valkey-io/valkey-helm

khtee · 2026-02-06T08:56:32Z

Feat #22

dmaes · 2026-02-11T13:03:11Z

Nice work!

I have some notes/questions:

You'll want to disable the default statefulset when sentinel is enabled.
Do you plan on adding a HAProxy deployment, so clients don't have to be aware of sentinel? ( See for example https://arystech.com/blog/setting-up-haproxy-with-redis-sentinel-for-high-availability-on-microk8s-kubernetes )

khtee · 2026-02-11T13:07:44Z

Nice work!

I have some notes/questions:

You'll want to disable the default statefulset when sentinel is enabled.

Do you plan on adding a HAProxy deployment, so clients don't have to be aware of sentinel? ( See for example https://arystech.com/blog/setting-up-haproxy-with-redis-sentinel-for-high-availability-on-microk8s-kubernetes )

Good points! Will work on both improvements.

yoannrt · 2026-02-11T15:17:39Z

Good job,

I have some questions too,
I assume to discover who is master, you have to ask sentinel SENTINEL GET-MASTER-ADDR-BY-NAME mymaster ?
(Currently, the replica service is sticked to pod-0 in the STS)

Also, would that make sense to run sentinel as a side container in the replica STS pods ?

amontalban · 2026-02-13T22:47:56Z

Thank you @khtee can't wait to get this merged in 🙏

dmaes · 2026-02-14T06:50:58Z

Also, would that make sense to run sentinel as a side container in the replica STS pods ?

You'll want to disable the default statefulset when sentinel is enabled

I got confused, thinking valkey-sentinel also runs the server process, it's been a while since I used redis/valkey. But yes, @khtee you'll want to run sentinel and the server side-to-side in the same pod, don't split them out in different statefulsets.

yoannrt · 2026-02-17T15:38:51Z

Do you plan on adding a HAProxy deployment, so clients don't have to be aware of sentinel?

@dmaes I guess when you suggest to implement HAproxy, it's for sentinel incompatible clients workloads ?
So this would be optional and the default implementation would be a valkey/sentinel HA setup ?

khtee · 2026-02-19T06:14:04Z

Also, would that make sense to run sentinel as a side container in the replica STS pods ?

You'll want to disable the default statefulset when sentinel is enabled

I got confused, thinking valkey-sentinel also runs the server process, it's been a while since I used redis/valkey. But yes, @khtee you'll want to run sentinel and the server side-to-side in the same pod, don't split them out in different statefulsets.

Added following enhancements.

Sentinel now runs as a sidecar in valkey pod.
HAProxy with sidecar watcher to track failover.

dmaes · 2026-02-19T07:58:51Z

Do you plan on adding a HAProxy deployment, so clients don't have to be aware of sentinel?

@dmaes I guess when you suggest to implement HAproxy, it's for sentinel incompatible clients workloads ? So this would be optional and the default implementation would be a valkey/sentinel HA setup ?

That's correct.

The truly kubernetes-native way would probably be to have a sentinel-master Service, using a sentinel.valkey.io/master: "true" label selector, and then some watcher that updates that label on the correct Pod,
but a HAproxy deployment is the easier option to implement, and is how most other Redis charts do it.

Signed-off-by: KHTee <teekahhui@hotmail.com>

Co-authored-by: Dieter Maes <dieter.maes@dmaes.be> Signed-off-by: khtee <75174583+khtee@users.noreply.github.com>

Signed-off-by: KHTee <teekahhui@hotmail.com>

Allow HAProxy to retry DNS resolution during startup when pending for Valkey node to start. Essentially it does the following - Try to use the last known IP. - If none, query the libc resolver (DNS). - If that fails, resolve to none (meaning the server has no IP address yet, but HAProxy won't crash) and wait for the runtime resolver health-checks to pick up the DNS correctly. Signed-off-by: KHTee <teekahhui@hotmail.com>

lazariv · 2026-03-02T12:50:57Z

Waiting for this to be merged. Thanks for your work!

- Updating HAProxy watcher for near-instant IP-based failover. - Refactoring init scripts to support dynamic topology and universal auth/TLS injection. - Adding smart L7 health checks in HAProxy to handle ACL-protected nodes. - Fully parameterizing Service and ConfigMap ports for end-to-end flexibility.

…notations in haproxy-deployment

…schema Valkey Sentinel: Fixing Auth/ACL gaps in TLS environments and security hardening (Enhancements for PR valkey-io#137)

yoannrt · 2026-04-09T22:49:50Z

@@ -188,3 +188,34 @@ Validate replica authentication configuration
 {{- end }}
 {{- end -}}



Suggested change

{{/*

Validate haproxy is used in replica mode

*/}}

{{- define "valkey.validateHaproxyRequirements" -}}

{{- if and .Values.haproxy.enabled (not .Values.replica.enabled) }}

{{- fail "Haproxy is only relevant in replica mode with clients incompatible with Sentinel." }}

{{- end }}

{{- end -}}

update of deploy_valkey.yaml to fail if haproxy is enabled in standalone mode.

Suggested change

{{- include "valkey.validateHaproxyRequirements" . }}

lazariv

Redis/Valkey pub/sub (SUBSCRIBE) connections are long-lived and idle by design — they block waiting for messages. HAProxy's timeout client/timeout server treats them as stale and drops them, causing clients to see ConnectionError: Connection closed by server and forcing reconnect loops.

Adding timeout tunnel to the HAProxy defaults section solves this cleanly. In HAProxy, timeout tunnel governs bidirectional connections after the initial handshake — exactly the pattern pub/sub uses. Setting it to 0 keeps SUBSCRIBE connections alive indefinitely while preserving normal client/server timeouts as a safety net for regular command connections.

Without this, users running any pub/sub workload (Socket.IO, Celery, Sidekiq, etc.) through HAProxy must either set client/server timeouts to 0 (triggering HAProxy warnings) or accept periodic disconnects.

lazariv · 2026-04-13T11:21:38Z

+      log global
+      timeout connect {{ .Values.haproxy.config.timeout.connect }}
+      timeout client {{ .Values.haproxy.config.timeout.client }}
+      timeout server {{ .Values.haproxy.config.timeout.server }}


Suggested change

timeout server {{ .Values.haproxy.config.timeout.server }}

timeout server {{ .Values.haproxy.config.timeout.server }}

timeout tunnel {{ .Values.haproxy.config.timeout.tunnel }}

option clitcpka

option srvtcpka

lazariv · 2026-04-13T11:23:05Z

+                    },
+                    "server": {
+                      "type": "string"
+                    }


Suggested change

}

},

"tunnel": {

"type": "string"

}

lazariv · 2026-04-13T11:24:30Z

+    timeout:
+      connect: 5s
+      client: 1m
+      server: 1m


Suggested change

server: 1m

server: 1m

# Timeout for long-lived bidirectional connections (e.g. pub/sub).

# Set to 0 to keep pub/sub SUBSCRIBE connections alive indefinitely.

tunnel: 0s

jose-10000 · 2026-04-27T06:56:55Z

I've implemented a preStop hook. I found that improper shutdowns during updates were tripping up Sentinel and risked data loss, so this change ensures a more graceful exit. I made a new pull request @khtee

jose-10000 · 2026-04-30T11:16:04Z

I've added the workloadAnnotations block to haproxy-deployment.yaml. I noticed that the upstream main branch recently introduced workloadAnnotations for other workloads, so adding it here ensures HAProxy is aligned with main once this PR is eventually merged.

Do you have a rough ETA for this merge? Please let me know if there is anything else I can help review or test to get this PR over the finish line. I'm planning to adopt these changes soon and would like to have an idea of when it might be merged. Thanks!

jose-10000 · 2026-05-18T07:02:40Z

I have some reservations about the approach of running Sentinel as a sidecar within the same pod as the Valkey instance.

The main concern is related to Helm upgrade workflows: when a pod is taken down during a rolling update or a Helm upgrade, the entire pod is terminated — including both the Valkey container and the Sentinel sidecar. This means we lose a Sentinel node during the upgrade process.

Losing a Sentinel during an upgrade can temporarily break the quorum. For example, in a typical 3-Sentinel setup, losing one during an upgrade leaves only 2 Sentinels, which may still meet quorum — but in smaller or less redundant setups, this can prevent failover from working correctly during the exact moment it may be needed most.

A more resilient architecture would deploy Sentinels as independent pods (e.g., via a separate StatefulSet), decoupled from the Valkey data pods. This ensures that Sentinel availability is not tied to the lifecycle of the data nodes, preserving quorum integrity throughout rolling updates and upgrades.

dmaes · 2026-05-18T08:18:06Z

@jose-10000 across all other best known redis sentinel helm charts, running in the same pod is the most common setup, and has been working just fine for most people. So I (and I think that goes for others too), never really considered using different statefulsets?

jose-10000 · 2026-05-18T08:43:45Z

@jose-10000 across all other best known redis sentinel helm charts, running in the same pod is the most common setup, and has been working just fine for most people. So I (and I think that goes for others too), never really considered using different statefulsets?

Thanks for the context, @dmaes! Totally makes sense, I know most charts (like Bitnami) do it this way to keep things simplier. I just wanted to point out that coupling their lifecycles can be tricky for quorum during upgrades, but I'm fine keeping the current approach. Just wanted to share my two cents

trabelsieyal · 2026-05-20T14:47:18Z

Any idea when will this be released? can be amazing

mlimardo1984 · 2026-06-04T14:41:21Z

Any idea when will this be released? can be amazing

Hello Team, just want to reiterate here that implementation is great and we are very much looking forward to have it released. Is there any news about it?

Thanks!

jsoref · 2026-06-09T17:42:12Z

@khtee this repository appears to expect DCO sign-offs, and unfortunately it looks like your fork didn't enforce that when you merged @jose-10000's Valkey Sentinel: Fixing Auth/ACL gaps in TLS environments and security hardening (Enhancements for PR #137) khtee/valkey-helm#1 -- I'm not sure how to fix this...

Nils98Ar · 2026-06-10T21:35:26Z

@khtee this repository appears to expect DCO sign-offs, and unfortunately it looks like your fork didn't enforce that when you merged @jose-10000's Valkey Sentinel: Fixing Auth/ACL gaps in TLS environments and security hardening (Enhancements for PR #137) khtee/valkey-helm#1 -- I'm not sure how to fix this...

@khtee You could add missing sign-offs in the last e.g. 10 commits via commit amend like this and force push:

git rebase HEAD~10 --exec 'git log -1 --format=%B | grep -qi "^Signed-off-by:" || git commit --amend -s --no-edit'
git push --force-with-lease

geodimm · 2026-06-11T15:54:28Z

            - name: {{ $key }}
              value: "{{ $val }}"


Isn't it even cleaner to do this instead? #133

kaisen-san · 2026-06-18T16:07:49Z

Just FYI, besides the valid review comments people left on this PR, which I think should be taken into consideration, it also needs to be rebased, and some tests are failing. I also ran some local analysis and noted a few things that could be improved – I'll post the findings in another comment. I'm saying all this because I've noticed some people are expecting this to be merged, but I'm not sure if that will happen anytime soon (hope I'm wrong! 🙂)

yoannrt · 2026-06-18T19:20:16Z

across all other best known redis sentinel helm charts, running in the same pod is the most common setup, and has been working just fine for most people. So I (and I think that goes for others too), never really considered using different statefulsets?

Hi @dmaes , I thought too running sentinel as a sidecar was the simplest/cleaner option but I'm reconsidering it since having those components in 2 different sts has some advantages.
@khtee I think you were in the right way from the beginning so I sent you a PR, see khtee#4 . Thanks again

pat-s · 2026-07-28T07:33:06Z

@daanvinken @Bloodraven21
Pinging you here as you've been most active lately in committing to main.

It would really be great to finalize this feature (in here or a separate clean branch) so this can finally find it's way in. I am also happy to clean up if there's a commitment to review and get this merged then, avoiding additional future conflicts.

Looking forwarding hearing back from you and getting HA support into the chart!

kaisen-san · 2026-07-28T14:01:08Z

@daanvinken @Bloodraven21 Pinging you here as you've been most active lately in committing to main.

It would really be great to finalize this feature (in here or a separate clean branch) so this can finally find it's way in. I am also happy to clean up if there's a commitment to review and get this merged then, avoiding additional future conflicts.

Looking forwarding hearing back from you and getting HA support into the chart!

Same. I have some improvements I'd like to submit, but I'm not sure if it's worth trying to include them in this PR or creating a separate one.

Bloodraven21 · 2026-07-28T14:24:01Z

first can you rebase and make it up to date

second give me a exact values files to test it out on local

Bloodraven21 · 2026-07-28T14:25:06Z

@daanvinken @Bloodraven21 Pinging you here as you've been most active lately in committing to main.
It would really be great to finalize this feature (in here or a separate clean branch) so this can finally find it's way in. I am also happy to clean up if there's a commitment to review and get this merged then, avoiding additional future conflicts.
Looking forwarding hearing back from you and getting HA support into the chart!

Same. I have some improvements I'd like to submit, but I'm not sure if it's worth trying to include them in this PR or creating a separate one.

please make a spearate pr and raise it will look into it

Bloodraven21 · 2026-07-28T14:27:10Z

@daanvinken @Bloodraven21 Pinging you here as you've been most active lately in committing to main.

It would really be great to finalize this feature (in here or a separate clean branch) so this can finally find it's way in. I am also happy to clean up if there's a commitment to review and get this merged then, avoiding additional future conflicts.

Looking forwarding hearing back from you and getting HA support into the chart!

please create a spearate one

PR valkey-io#137 hardcodes `podManagementPolicy: Parallel`, unguarded by sentinel and with no values knob. Two problems for existing non-sentinel users: - the field is IMMUTABLE, so applying it to an existing StatefulSet is rejected by the API server; adopting this chart would break the sync - Parallel restarts every pod at once, where OrderedReady rolls the primary last -- for a replicated cache that turns a brief blip into a full outage Now defaults to Parallel only when replica.sentinel.enabled (sentinel does need pods up together to form quorum) and OrderedReady otherwise, matching upstream. Overridable via .Values.podManagementPolicy. Verified: production values (sentinel off) render OrderedReady, sentinel values render Parallel. helm lint clean; unittest unchanged at 136 passed / 10 failed (those 10 pre-date this branch). Worth upstreaming to valkey-io#137.

The watcher polls each Sentinel in turn to find the current master, using valkey-cli with no -t. When the pod hosting the first Sentinel is the one that just died, that connect blocks on TCP SYN retries -- ~127s with the default tcp_syn_retries=6 -- before falling through to a live peer. Measured in qa1 on a settled 3-node cluster: Sentinel promoted a replica in ~6s, but the watcher did not observe it for 141s, so HAProxy kept pointing at the dead primary the whole time. Writes through the HAProxy endpoint failed for the entire window (0 OK / 7 FAIL over a 150s poll). That is worse than having no failover at all, and it is the single thing preventing this from being production-usable. Now bounded by haproxy.sentinelWatcher.connectTimeout (default 1s), so a dead peer costs ~1s instead of ~127s. Worth upstreaming to valkey-io#137 -- it makes the difference between advertised "near-instant IP-based failover" and a two-minute outage.

…ontrols Two problems found running this in production. 1. The PDB selects on name+instance only, and the HAProxy pods carry both. With haproxy.enabled the PDB covered 6 pods instead of 3, so minAvailable:2 allowed 4 disruptions -- a drain could evict every StatefulSet pod while HAProxy alone satisfied the budget. Observed live: expected=6, allowed=4. Now excludes component=haproxy when HAProxy is on. 2. haproxy-deployment had no nodeSelector/affinity/topologySpreadConstraints/ tolerations at all, so all replicas can land on one node. It fronts the write path, so it needs the same scheduling controls the StatefulSet has. Both worth upstreaming to valkey-io#137.

sgissi · 2026-09-10T14:17:11Z

Closing in favor of #234

khtee force-pushed the main branch from a9d23ba to c9ef751 Compare February 9, 2026 00:12

dmaes mentioned this pull request Feb 11, 2026

Implement HA with Sentinel #22

Open

khtee marked this pull request as draft February 18, 2026 23:57

khtee changed the title ~~feat: Add Valkey Sentinel support in High Availability setup.~~ feat: Add Valkey Sentinel & HAProxy support in High Availability setup. Feb 19, 2026

khtee marked this pull request as ready for review February 19, 2026 01:19

khtee marked this pull request as draft February 19, 2026 01:30

khtee marked this pull request as ready for review February 19, 2026 06:11

dmaes suggested changes Feb 19, 2026

View reviewed changes

Comment thread valkey/values.yaml Outdated

Comment thread valkey/templates/haproxy-deployment.yaml Outdated

Comment thread valkey/templates/haproxy-deployment.yaml

Comment thread valkey/values.yaml Outdated

Comment thread valkey/values.yaml

khtee force-pushed the main branch from 35e72a3 to 26d5117 Compare February 20, 2026 01:47

khtee and others added 5 commits February 20, 2026 09:53

feat: Add Valkey Sentinel support in High Availability setup.

4c7c981

Signed-off-by: KHTee <teekahhui@hotmail.com>

feat: Add Valkey Sentinel & HAProxy support in High Availability setup.

32de638

Signed-off-by: KHTee <teekahhui@hotmail.com>

Add securityContext to HAProxy & Sentinel watcher

708c707

Co-authored-by: Dieter Maes <dieter.maes@dmaes.be> Signed-off-by: khtee <75174583+khtee@users.noreply.github.com>

Use .Chart.AppVersion for sentinel watcher

adc429d

Signed-off-by: KHTee <teekahhui@hotmail.com>

khtee force-pushed the main branch from 26d5117 to 7184810 Compare February 20, 2026 01:55

khtee requested a review from dmaes February 20, 2026 01:56

LouisVallat mentioned this pull request Feb 25, 2026

Replace Bitnami Redis dependency and provide Helm chart support for Valkey passbolt/charts-passbolt#122

Open

tkarger reviewed Mar 11, 2026

View reviewed changes

Comment thread valkey/templates/haproxy-configmap.yaml Outdated

tkarger reviewed Mar 12, 2026

View reviewed changes

Comment thread valkey/templates/haproxy-deployment.yaml

tkarger reviewed Mar 12, 2026

View reviewed changes

Comment thread valkey/values.yaml Outdated

tkarger reviewed Mar 12, 2026

View reviewed changes

Comment thread valkey/values.schema.json

jose and others added 3 commits April 7, 2026 14:29

Clenning haproxy-service and added support for commonLabels and podAn…

d6b55d5

…notations in haproxy-deployment

Merge pull request #1 from jose-10000/fix/sentinel-acl-hardening-and-…

d8f854d

…schema Valkey Sentinel: Fixing Auth/ACL gaps in TLS environments and security hardening (Enhancements for PR valkey-io#137)

yoannrt reviewed Apr 9, 2026

View reviewed changes

lazariv suggested changes Apr 13, 2026

View reviewed changes

geodimm reviewed Jun 11, 2026

View reviewed changes

This was referenced Aug 1, 2026

feat(valkey): add Valkey Sentinel for automatic failover #234

Open

feat(valkey): add an optional HAProxy front-end for Sentinel #235

Open

sgissi closed this Sep 10, 2026

discostur mentioned this pull request Sep 10, 2026

feat(valkey): let the Sentinel password belong to the default user pat-s/valkey-helm#1

Open

		@@ -188,3 +188,34 @@ Validate replica authentication configuration
		{{- end }}
		{{- end -}}

+{{/*
+Validate haproxy is used in replica mode
+*/}}
+{{- define "valkey.validateHaproxyRequirements" -}}
+{{- if and .Values.haproxy.enabled (not .Values.replica.enabled) }}
+  {{- fail "Haproxy is only relevant in replica mode with clients incompatible with Sentinel." }}
+{{- end }}
+{{- end -}}

-      timeout server {{ .Values.haproxy.config.timeout.server }}
+      timeout server {{ .Values.haproxy.config.timeout.server }}
+      timeout tunnel {{ .Values.haproxy.config.timeout.tunnel }}
+      option clitcpka
+      option srvtcpka

-      server: 1m
+      server: 1m
+      # Timeout for long-lived bidirectional connections (e.g. pub/sub).
+      # Set to 0 to keep pub/sub SUBSCRIBE connections alive indefinitely.
+      tunnel: 0s

Conversation

khtee commented Feb 6, 2026

Uh oh!

dmaes commented Feb 11, 2026

Uh oh!

khtee commented Feb 11, 2026

Uh oh!

yoannrt commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amontalban commented Feb 13, 2026

Uh oh!

dmaes commented Feb 14, 2026

Uh oh!

yoannrt commented Feb 17, 2026

Uh oh!

khtee commented Feb 19, 2026

Uh oh!

dmaes commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lazariv commented Mar 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yoannrt Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

lazariv left a comment

Choose a reason for hiding this comment

Uh oh!

lazariv Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lazariv Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

lazariv Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

jose-10000 commented Apr 27, 2026

Uh oh!

jose-10000 commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jose-10000 commented May 18, 2026

Uh oh!

dmaes commented May 18, 2026

Uh oh!

jose-10000 commented May 18, 2026

Uh oh!

trabelsieyal commented May 20, 2026

Uh oh!

mlimardo1984 commented Jun 4, 2026

Uh oh!

jsoref commented Jun 9, 2026

Uh oh!

Nils98Ar commented Jun 10, 2026

Uh oh!

geodimm Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

kaisen-san commented Jun 18, 2026

Uh oh!

yoannrt commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pat-s commented Jul 28, 2026

Uh oh!

kaisen-san commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yoannrt commented Feb 11, 2026 •

edited

Loading

dmaes commented Feb 19, 2026 •

edited

Loading

lazariv Apr 13, 2026 •

edited

Loading

jose-10000 commented Apr 30, 2026 •

edited

Loading

yoannrt commented Jun 18, 2026 •

edited

Loading

kaisen-san commented Jul 28, 2026 •

edited

Loading

Bloodraven21 commented Jul 28, 2026 •

edited

Loading