feat(valkey): add Valkey Sentinel for automatic failover

[pat-s]

Supersedes #137, rebased onto current main with the review feedback applied.
This PR adds Sentinel only; the HAProxy front-end follows separately.

Replication mode cannot recover from a primary failure on its own: pod 0 remains the configured primary until someone intervenes.
This change adds optional Valkey Sentinel support through replica.sentinel.enabled, allowing a replica to be promoted automatically.

Architecture

Sentinel runs in its own StatefulSet rather than as a Valkey sidecar.
The default topology is three Sentinel pods, independently of the number of Valkey pods.
This keeps Sentinel quorum and data-replica count separate, so Valkey can run with one primary and one replica while retaining three Sentinels.

• A dedicated headless Service provides stable DNS identities for Sentinel peer discovery.
• A separate client-facing Service lets applications resolve the current primary through Sentinel.
• Sentinel and Valkey have independent replica counts, storage, rollout lifecycles, and selectors.
• Sentinel pods cannot become endpoints of the Valkey Services.

Failover and recovery

• The valkey Service no longer pins its selector to pod 0 when Sentinel is enabled, because the primary can move, and is labelled app.kubernetes.io/component: nodes rather than primary to match.
• A starting pod asks Sentinel which node is the primary, then the other nodes, and only then the topology it recorded itself.
• The generated config carries the replication credentials, so it lives on a memory-backed volume and nothing in it outlives the pod.
  Each pod therefore keeps a credential-free record of the current primary on its data volume, refreshed while it runs, and that is what a cold start falls back on when no Sentinel and no node can be reached.
• A pod that finds none of the three waits to be told rather than exiting, which is what a first install spends.
• Sentinel pods discover the current primary from existing Sentinel peers first and the Valkey nodes second.
• Sentinel learns which nodes are replicas from the primary, so a node replicating from another replica is invisible to it and is never repaired. Each Sentinel reattaches those.
• A Valkey preStop hook requests failover before terminating the current primary.
• Sentinel uses stable pod hostnames instead of ephemeral pod IPs.

Authentication

Sentinel has a dedicated ACL identity and password, independently of whether authentication is enabled for the Valkey data endpoint.
When Valkey authentication is enabled, Sentinel uses replica.sentinel.monitorUser to monitor nodes and promote replicas, and a starting pod asks that same user which node is the primary, since the documented minimum for a replication user cannot run INFO.
Credentials are mounted from Secrets and passed to valkey-cli through REDISCLI_AUTH rather than command-line arguments.
Generated Valkey and Sentinel configuration quotes and escapes credentials containing whitespace, double quotes, or backslashes.
An empty value in a Secret is refused rather than hashed into an ACL, because the hash of an empty password is the same in every installation.

Testing

• just validate: 285 unit tests across all three charts.
• helm lint ./valkey.
• Template rendering against Kubernetes 1.28, 1.35, and 1.36.
• A kind cluster with ACL authentication using valkey/examples/ha-sentinel.yaml.
• A clean install reached three healthy pods and accepted writes on the primary Sentinel had chosen.
• Three independent Sentinels formed a quorum and discovered the Valkey topology.
• Sentinel promoted a replica and all peers agreed on the new primary.
• Valkey scaled from three pods to two while retaining the Sentinel quorum.
• Sentinel scaled from three pods to four without restarting either Valkey pod.
• Graceful removal of the current primary triggered failover before termination.
• A forced full restart preserved a primary that had moved away from pod 0.
• Special-character Sentinel and replication passwords reached full workload readiness and authenticated successfully.
• Graceful failover succeeded with the optional Sentinel client Service disabled and its external port changed.

One drive-by fix removes a stray tab from statefulset.yaml that caused strict YAML parsers to reject the rendered manifest.

Credit to @khtee, @jose-10000, @yoannrt, @tkarger, @lazariv, and @dmaes, whose work in #137 this builds on.

[greptile-apps]

Greptile Summary

This change adds Sentinel-based high-availability deployment and connection guidance for the Valkey chart, including TLS-aware client examples. The documented Python TLS path was rendered, compiled, and exercised with redis-py 8.1.0.

Disproved and dropped: TLS Python example is malformed. The TLS example compiles successfully, and the master client inherits its TLS configuration from the Sentinel constructor as intended.

Confidence Score: 5/5

The PR is safe to merge; no blocking failure remains.

The final defect set is empty. The reported TLS documentation failure was disproved by compiling the rendered example and confirming real redis-py master-connection TLS inheritance.

T-Rex Logs

What T-Rex did

• The TLS-enabled Sentinel reproduction script from NOTES.txt:36-38 was executed and compiled with SYNTAX_COMPILE=PASS.
• Exercising the same call with redis-py 8.1.0 showed that the master pool uses SentinelManagedSSLConnection and retains ssl_ca_certs.
• The final run exited with code 0 and the recorder captured TLS constructor keyword arguments, with real redis-py reporting the master connection class as SentinelManagedSSLConnection and ssl_ca_certs retained.

_{Ran code and verified through T-Rex}

_{Reviews (32): Last reviewed commit: "docs(valkey): give the python example th..." | Re-trigger Greptile}

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The Helm chart adds Valkey Sentinel high-availability mode. It introduces Sentinel configuration, StatefulSets, services, topology discovery, authentication, TLS support, graceful failover, validation, examples, documentation, and Helm tests.

Changes

Valkey Sentinel high availability

Layer / File(s)	Summary
Sentinel configuration and validation valkey/values.yaml, valkey/values.schema.json, valkey/templates/_helpers.tpl, valkey/templates/secret.yaml, valkey/examples/ha-sentinel.yaml, valkey/README.md, valkey/tests/secret_test.yaml, valkey/tests/sentinel_test.yaml	Adds the replica.sentinel configuration, schema validation, Sentinel credentials, example values, documentation, and validation tests.
Sentinel bootstrap and failover runtime valkey/templates/init_config.yaml, valkey/templates/sentinel-configmap.yaml, valkey/tests/sentinel_test.yaml	Adds topology discovery, atomic Valkey configuration generation, authenticated and TLS-enabled Sentinel startup, health checks, pre-stop failover, and runtime tests.
StatefulSet, services, and rendered guidance valkey/templates/sentinel-statefulset.yaml, valkey/templates/statefulset.yaml, valkey/templates/sentinel-service.yaml, valkey/templates/sentinel-headless-service.yaml, valkey/templates/service.yaml, valkey/templates/deploy_valkey.yaml, valkey/templates/NOTES.txt, valkey/tests/sentinel_test.yaml	Adds Sentinel Kubernetes resources, storage, mounts, probes, lifecycle hooks, service routing, deployment validation, and Sentinel-specific installation notes. Validators cover rendered resources and disabled-Sentinel behavior.

Sequence Diagram(s)

sequenceDiagram
  participant SentinelStatefulSet
  participant SentinelStartup
  participant ValkeyInit
  participant ValkeyNode
  participant SentinelService
  SentinelStatefulSet->>SentinelStartup: start Sentinel
  SentinelStartup->>SentinelService: discover current master
  ValkeyInit->>ValkeyNode: restore or assign replication target
  SentinelStartup->>ValkeyNode: monitor Valkey configuration
  SentinelStatefulSet->>SentinelStartup: run pre-stop hook
  SentinelStartup->>SentinelService: request master failover

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding Valkey Sentinel for automatic failover.
Description check	✅ Passed	The description explains the Sentinel architecture, failover behavior, authentication, recovery, testing, and related scope. It is directly related to the changeset.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 8

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

valkey/README.md (1)

43-45: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Remove the empty **IMPORTANT** label.

The content that followed this label moved out of the Replication Mode section. The label now renders as a bold word with no body.

📝 Proposed change

-**IMPORTANT**
-
 **Services:**

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@valkey/README.md` around lines 43 - 45, Remove the standalone **IMPORTANT**
label immediately before the Services section in the README, leaving the
Services content and surrounding documentation unchanged.

🧹 Nitpick comments (2)

valkey/examples/ha-sentinel.yaml (1)

29-36: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove the on flag from replicator.permissions.

The chart emits the ACL rule as user <name> on >password <permissions>, as shown in valkey/values.yaml line 238. The default user above and the examples in values.yaml omit on. Keep the same form here so users do not copy a duplicated flag.

♻️ Proposed change

       permissions: >-
-        on ~* &* +multi +exec +ping +info +role +subscribe +publish +slaveof
+        ~* &* +multi +exec +ping +info +role +subscribe +publish +slaveof
         +replicaof +config|rewrite +client|setname +client|kill +script|kill
         +psync +replconf

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@valkey/examples/ha-sentinel.yaml` around lines 29 - 36, Remove the leading
“on” token from the replicator.permissions ACL rule while preserving all
existing key patterns and command permissions, matching the format used by the
chart’s ACL rendering and other examples.

valkey/templates/service.yaml (1)

34-38: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

The service keeps write semantics that it no longer provides in Sentinel mode.

Removing the pod-0 pin is correct, because the master moves after a failover. The service still carries app.kubernetes.io/component: primary at line 7, and NOTES.txt still presents it under "WRITE Operations". Clients that follow the label or the notes will send writes to replicas and receive -READONLY.

Consider setting a different component label in Sentinel mode, so that consumers of the label do not treat this service as master-only.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@valkey/templates/service.yaml` around lines 34 - 38, Update the service
metadata around the component label and the Sentinel configuration condition so
Sentinel mode no longer advertises the service as primary/master-only. Preserve
the primary component label for non-Sentinel deployments, and align the service
label with the behavior documented for Sentinel consumers.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@valkey/README.md`:
- Around line 104-107: Update the fenced code block containing the ACL
permission list in the README to specify the text language, changing the opening
fence to use text while preserving the block contents.

In `@valkey/templates/_helpers.tpl`:
- Around line 219-224: Update the Sentinel monitor-user resolution in the
auth-enabled block around $monitorUser and valkey.validateSentinelConfig so
auth.aclConfig-only configurations do not dereference a missing auth.aclUsers
map. Add the same nil-safe guard before reading auth.aclUsers, or ensure the
validator emits the existing explicit missing-user failure before this lookup,
while preserving the current fallback and fail message for defined ACL users.

In `@valkey/templates/init_config.yaml`:
- Around line 181-196: The sentinel bootstrap decision in init.sh must not let
pod 0 become MASTER solely because local PVC state is missing after failover.
Before the POD_INDEX master branch, reuse sentinel-start.sh’s peer-discovery
mechanism to query a reachable Sentinel and, when it reports a promoted master,
set REPLICATION_TARGET to replicaof that address; only fall back to pod 0 as
MASTER when discovery finds no existing master.
- Around line 169-179: Update the listening-port configuration to use
.Values.service.port instead of a hardcoded 6379, preserving the existing
TLS/non-TLS directive ordering. Apply this in valkey/templates/init_config.yaml
lines 169-179 and update the corresponding hardcoded port references in
valkey/templates/sentinel-configmap.yaml lines 75-81 and 279-285 so Sentinel
uses the same configured port.

In `@valkey/templates/NOTES.txt`:
- Around line 26-32: Update the Sentinel connection examples in NOTES.txt to
branch on .Values.replica.sentinel.service.enabled: use the existing service
hostname and sentinel.service.port when enabled, and use the headless service
hostname with .Values.replica.sentinel.port when disabled. Apply the same
conditional endpoints to both the valkey-cli command and the Python Sentinel
example.

In `@valkey/templates/sentinel-configmap.yaml`:
- Around line 86-101: The master_from_peers function must validate sentinel_cli
output before accepting it: require exactly two stdout lines containing a valid
master host and port, and reject empty, malformed, or error-reply output when
the peer is not configured for MASTER_SET. Only log, echo, and return success
for an unambiguous validated address; otherwise continue checking the remaining
peers.

In `@valkey/templates/sentinel-service.yaml`:
- Around line 21-23: Update the loadBalancerSourceRanges rendering in the
sentinel service template to serialize the list with Helm’s toYaml helper,
preserving valid YAML sequence syntax and the existing conditional block.

In `@valkey/templates/statefulset.yaml`:
- Around line 19-25: Add an upgrade note under the Sentinel section in
valkey/README.md documenting that enabling Sentinel may change immutable
StatefulSet fields, including podManagementPolicy and persistence
volumeClaimTemplates, so users must delete the StatefulSet with --cascade=orphan
before helm upgrade to allow existing pods and PVCs to be re-adopted.

---

Outside diff comments:
In `@valkey/README.md`:
- Around line 43-45: Remove the standalone **IMPORTANT** label immediately
before the Services section in the README, leaving the Services content and
surrounding documentation unchanged.

---

Nitpick comments:
In `@valkey/examples/ha-sentinel.yaml`:
- Around line 29-36: Remove the leading “on” token from the
replicator.permissions ACL rule while preserving all existing key patterns and
command permissions, matching the format used by the chart’s ACL rendering and
other examples.

In `@valkey/templates/service.yaml`:
- Around line 34-38: Update the service metadata around the component label and
the Sentinel configuration condition so Sentinel mode no longer advertises the
service as primary/master-only. Preserve the primary component label for
non-Sentinel deployments, and align the service label with the behavior
documented for Sentinel consumers.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c6fd83c9-180e-40e7-9759-3b120b5dc905

📥 Commits

Reviewing files that changed from the base of the PR and between 01b4fa4 and fa6cfb8.

📒 Files selected for processing (16)

• valkey/.helmignore
• valkey/Chart.yaml
• valkey/README.md
• valkey/examples/ha-sentinel.yaml
• valkey/templates/NOTES.txt
• valkey/templates/_helpers.tpl
• valkey/templates/deploy_valkey.yaml
• valkey/templates/init_config.yaml
• valkey/templates/sentinel-configmap.yaml
• valkey/templates/sentinel-service.yaml
• valkey/templates/service-headless.yaml
• valkey/templates/service.yaml
• valkey/templates/statefulset.yaml
• valkey/tests/sentinel_test.yaml
• valkey/values.schema.json
• valkey/values.yaml

[pat-s]

Thanks bots. Pushed a fix for two of the three findings:

Bootstrap marker split-brain (Greptile, P1) — real, fixed. The marker is gone. The role now comes from the previous config, which already carries the answer: a replicaof directive means replica, a complete config without one means master, no config means first boot and the pod index decides. The config is assembled in a temp file and moved into place at the end, so a config on disk is always complete and that distinction holds.

Verified on a kind cluster by wiping valkey.conf on a replica and restarting it: it comes back as a replica and Sentinel re-points it at the master. Before the fix that path produced a second writable master.

Unauthenticated Sentinel port (Greptile, P1) — protected-mode no is now only emitted when auth.enabled is false, where Sentinel peers could not connect otherwise. Worth noting the exposure is not specific to Sentinel: with auth disabled the Valkey port is equally open. Called out in values.yaml so it is not a surprise.

Nil auth.aclUsers.default with aclConfig only (CodeRabbit) — not reachable. Sentinel requires replica.enabled, and validateReplicaAuth already fails for an aclConfig-only setup, while validateAuthConfig requires default in aclUsers whenever aclUsers is set. Both produce a clear message before the Sentinel templates index the map:

$ helm template t ./valkey --set replica.enabled=true --set replica.sentinel.enabled=true \
    --set auth.enabled=true --set auth.aclConfig="user default on >pw ~* &* +@all" ...
Error: Replication user 'default' (replica.replicationUser) must be defined in auth.aclUsers.

Also took the MD040 suggestion.

[pat-s]

Second round of review fixes pushed. Verified on a kind cluster with ACL auth.

Fixed

• Pod-0 restarts as a second master after losing its volume (CodeRabbit) — real. init.sh now asks the running Sentinels for the current master before falling back to the pod index, the same discovery sentinel-start.sh already did. Reproduced by failing over away from pod-0, deleting pod-0's entire data directory and restarting it. Before: pod-0 came back a writable master. Now:

  Sentinel reports valkey-2.valkey-headless...:6379 as the master, starting as REPLICA
  valkey-0: role:slave  master_host:valkey-2...
  valkey-1: role:slave  master_host:valkey-2...
  valkey-2: role:master
  

• Hardcoded listen port (CodeRabbit) — real, and pre-existing. init.sh wrote port 6379 while the container port and Services use service.port, so --set service.port=6380 produced a server listening on the wrong port. Now rendered from service.port.

• Password in argv (CodeRabbit) — all three scripts use REDISCLI_AUTH instead of -a, so credentials are no longer in /proc/<pid>/cmdline.

• Unvalidated peer answer (CodeRabbit, both comments) — a host/port pair is now required, and the port must be numeric, before it becomes a sentinel monitor directive. Peer queries are bounded with -t 2 and the startup probe budget widened to cover discovery.

• preStop timeout vs grace period (CodeRabbit) — added replica.terminationGracePeriodSeconds (30) and rendering fails if preStopFailoverTimeoutSeconds is not lower than it.

• Immutable StatefulSet fields (CodeRabbit) — good catch, this would have bitten every existing replication user. Added replica.podManagementPolicy so a release can enable Sentinel without a recreate, plus the --cascade=orphan procedure in the README for anyone taking the default.

• loadBalancerSourceRanges (CodeRabbit) — now toYaml. Confirmed the old form rendered [10.0.0.0/8 10.1.0.0/16], which the API server rejects. The same bug exists in service.yaml and service-read.yaml; left alone here since it is unrelated to this feature, happy to send a separate PR.

• Notes point at a Service that may not exist (CodeRabbit) — guarded on replica.sentinel.service.enabled, with the headless endpoints printed otherwise.

Not changing

• Unauthenticated Sentinel port with auth.enabled: false (Greptile) — protected-mode no is required here, Sentinel peers connect over the pod network and protected mode rejects them when no password is set. The exposure is not specific to Sentinel: with auth disabled the Valkey port is equally open. protected-mode no is now only emitted in that case, and values.yaml states it plainly. Enabling auth secures both ports.

• Nil auth.aclUsers.default with aclConfig-only auth (CodeRabbit) — not reachable. Sentinel requires replica.enabled, and the existing validations already stop this before any Sentinel template indexes the map:

  $ helm template t ./valkey --set replica.enabled=true --set replica.sentinel.enabled=true \
      --set auth.enabled=true --set auth.aclConfig="user default on >pw ~* &* +@all" ...
  Error: Replication user 'default' (replica.replicationUser) must be defined in auth.aclUsers.
  

  and with aclUsers set but no default, validateAuthConfig fails first. If either guard is ever relaxed, this becomes reachable and worth revisiting.

• replicaof could hold an IP (CodeRabbit) — the chart forces sentinel resolve-hostnames yes and announce-hostnames yes and writes replica-announce-ip as the pod FQDN, so Sentinel hands out hostnames. Verified on the cluster: the restored line is replicaof valkey-2.valkey-headless.default.svc.cluster.local 6379.

• The valkey Service load balances across all pods in Sentinel mode (CodeRabbit) — intentional, and the reason the pod-0 pin had to go: pinning leaves the write endpoint on a demoted replica after a failover. This matches how other Sentinel charts do it; clients resolve the master through Sentinel, or use the HAProxy front-end in feat(valkey): add an optional HAProxy front-end for Sentinel #235. Called out in the notes and the README.

167 unit tests, helm lint, and rendering against 1.28/1.35/1.36 all pass locally. CI is still action_required, it needs a maintainer to approve workflows for a first-time contributor.

[Bloodraven21]

thanks @pat-s for the pr can you provide me an example values.yaml for testing

[pat-s]

@Bloodraven21 sure — here is a complete example using an existing Secret (recommended):

First create the three credentials.
The Sentinel password is deliberately separate from the Valkey application and replication passwords:

kubectl create secret generic valkey-users \
  --from-literal=default="replace-with-a-strong-app-password" \
  --from-literal=replicator="replace-with-a-strong-replication-password" \
  --from-literal=sentinel="replace-with-a-strong-sentinel-password"

Save this as values-sentinel.yaml:

replica:
  enabled: true
  replicas: 2
  persistence:
    size: 1Gi
  replicationUser: replicator
  sentinel:
    enabled: true
    quorum: 2
    passwordKey: sentinel

auth:
  enabled: true
  usersExistingSecret: valkey-users
  aclUsers:
    default:
      permissions: "~* &* +@all"
    replicator:
      permissions: >-
        ~* &* +multi +exec +ping +info +role +subscribe +publish +slaveof
        +replicaof +config|rewrite +client|setname +client|kill +script|kill
        +psync +replconf

Install from the PR checkout with:

helm install valkey ./valkey -f values-sentinel.yaml

The chart also contains an inline-password example at valkey/examples/ha-sentinel.yaml.
I verified these values with helm template.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@valkey/values.yaml`:
- Around line 356-361: Update the comment beside replica.sentinel.password and
passwordKey to state that the existing secret’s sentinel key takes priority,
while the inline password is used as a fallback when that key is absent. Remove
the wording that claims the inline value is ignored, while preserving the
explanation that Sentinel credentials are independent from Valkey ACLs.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ef0b6a6e-2254-4105-875c-464b85033776

📥 Commits

Reviewing files that changed from the base of the PR and between fa6cfb8 and ff19130.

📒 Files selected for processing (13)

• valkey/README.md
• valkey/examples/ha-sentinel.yaml
• valkey/templates/NOTES.txt
• valkey/templates/_helpers.tpl
• valkey/templates/init_config.yaml
• valkey/templates/secret.yaml
• valkey/templates/sentinel-configmap.yaml
• valkey/templates/sentinel-service.yaml
• valkey/templates/statefulset.yaml
• valkey/tests/secret_test.yaml
• valkey/tests/sentinel_test.yaml
• valkey/values.schema.json
• valkey/values.yaml

🚧 Files skipped from review as they are similar to previous changes (6)

• valkey/templates/NOTES.txt
• valkey/templates/statefulset.yaml
• valkey/templates/sentinel-service.yaml
• valkey/templates/sentinel-configmap.yaml
• valkey/values.schema.json
• valkey/README.md

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@valkey/templates/sentinel-configmap.yaml`:
- Line 204: Update the SENTINEL_PASSHASH assignment in the Sentinel
configuration template to use printf with a %s format for SENTINEL_PASSWORD
before piping it to sha256sum, replacing echo -n while preserving the existing
hash extraction.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: cd8aa689-6463-4a4e-9d10-97e969d63384

📥 Commits

Reviewing files that changed from the base of the PR and between ff19130 and b42995d.

📒 Files selected for processing (10)

• valkey/README.md
• valkey/examples/ha-sentinel.yaml
• valkey/templates/_helpers.tpl
• valkey/templates/init_config.yaml
• valkey/templates/secret.yaml
• valkey/templates/sentinel-configmap.yaml
• valkey/tests/secret_test.yaml
• valkey/tests/sentinel_test.yaml
• valkey/values.schema.json
• valkey/values.yaml

🚧 Files skipped from review as they are similar to previous changes (9)

• valkey/values.schema.json
• valkey/tests/secret_test.yaml
• valkey/templates/secret.yaml
• valkey/values.yaml
• valkey/examples/ha-sentinel.yaml
• valkey/tests/sentinel_test.yaml
• valkey/templates/init_config.yaml
• valkey/templates/_helpers.tpl
• valkey/README.md

[pat-s]

Addressed the valid findings in b4f8d62.

• Replaced echo with printf throughout password handling so backslashes and option-like values remain unchanged.
• Fixed the inline replica.sentinel.password fallback when auth.usersExistingSecret is set.
• Corrected the documentation to describe existing-Secret priority rather than saying the inline password is ignored.

I left unauthenticated Sentinel behavior unchanged when auth.enabled: false because Sentinel peers require protected-mode no without credentials, and Valkey is intentionally unauthenticated in the same configuration.

Verified on kind with backslash-containing application, replication, and Sentinel passwords.
All 172 unit tests, helm lint, and Kubernetes 1.28/1.35/1.36 renders pass.

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[jose-10000]

Hi! Thanks for creating the new branch.

I tested this branch in an environment with TLS and ACLs enabled.

When trying to connect through HAProxy using a TLS-enabled client (valkey-cli --cacert ... --tls), the connection times out.

Looking at the current HAProxy configuration, the frontend is bound as plain TCP (bind *:6379), but the backend definition forces ssl. Because HAProxy doesn't terminate TLS on the frontend, it receives the client's raw TLS ClientHello over a plain TCP socket and proxies those bytes into its own backend TLS session to Valkey, which breaks the end-to-end TLS negotiation.

For environments where Valkey expects TLS directly from the client, having HAProxy act as a transparent L4 TCP passthrough works seamlessly:

1. Keep the data path as pure TCP passthrough: Remove ssl / ca-file from the default-server or server definitions so HAProxy doesn't try to negotiate TLS on behalf of the client.
2. Handle HAProxy health checks with TLS: Use tcp-check connect port 6379 ssl for the health check probe, and add ssl-server-verify none under global to prevent CA path issues during health checking.

I updated the config locally with this approach and confirmed that TLS connections from clients through HAProxy work as expected.

Here is the backend config snippet that resolved the issue:

global
  ssl-server-verify none

backend valkey_master
  mode tcp
  balance first
  option tcp-check

  tcp-check connect port 6379 ssl
  tcp-check send AUTH\ redis_haproxy_watcher\ "${VALKEY_CHECK_PASSWORD}"\r\n
  tcp-check expect string +OK
  tcp-check send info\ replication\r\n
  tcp-check expect rstring role:master
  tcp-check send QUIT\r\n
  tcp-check expect string +OK

  timeout check 5s
  default-server check inter 5s fall 1 rise 1 init-addr last,libc,none init-state down resolvers kubernetes on-marked-down shutdown-sessions

backend valkey_replicas
  mode tcp
  balance roundrobin
  option tcp-check

  tcp-check connect port 6379 ssl
  tcp-check send AUTH\ redis_haproxy_watcher\ "${VALKEY_CHECK_PASSWORD}"\r\n
  tcp-check expect string +OK
  tcp-check send PING\r\n
  tcp-check expect string +PONG
  tcp-check send QUIT\r\n
  tcp-check expect string +OK

  timeout check 5s
  default-server check inter 5s fall 2 rise 1 init-addr last,libc,none init-state down resolvers kubernetes

[pat-s]

@jose-10000 thanks, reproduced and fixed. Your diagnosis was exactly right. The fix is in #235, which is where the HAProxy templates live.

Reproduced first on kind with TLS and ACLs, using a client cert and CA from the same secret the chart mounts:

A) TLS client -> valkey pod directly ..... PONG
B) TLS client -> haproxy ................. timeout (exit 124)
C) plaintext client -> haproxy ........... PONG

So the frontend only ever served plaintext clients and silently hung TLS ones, which is a bad default when someone has deliberately enabled TLS.

I took the passthrough approach you suggested and made it the default, with the old behaviour kept as an option, since bridging plaintext clients into a TLS cluster is a legitimate reason to run HAProxy in the first place. haproxy.tls.mode:

• passthrough (default) — HAProxy forwards the encrypted stream untouched, the client completes the handshake with the Valkey node, TLS stays end to end.
• bridge — the previous behaviour: clients connect in plaintext, HAProxy speaks TLS to the nodes. Documented as leaving the client leg unencrypted.

One difference from your snippet: instead of ssl-server-verify none I used check-ssl on default-server, which forces the health check over TLS independently of the data path. That keeps ca-file and verify required, so the checks still validate the node certificates against tls.caPublicKey rather than trusting anything. haproxy.tls.verify: none is there if the certificates do not cover the pod names.

Verified on kind after the change, TLS end to end through HAProxy:

ping .................... PONG
info replication ........ role:master
set k1 v1 ............... OK
read port 6380 .......... PONG
failover ................ new master after 6s, k1 still readable, writes resume

and bridge still serves plaintext clients against the same TLS-enabled cluster.

One thing worth flagging for your setup: in passthrough the client validates the certificate presented by the Valkey pod while connecting to the HAProxy service name, so the server certificate needs a SAN for that name too (e.g. valkey-haproxy.<namespace>.svc.cluster.local). I had to add it to my test CA to make the handshake pass, and it is now called out in the README.

You may also see a brief -READONLY right after a recovered old master rejoins, before Sentinel demotes it. That is inherent to Sentinel rather than to this change, it cleared on the next attempt in my run, and it is documented under the HAProxy section.

[RodargDev]

@Bloodraven21 @jose-10000 is there any status update on this pr?

[discostur]

Thanks for picking this up and splitting it — I gave the branch a proper workout on a cluster and it holds together really well. Numbers below, plus a patch for the one open blocker.

Setup

6c7c9f9 on kind: 3 workers plus a control-plane, Kubernetes v1.37.0, Valkey 9.1.2, local-path storage. ACL auth via auth.usersExistingSecret, replica.replicas: 2, replica.sentinel.replicas: 3, quorum: 2, chart-default failover timings.

Each disruption ran under a monotonic-counter writer that resolves the master through Sentinel and reconnects on every write, at ~400-500 writes/s. So the outage column is what a client actually saw, and the counter is checked for regressions to catch a lost acknowledged write.

Results

Scenario	Result
Baseline topology	1 master + 2 replicas, master_link_status:up, num-other-sentinels:2, all three Sentinels agreeing, master reported as an FQDN
kubectl delete pod on the master	33922/33923 writes, max outage 0.0s, 1 -READONLY, 0 regressions
Rolling restart of the data StatefulSet	2 handovers, 27150/27152 writes, max outage 0.0s, 0 regressions
Ungraceful node loss (docker stop)	+sdown → +switch-master in 2.02s with #quorum 2/2; client-visible outage 9.0s, 18435/18440 writes
Cold start (both StatefulSets to 0 and back)	Pre-shutdown master returned as the only master in 32.3s
Ex-master loses its volume	Fresh PVC, came back as a replica, resynced, no split brain
Scale the data tier to 2 pods	3 Sentinels stay, quorum holds, a failover with 2 data pods completes
Upgrade from main then failover	Converged in 35.7s, data survived, quorum formed, failover works, helm uninstall leaves only the PVCs the retention policy documents

A repeat of the delete-master run measured 35997/36321 with a 2.0s outage, so expect roughly 0-2s of client-visible write outage depending on where in the preStop window the pod is caught. Neither run lost an acknowledged write.

The cold start is my favourite bit, because the log shows the mechanism. Master on pod-2 before the shutdown, so pod-0 has an index saying "you are the master" and no reachable Sentinel yet:

valkey-0  No Sentinel answered; last recorded master is valkey-2.valkey-headless...:6379, starting as REPLICA
valkey-1  Sentinel reports valkey-2.valkey-headless...:6379 as the master, starting as REPLICA
valkey-2  Sentinel reports this pod as the master, starting as MASTER

And after wiping pod-0's PVC (confirmed replaced, not reused) it takes the other branch and asks Sentinel instead. Both paths do exactly what they say.

Also worth noting for @jdheyburn's point: two data pods with three Sentinels is a working configuration here, and a failover still completes in it. That is the concrete payoff of the separate StatefulSet.

One thing worth knowing operationally rather than fixing: every rolling restart left one replica chained behind another for a while — valkey-1 replicating from valkey-0, which is itself a replica, so invisible to Sentinel. adopt_orphans repaired it in 40.2s and 41.2s across two runs, consistent with orphanCheckSeconds: 30 plus a check cycle. Working as designed; might deserve a sentence in the README next to the failover timing note, since a failover in that window would not reconfigure the chained node.

The default user question from @Mazorius

I could reproduce both halves of the report. The generated ACL renders as user default off resetchannels -@all, so:

$ valkey-cli -h valkey-sentinel -p 26379 -a "$PW" SENTINEL get-master-addr-by-name mymaster
NOAUTH Authentication required.
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.

$ valkey-cli -h valkey-sentinel -p 26379 --user sentinel -a "$PW" SENTINEL get-master-addr-by-name mymaster
valkey-0.valkey-headless.valkey.svc.cluster.local
6379

Correct for the ACL as written — the catch is just that AUTH <password> authenticates as default, and some clients have no username field at all to send. Self-hosted GitLab is the case here, and its own documented Sentinel topology uses requirepass, which is the default-user mechanism.

I implemented the switch @Mazorius suggested and sent it as pat-s#1, so it is a commit you can take or leave rather than more text here. replica.sentinel.aclUser, sentinel (unchanged, still the default) or default; sentinel sentinel-user drops out in default mode so the peers authenticate on sentinel sentinel-pass alone, and the runtime hashing stays exactly as it is.

Verification: helm lint clean, helm template clean on Kubernetes 1.36.0 / 1.35.3 / 1.28.15, helm unittest 216/216 (210 existing plus 6 new). With aclUser: sentinel the rendered manifests are byte for byte identical to this branch, so nothing changes for anyone who does not set it.

The assertion I cared about most was inter-Sentinel auth, since dropping sentinel sentinel-user is the same shape as the "empty array quorum discovery" problem @jose-10000 hit on #137:

$ valkey-cli -p 26379 -a "$PW" SENTINEL master mymaster | grep -A1 num-other-sentinels
num-other-sentinels
2

Quorum forms cleanly, all three probes pass with 0 restarts, and a password-only client now reaches the endpoint. I re-ran the failover matrix in that mode too and it all passes, preStopFailover, cold start and lost-volume discovery included.

@Mazorius, if you still have that cluster, aclUser: default is the path to try.

One unrelated thing I tripped over

podDisruptionBudget.minAvailable: 0 is silently dropped on the Valkey side, because templates/poddisruptionbudget.yaml tests truthiness — the same bug @jose-10000 caught on the HAProxy PDB in #235 and you fixed there in 93aa4e0. It predates this PR (it comes from #106 and is unchanged on main), so I will open it separately rather than adding scope here.

[Mazorius]

@discostur — confirmed on our cluster, aclUser: default is exactly what was needed. Thanks for
implementing it.

Setup: 64fc2e4 on Kubernetes v1.36.2 / OpenStack, Valkey 9.1.2, Flux. 3 data pods + 3 Sentinels,
quorum: 2, preStopFailover: true, ACL auth from an existing Secret (default / replicator /
sentinel). A real self-hosted GitLab instance holding shared_state, queues, actioncable and
trace_chunks.

$ valkey-cli -p 26379 -a "$PW" SENTINEL get-master-addr-by-name gitlab-kv
gitlab-kv-valkey-1.gitlab-kv-valkey-headless.gitlab-kv.svc.cluster.local
6379

$ valkey-cli -p 26379 -a "$PW" ACL WHOAMI
default

$ valkey-cli --user sentinel -p 26379 PING      # stale username now fails closed
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.

Inter-Sentinel auth was my one worry, since dropping sentinel sentinel-user is the same shape as the
quorum-discovery problem @jose-10000 hit on #137. It forms cleanly:

$ valkey-cli -p 26379 -a "$PW" SENTINEL master gitlab-kv
flags
master
num-slaves
2
num-other-sentinels
2

0 restarts. Enabling the switch rolled both StatefulSets and Sentinel tracked the primary through it
(master moved to pod 1). I didn't instrument writes, so no outage numbers from me — yours stand.

For context on why no username can be sent: GitLab's chart exposes global.redis.sentinelAuth as a
secret and a key with no username field
(gitlab-org/charts/gitlab#6555, open),
so AUTH <password> — authenticating as default — is the only shape available on that side.

[pat-s]

Please stop posting these verbose and non-helpful walls of text. I use AI myself and I am not against it but these kind of postings are really not helpful and close to spam.

[teeram]

This is a great feature and I'm looking forward to being able to use it. Thanks for all the hard work @pat-s. Any idea when this can get merged in?