Skip to content

Add Raft identity lease fencing#593

Merged
Osvaldo Andrade (osvaldoandrade) merged 1 commit into
masterfrom
work/raft-identity-fencing
Jun 25, 2026
Merged

Add Raft identity lease fencing#593
Osvaldo Andrade (osvaldoandrade) merged 1 commit into
masterfrom
work/raft-identity-fencing

Conversation

@osvaldoandrade

Copy link
Copy Markdown
Collaborator

Summary

  • Add a Raft identity lease guard with file and Kubernetes Lease backends.
  • Acquire and renew the lease before opening Raft or multi-Raft, and shut the process down if the lease is lost.
  • Wire Helm to use Kubernetes Lease objects for pod raft-id ownership, including ServiceAccount and minimal Role/RoleBinding.

Why

A stale CefasDB process must not continue serving after Kubernetes starts a replacement with the same raft-id. The lease epoch gives each process ownership proof and blocks duplicate identity startup until the old holder is fenced or the lease expires.

Validation

  • go test ./internal/identitylease ./internal/config ./internal/bootstrap/server ./cmd/cefasdb/...
  • helm template test dist/helm/cefas
  • go test ./...
  • RESET_CLUSTER=1 RESULT_DIR=/tmp/cefas-bench/raft-identity PROJECT=cefas-raft-identity-bench scripts/bench/bench_cluster.sh

Bench results:

  • bulk write: 2,000,000 units, 0 errors, 319,225 units/s
  • bulk read: 500,000 units, 0 errors, 63,979 units/s
  • soak write: 13,500,000 units over 15m, 0 errors, 15,000 units/s, p99 19.768ms
  • soak read: 18,000,000 units over 15m, 0 errors, 19,997 units/s, p99 783us

Closes #570
Closes #576
Closes #577
Closes #578

@osvaldoandrade Osvaldo Andrade (osvaldoandrade) merged commit 8230788 into master Jun 25, 2026
8 checks passed
@osvaldoandrade Osvaldo Andrade (osvaldoandrade) deleted the work/raft-identity-fencing branch June 25, 2026 02:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant