feat: hold /health and /ready open after a failed startup - #27612
davidby-influx wants to merge 6 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Freshness snapshots can still combine fields from different observations when wrapping another stateful response.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a configurable linger window so failed startup diagnostics remain available through /health and /ready.
Changes:
- Adds
--startup-error-lingerwith a 30-minute cap. - Introduces phased shutdown and frozen health snapshots.
- Documents and tests lifecycle, authorization, and configuration behavior.
File summaries
| File | Description |
|---|---|
kit/check/response.go |
Adds coherent response snapshots. |
kit/check/helpers.go |
Preserves snapshots through renamed responses. |
kit/check/freshness.go |
Adds freshness snapshots and JSON rendering. |
kit/check/freshness_test.go |
Tests freshness snapshot behavior. |
kit/check/freeze_test.go |
Tests frozen check sets. |
kit/check/checktest/checktest.go |
Adds response normalization helpers. |
kit/check/check.go |
Implements terminal check freezing. |
kit/check/check_race_test.go |
Exercises concurrent freezing. |
http/check_handler.go |
Freezes endpoints and retires auth dependencies. |
http/check_handler_freeze_test.go |
Tests frozen HTTP responses. |
HEALTH_READY.md |
Documents linger behavior and operations. |
cmd/influxd/launcher/startup_failure_test.go |
Tests end-to-end startup failure lingering. |
cmd/influxd/launcher/shutdown_test.go |
Tests phased shutdown and error aggregation. |
cmd/influxd/launcher/launcher.go |
Implements linger and phased teardown. |
cmd/influxd/launcher/export_test.go |
Exposes lifecycle helpers to tests. |
cmd/influxd/launcher/cmd.go |
Adds CLI configuration and failure-path shutdown. |
cmd/influxd/launcher/cmd_test.go |
Tests option resolution and output. |
Review details
- Files reviewed: 17/17 changed files
- Comments generated: 3
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
🟡 Changes recommended
Freeze deadlines are not externally enforced, and concurrent calls can execute probes repeatedly.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
kit/check/check.go:227
- Concurrent
Freezecalls can all observefrozen == falseand evaluate every checker before one wins the final lock. That contradicts the documented first-freeze-wins/no-op behavior and can run expensive or stateful probes multiple times; the new concurrent test starts four freezers but does not assert invocation counts. Please coordinate the in-progress freeze so only one caller probes and competing callers wait or return.
func (c *Check) Freeze(ctx context.Context) {
c.mu.RLock()
frozen := c.frozen
health := append([]Checker(nil), c.healthChecks...)
ready := append([]Checker(nil), c.readyChecks...)
c.mu.RUnlock()
if frozen {
return
- Files reviewed: 17/17 changed files
- Comments generated: 1
- Review effort level: Balanced
f703f0c to
555f16a
Compare
A failed start names the failing subsystem on both endpoints, and the process exits before a scraper can read it. The body lives for microseconds, so the log line was the one durable copy. --startup-error-linger keeps both endpoints answering for a fixed duration after a failed start, capped at 30 minutes: the window holds the HTTP port on a process a supervisor is waiting to restart. The default of 0 preserves the current behavior. Teardown splits around the wait. Every subsystem except the HTTP listener and the PID file closes ahead of it, so the bolt flock, the sqlite file and the engine directory belong to the next run. The listener serves the window; the PID file keeps a second influxd from starting against a port this process holds. Shutdown releases both, and consumes each closer as it runs, so no phase can close a store a second time. The check set freezes ahead of that teardown. A closing store reports its own shutdown as a fresh failure, and failures sort ahead of passes, so it would outrank the startup error the window exists to publish. Check.Freeze bounds each probe on its own, so one wedged subsystem cannot spend the budget the checks after it need. Supporting changes: Response gains Snapshotter, for a coherent read of a FreshnessResponse; Shutdown aggregates closer failures with errors.Join and runs on the startup-failure path, which leaves no orphaned PID file. HEALTH_READY.md documents the flag, the cap, and the reduced check detail an operator sees during the window under --health-auth-enabled.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Code review sugestion Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1c1a684 to
f85a4ab
Compare
A failed start names the failing subsystem on both endpoints, and the process exits before a scraper can read it. The body lives for microseconds, so the log line was the one durable copy.
--startup-error-linger keeps both endpoints answering for a fixed duration after a failed start, capped at 30 minutes: the window holds the HTTP port on a process a supervisor is waiting to restart. The default of 0 preserves the current behavior.
Teardown splits around the wait. Every subsystem except the HTTP listener and the PID file closes ahead of it, so the bolt flock, the sqlite file and the engine directory belong to the next run. The listener serves the window; the PID file keeps a second influxd from starting against a port this process holds. Shutdown releases both, and consumes each closer as it runs, so no phase can close a store a second time.
The check set freezes ahead of that teardown. A closing store reports its own shutdown as a fresh failure, and failures sort ahead of passes, so it would outrank the startup error the window exists to publish. Check.Freeze bounds each probe on its own, so one wedged subsystem cannot spend the budget the checks after it need.
Supporting changes: Response gains Snapshotter, for a coherent read of a FreshnessResponse; Shutdown aggregates closer failures with errors.Join and runs on the startup-failure path, which leaves no orphaned PID file. HEALTH_READY.md documents the flag, the cap, and the reduced check detail an operator sees during the window under --health-auth-enabled.