Skip to content

feat: hold /health and /ready open after a failed startup - #27612

Open
davidby-influx wants to merge 6 commits into
DSB/startup_errorsfrom
DSB/health_pause_exit
Open

davidby-influx wants to merge 6 commits into
DSB/startup_errorsfrom
DSB/health_pause_exit

Conversation

@davidby-influx

Copy link
Copy Markdown
Contributor

A failed start names the failing subsystem on both endpoints, and the process exits before a scraper can read it. The body lives for microseconds, so the log line was the one durable copy.

--startup-error-linger keeps both endpoints answering for a fixed duration after a failed start, capped at 30 minutes: the window holds the HTTP port on a process a supervisor is waiting to restart. The default of 0 preserves the current behavior.

Teardown splits around the wait. Every subsystem except the HTTP listener and the PID file closes ahead of it, so the bolt flock, the sqlite file and the engine directory belong to the next run. The listener serves the window; the PID file keeps a second influxd from starting against a port this process holds. Shutdown releases both, and consumes each closer as it runs, so no phase can close a store a second time.

The check set freezes ahead of that teardown. A closing store reports its own shutdown as a fresh failure, and failures sort ahead of passes, so it would outrank the startup error the window exists to publish. Check.Freeze bounds each probe on its own, so one wedged subsystem cannot spend the budget the checks after it need.

Supporting changes: Response gains Snapshotter, for a coherent read of a FreshnessResponse; Shutdown aggregates closer failures with errors.Join and runs on the startup-failure path, which leaves no orphaned PID file. HEALTH_READY.md documents the flag, the cap, and the reduced check detail an operator sees during the window under --health-auth-enabled.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Freshness snapshots can still combine fields from different observations when wrapping another stateful response.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a configurable linger window so failed startup diagnostics remain available through /health and /ready.

Changes:

  • Adds --startup-error-linger with a 30-minute cap.
  • Introduces phased shutdown and frozen health snapshots.
  • Documents and tests lifecycle, authorization, and configuration behavior.
File summaries
File Description
kit/check/response.go Adds coherent response snapshots.
kit/check/helpers.go Preserves snapshots through renamed responses.
kit/check/freshness.go Adds freshness snapshots and JSON rendering.
kit/check/freshness_test.go Tests freshness snapshot behavior.
kit/check/freeze_test.go Tests frozen check sets.
kit/check/checktest/checktest.go Adds response normalization helpers.
kit/check/check.go Implements terminal check freezing.
kit/check/check_race_test.go Exercises concurrent freezing.
http/check_handler.go Freezes endpoints and retires auth dependencies.
http/check_handler_freeze_test.go Tests frozen HTTP responses.
HEALTH_READY.md Documents linger behavior and operations.
cmd/influxd/launcher/startup_failure_test.go Tests end-to-end startup failure lingering.
cmd/influxd/launcher/shutdown_test.go Tests phased shutdown and error aggregation.
cmd/influxd/launcher/launcher.go Implements linger and phased teardown.
cmd/influxd/launcher/export_test.go Exposes lifecycle helpers to tests.
cmd/influxd/launcher/cmd.go Adds CLI configuration and failure-path shutdown.
cmd/influxd/launcher/cmd_test.go Tests option resolution and output.
Review details
  • Files reviewed: 17/17 changed files
  • Comments generated: 3
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread HEALTH_READY.md Outdated
Comment thread kit/check/freshness.go
Comment thread cmd/influxd/launcher/startup_failure_test.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Freeze deadlines are not externally enforced, and concurrent calls can execute probes repeatedly.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

kit/check/check.go:227

  • Concurrent Freeze calls can all observe frozen == false and evaluate every checker before one wins the final lock. That contradicts the documented first-freeze-wins/no-op behavior and can run expensive or stateful probes multiple times; the new concurrent test starts four freezers but does not assert invocation counts. Please coordinate the in-progress freeze so only one caller probes and competing callers wait or return.
func (c *Check) Freeze(ctx context.Context) {
	c.mu.RLock()
	frozen := c.frozen
	health := append([]Checker(nil), c.healthChecks...)
	ready := append([]Checker(nil), c.readyChecks...)
	c.mu.RUnlock()
	if frozen {
		return
  • Files reviewed: 17/17 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread kit/check/check.go
@davidby-influx
davidby-influx marked this pull request as ready for review September 1, 2026 18:34
davidby-influx and others added 6 commits September 16, 2026 10:27
A failed start names the failing subsystem on both endpoints, and the
process exits before a scraper can read it. The body lives for
microseconds, so the log line was the one durable copy.

--startup-error-linger keeps both endpoints answering for a fixed
duration after a failed start, capped at 30 minutes: the window holds
the HTTP port on a process a supervisor is waiting to restart. The
default of 0 preserves the current behavior.

Teardown splits around the wait. Every subsystem except the HTTP
listener and the PID file closes ahead of it, so the bolt flock, the
sqlite file and the engine directory belong to the next run. The
listener serves the window; the PID file keeps a second influxd from
starting against a port this process holds. Shutdown releases both, and
consumes each closer as it runs, so no phase can close a store a second
time.

The check set freezes ahead of that teardown. A closing store reports
its own shutdown as a fresh failure, and failures sort ahead of passes,
so it would outrank the startup error the window exists to publish.
Check.Freeze bounds each probe on its own, so one wedged subsystem
cannot spend the budget the checks after it need.

Supporting changes: Response gains Snapshotter, for a coherent read of a
FreshnessResponse; Shutdown aggregates closer failures with errors.Join
and runs on the startup-failure path, which leaves no orphaned PID file.
HEALTH_READY.md documents the flag, the cap, and the reduced check
detail an operator sees during the window under --health-auth-enabled.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Code review sugestion

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api area/2.x OSS 2.0 related issues and PRs team/edge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants