[Bug]: Metric query execution has no configurable timeout, silently capped at a low hardcoded default

[behzad-taghipour-hs]

Description

Metric queries can time out well before the system's actual per-request time budget is reached. The step that executes a metric's query doesn't specify how long it's allowed to run, so it silently falls back to a low, hardcoded default built into the query executor — far shorter than the overall time the system otherwise allows for a request. This makes metric queries fail on sources or queries that are simply a bit slower than usual, even though the request as a whole still has plenty of time budget left, and there's no way to tune this without a code change.

Environment

• Commit ID: 42b5c6b (v0.2.2)
• Environment: prod

Step to reproduce

• Run a metric query whose execution against its data source takes longer than ~10 seconds but well under the overall request timeout
• Observe the query fail with an execution timeout, even though the overall request budget was not close to exhausted

A root cause

The metric-execution step calls the query executor without passing an explicit execution timeout. The executor's own default (a low, fixed value sized for ad-hoc/exploratory calls) applies instead, regardless of how much of the overall request's time budget is actually still available. A comparable, later-added query path in the system already passes its own explicit, environment-configurable timeout for exactly this reason — this one was never wired up the same way.

Context

The overall per-request time budget is deliberately generous and configurable. This step bypassed that intent by not forwarding any timeout at all, so it was bound instead by an executor-internal default never meant to be a system-wide limit.

Risks & Known Unknowns

• Any metric query against a source with variable or occasionally elevated latency is at risk of hitting this same low ceiling, independent of source health.
• The new configurable default was chosen conservatively; it may need tuning based on observed p95/p99 execution times once more data is available.

[noah-paige]

Tier-1 metric execution will now pass an explicit timeout instead of falling through to the query executor's short, fixed internal default. The new default is 35 seconds, exposed as a validated, environment-configurable override, and each execution is clamped to whatever remains of the overall request's time budget — if less than a second is left, Tier-1 skips execution and falls through rather than firing a query that can't possibly finish. The effective timeout used is now recorded in both the success and failure paths, and the configured value is logged on startup, so it's visible without a code change going forward.