query: retry initial DNS resolution instead of becoming ready with missing endpoints

The initial resolution ran once and a failure was only logged, so a
transient DNS error at startup let the querier become ready with a
partial endpoint list. Retry every 2s until every address resolves or
--store.sd-dns-interval elapses.

Signed-off-by: Sang Hyun Lee <shyundev@gmail.com>

Author: shyundev

CHANGELOG.md

@@ -18,6 +18,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
 
 ### Fixed
 
+- [#9033](https://github.com/thanos-io/thanos/pull/9033): Query: Retry the initial DNS resolution of endpoints for up to `--store.sd-dns-interval` instead of becoming ready with a partial endpoint list after a transient DNS failure.
 - [#9014](https://github.com/thanos-io/thanos/pull/9014): Reloader: Optimize `Watch` to allocate constant memory despite the config size.
 - [#8990](https://github.com/thanos-io/thanos/pull/8990): Receive: Avoid a panic when pruning starts before a tenant TSDB is ready.
 - [#8968](https://github.com/thanos-io/thanos/pull/8968): *: Bump `google.golang.org/grpc` to v1.82.1 to fix GHSA-hrxh-6v49-42gf (CVSS 8.6): HTTP/2 Rapid Reset DoS bypass, xDS RBAC authorization bypass, and NOT-rule panic.

cmd/thanos/endpointset.go

@@ -370,6 +370,8 @@ func setupEndpointSet(
 
 	// Perform initial DNS resolution before starting periodic updates.
 	// This ensures that DNS providers have addresses when the first endpoint update runs.
+	// Resolution is retried until every address resolves or the interval elapses, so that
+	// the first update sees the full endpoint list even when DNS fails transiently at startup.
 	{
 		resolveCtx, resolveCancel := context.WithTimeout(context.Background(), dnsSDInterval)
 		defer resolveCancel()
@@ -387,7 +389,13 @@ func setupEndpointSet(
 		}
 		// Note: legacyFileSDCache will be empty at this point since file SD hasn't started yet
 		if len(addresses) > 0 {
-			if err := dnsEndpointProvider.Resolve(resolveCtx, addresses, true); err != nil {
+			if err := runutil.Retry(2*time.Second, resolveCtx.Done(), func() error {
+				if err := dnsEndpointProvider.Resolve(resolveCtx, addresses, true); err != nil {
+					level.Warn(logger).Log("msg", "initial DNS resolution failed, retrying", "err", err)
+					return err
+				}
+				return nil
+			}); err != nil {
 				level.Error(logger).Log("msg", "initial DNS resolution failed", "err", err)
 			}
 		}