Config Server stability improvements (#1667)

* Remove duplicate tests

* Update tests using Sandbox to use MemoryFileProvider (and without delay), to reduce test flakiness

* Update existing test to verify precedence between options and appsettings

* Make ConfigServerClientOptions reactive to configuration changes

`ConfigServerClientOptions` are now re-evaluated from initial options + configuration on every settings change, instead of being mutated in-place. This prevents stale or torn reads when the provider runs concurrently (e.g. polling timer vs reload).

Key changes:
- The provider clones options on each configuration reload, starting from the initial options passed via code, then applying configuration on top. This ensures code-level defaults are restored when keys are removed from configuration.
- Client settings (spring:cloud:config:*) are no longer written into the provider's data dictionary, eliminating a circular feedback loop where the provider's own output could influence its input.
- Discovery lookup results are tracked separately and applied on top of the options snapshot at load time, rather than mutating shared state.
- Client certificate configuration is moved from the source's `Build` method into `ConfigureConfigServerClientOptions`, so it participates in the options pipeline.
- `ConfigServerClientOptions.Clone()` is introduced to produce isolated snapshots, preventing tearing when options are read during a concurrent reload. A bug was fixed that prevented using global certificates.
- An internal `HttpClientHandler` parameter is threaded through the source and builder extensions, enabling direct handler injection for tests without reflection.
- `IOptionsChangeTokenSource<ConfigServerClientOptions>` is registered in DI so that `IOptionsMonitor` properly triggers on configuration changes.

* Improve test coverage

* Fix references to Config Server in comments

* Fix threading bugs and resource leaks in Config Server provider

The provider had several concurrency and lifecycle issues:

- Vault token renewal timers were created unboundedly on every HTTP request that carried a token, leaking timers that were never disposed. Vault renewal is now managed as a single timer with the same lifecycle as the polling timer.
- Timer management, handler configuration, and disposal could race with each other without synchronization. A lifecycle lock now guards these operations, while non-blocking try-enter locks prevent timer callbacks from queueing up.
- The HTTP client handler's certificates were reconfigured on every HTTP request, racing with concurrent requests sharing the same handler. Certificate and validation configuration is now applied once per settings change under the lifecycle lock, with certificates cleared before re-adding to prevent unbounded accumulation across reloads.
- Options cloning did not copy the certificate issuer chain, losing intermediate CA certificates across reloads.
- Disposal is now coordinated via a volatile flag so that in-flight timer callbacks and Load() calls exit gracefully instead of throwing unobserved exceptions.

* Fixed: use latest options snapshot during discovery in Config Server

* Fixed: preserve original stack trace when load throws

* Fix Config Server health contributor to act on a consistent snapshot

* Fix torn reads in _lastDiscoveryLookupResult

* Fixed: act on single snapshot of _httpClientHandler

* Refresh discovery during polled reload and fix lazy initialization race

Polled configuration reloads now refresh the discovery service lookup before fetching from Config Server, enabling detection of server address changes between polling intervals. Unlike initial load, polled reload does not apply FailFast or retry semantics.
Fixed a race where concurrent calls to LoadInternalAsync could create multiple ConfigServerDiscoveryService instances, each constructing their own temporary DI container and discovery clients.

* Fix change callback accumulation on repeated configuration reloads

Each configuration reload re-registered a new change callback via RegisterChangeCallback, without disposing the previous one. Rapid reloads could accumulate stale callbacks, causing redundant OnSettingsChanged invocations on multiple threads.
Replaced with a single ChangeToken.OnChange registration in the constructor, which automatically handles re-registration and is disposed on shutdown.

* Fix HttpClientHandler mutation race and timer disposal race

Replace shared mutable HttpClientHandler with a per-request factory, eliminating the race where OnSettingsChanged reconfigured a handler concurrently in use by HTTP requests. Production code creates and disposes a fresh handler per request; tests inject a factory for mocking.
Replace _isDisposed flag with CancellationToken-based shutdown, enabling in-flight HTTP requests in timer callbacks to terminate promptly on disposal instead of running to completion.

* Fix threadpool starvation during retries

* Fix timer callbacks potentially using stale options when settings change

* Fix stale reads in ConfigServerDiscoveryService

* Add overloads to post-configure Config Server options

* Cleanup existing tests

* Various fixes

- Improved log messages (and don't log multiple times), fix exception stack traces
- Don't remote-fetch twice at startup (load + timer that immediately fired)
- Moved options bind and remote-fetch from constructor to Load, fix combination with placeholder

* Increase timeout to reduce failures in CI

* Update src/Configuration/test/ConfigServer.Test/ConfigServerConfigurationProviderTest.Loading.cs

Co-authored-by: Tim Hess <tim.hess@broadcom.com>

* Review feedback: replace overrule with override

* Review feedback: Change initial to default options

* Review feedback: remove redundant settings in test

* Review feedback: adjust test name

* Review feedback: remove duplicate configuration

* Review feedback: reformat JSON snippets in tests

---------

Co-authored-by: Tim Hess <tim.hess@broadcom.com>

Author: bart-vmware

src/Bootstrap/src/AutoConfiguration/BootstrapScanner.cs

@@ -91,7 +91,7 @@ public void ConfigureSteeltoe()
 
     private void WireConfigServer()
     {
-        _wrapper.AddConfigServer(_loggerFactory);
+        _wrapper.AddConfigServer(null, _loggerFactory);
 
         LogConfigServerConfigured();
     }

src/Common/src/Certificates/CertificateOptions.cs

@@ -14,6 +14,20 @@ public sealed class CertificateOptions
     internal const string ConfigurationKeyPrefix = "Certificates";
 
     public X509Certificate2? Certificate { get; set; }
-
     public IList<X509Certificate2> IssuerChain { get; } = [];
+
+    internal CertificateOptions Clone()
+    {
+        var clone = new CertificateOptions
+        {
+            Certificate = Certificate
+        };
+
+        foreach (X509Certificate2 issuer in IssuerChain)
+        {
+            clone.IssuerChain.Add(issuer);
+        }
+
+        return clone;
+    }
 }

src/Common/test/Certificates.Test/ConfigureCertificateOptionsTest.cs

@@ -148,7 +148,7 @@ public async Task CertificateOptions_update_on_changed_contents(string certifica
         string secondPrivateKeyContent = await File.ReadAllTextAsync("secondInstance.key", TestContext.Current.CancellationToken);
         using var secondX509 = X509Certificate2.CreateFromPemFile("secondInstance.crt", "secondInstance.key");
         string appSettings = BuildAppSettingsJson(certificateName, certificateFilePath, privateKeyFilePath);
-        string appSettingsPath = sandbox.CreateFile(MemoryFileProvider.DefaultAppSettingsFileName, appSettings);
+        string appSettingsPath = sandbox.CreateFile("appsettings.json", appSettings);
         var configurationBuilder = new ConfigurationBuilder();
         configurationBuilder.AddJsonFile(appSettingsPath, false, true);
         IConfiguration configuration = configurationBuilder.Build();
@@ -167,7 +167,7 @@ public async Task CertificateOptions_update_on_changed_contents(string certifica
         await File.WriteAllTextAsync(privateKeyFilePath, secondPrivateKeyContent, TestContext.Current.CancellationToken);
 
         using Task pollTask = WaitUntilCertificateChangedToAsync(secondX509, optionsMonitor, certificateName, TestContext.Current.CancellationToken);
-        await pollTask.WaitAsync(TimeSpan.FromSeconds(1), TestContext.Current.CancellationToken);
+        await pollTask.WaitAsync(TimeSpan.FromSeconds(5), TestContext.Current.CancellationToken);
 
         optionsMonitor.Get(certificateName).Certificate.Should().Be(secondX509);
     }
@@ -185,7 +185,7 @@ public async Task CertificateOptions_update_on_changed_path(string certificateNa
         string firstPrivateKeyFilePath = sandbox.CreateFile(Guid.NewGuid() + ".key", firstPrivateKeyContent);
         using var secondX509 = X509Certificate2.CreateFromPemFile("secondInstance.crt", "secondInstance.key");
         string appSettings = BuildAppSettingsJson(certificateName, firstCertificateFilePath, firstPrivateKeyFilePath);
-        string appSettingsPath = sandbox.CreateFile(MemoryFileProvider.DefaultAppSettingsFileName, appSettings);
+        string appSettingsPath = sandbox.CreateFile("appsettings.json", appSettings);
         var configurationBuilder = new ConfigurationBuilder();
         configurationBuilder.AddJsonFile(appSettingsPath, false, true);
         IConfiguration configuration = configurationBuilder.Build();

src/Common/test/Common.Test/ApplicationInstanceInfoTest.cs

@@ -7,7 +7,6 @@
 using Microsoft.Extensions.DependencyInjection;
 using Steeltoe.Common.Extensions;
 using Steeltoe.Common.TestResources;
-using Steeltoe.Common.TestResources.IO;
 
 namespace Steeltoe.Common.Test;
 
@@ -24,7 +23,7 @@ public void ConstructorSetsDefaults()
     [Fact]
     public async Task ReadsApplicationConfiguration()
     {
-        const string configJson = """
+        const string appSettings = """
             {
               "Spring": {
                 "Application": {
@@ -34,13 +33,11 @@ public async Task ReadsApplicationConfiguration()
             }
             """;
 
-        using var sandbox = new Sandbox();
-        string path = sandbox.CreateFile(MemoryFileProvider.DefaultAppSettingsFileName, configJson);
-        string directory = Path.GetDirectoryName(path)!;
-        string fileName = Path.GetFileName(path);
+        var fileProvider = new MemoryFileProvider();
+        fileProvider.IncludeAppSettingsJsonFile(appSettings);
+
         var builder = new ConfigurationBuilder();
-        builder.SetBasePath(directory);
-        builder.AddJsonFile(fileName);
+        builder.AddInMemoryAppSettingsJsonFile(fileProvider);
         IConfiguration configuration = builder.Build();
 
         var services = new ServiceCollection();

src/Common/test/TestResources/MemoryFileProvider.cs

@@ -12,8 +12,6 @@ namespace Steeltoe.Common.TestResources;
 
 public sealed class MemoryFileProvider : IFileProvider
 {
-    public const string DefaultAppSettingsFileName = "appsettings.json";
-
     private static readonly char[] DirectorySeparators =
     [
         Path.DirectorySeparatorChar,

src/Common/test/TestResources/MemoryFileProviderAppSettingsExtensions.cs

@@ -0,0 +1,36 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the Apache 2.0 License.
+// See the LICENSE file in the project root for more information.
+
+namespace Steeltoe.Common.TestResources;
+
+public static class MemoryFileProviderAppSettingsExtensions
+{
+    public static void IncludeAppSettingsJsonFile(this MemoryFileProvider fileProvider, string contents)
+    {
+        ArgumentNullException.ThrowIfNull(fileProvider);
+
+        fileProvider.IncludeFile(MemoryFileProviderConfigurationBuilderExtensions.AppSettingsJsonFileName, contents);
+    }
+
+    public static void IncludeAppSettingsXmlFile(this MemoryFileProvider fileProvider, string contents)
+    {
+        ArgumentNullException.ThrowIfNull(fileProvider);
+
+        fileProvider.IncludeFile(MemoryFileProviderConfigurationBuilderExtensions.AppSettingsXmlFileName, contents);
+    }
+
+    public static void IncludeAppSettingsIniFile(this MemoryFileProvider fileProvider, string contents)
+    {
+        ArgumentNullException.ThrowIfNull(fileProvider);
+
+        fileProvider.IncludeFile(MemoryFileProviderConfigurationBuilderExtensions.AppSettingsIniFileName, contents);
+    }
+
+    public static void ReplaceAppSettingsJsonFile(this MemoryFileProvider fileProvider, string contents)
+    {
+        ArgumentNullException.ThrowIfNull(fileProvider);
+
+        fileProvider.ReplaceFile(MemoryFileProviderConfigurationBuilderExtensions.AppSettingsJsonFileName, contents);
+    }
+}

src/Common/test/TestResources/MemoryFileProviderConfigurationBuilderExtensions.cs

@@ -0,0 +1,77 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the Apache 2.0 License.
+// See the LICENSE file in the project root for more information.
+
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.Configuration.Ini;
+using Microsoft.Extensions.Configuration.Json;
+using Microsoft.Extensions.Configuration.Xml;
+
+namespace Steeltoe.Common.TestResources;
+
+public static class MemoryFileProviderConfigurationBuilderExtensions
+{
+    internal const string AppSettingsJsonFileName = "appsettings.json";
+    internal const string AppSettingsXmlFileName = "appsettings.xml";
+    internal const string AppSettingsIniFileName = "appsettings.ini";
+
+    public static void AddInMemoryAppSettingsJsonFile(this IConfigurationBuilder builder, MemoryFileProvider fileProvider)
+    {
+        AddInMemoryJsonFile(builder, fileProvider, AppSettingsJsonFileName);
+    }
+
+    public static void AddInMemoryJsonFile(this IConfigurationBuilder builder, MemoryFileProvider fileProvider, string path)
+    {
+        ArgumentNullException.ThrowIfNull(builder);
+        ArgumentNullException.ThrowIfNull(fileProvider);
+        ArgumentException.ThrowIfNullOrEmpty(path);
+
+        var source = new JsonConfigurationSource
+        {
+            FileProvider = fileProvider,
+            Path = path,
+            Optional = false,
+            ReloadOnChange = true,
+            // Turn off debounce, so the change token triggers immediately. Then we don't need to sleep in tests.
+            ReloadDelay = 0
+        };
+
+        builder.Add(source);
+    }
+
+    public static void AddInMemoryAppSettingsXmlFile(this IConfigurationBuilder builder, MemoryFileProvider fileProvider)
+    {
+        ArgumentNullException.ThrowIfNull(builder);
+        ArgumentNullException.ThrowIfNull(fileProvider);
+
+        var source = new XmlConfigurationSource
+        {
+            FileProvider = fileProvider,
+            Path = AppSettingsXmlFileName,
+            Optional = false,
+            ReloadOnChange = true,
+            // Turn off debounce, so the change token triggers immediately. Then we don't need to sleep in tests.
+            ReloadDelay = 0
+        };
+
+        builder.Add(source);
+    }
+
+    public static void AddInMemoryAppSettingsIniFile(this IConfigurationBuilder builder, MemoryFileProvider fileProvider)
+    {
+        ArgumentNullException.ThrowIfNull(builder);
+        ArgumentNullException.ThrowIfNull(fileProvider);
+
+        var source = new IniConfigurationSource
+        {
+            FileProvider = fileProvider,
+            Path = AppSettingsIniFileName,
+            Optional = false,
+            ReloadOnChange = true,
+            // Turn off debounce, so the change token triggers immediately. Then we don't need to sleep in tests.
+            ReloadDelay = 0
+        };
+
+        builder.Add(source);
+    }
+}

src/Configuration/src/Abstractions/CompositeConfigurationProvider.cs

@@ -53,10 +53,9 @@ private void Load(bool isReload)
 
     public IEnumerable<string> GetChildKeys(IEnumerable<string> earlierKeys, string? parentPath)
     {
+        ArgumentNullException.ThrowIfNull(earlierKeys);
+
         string[] earlierKeysArray = earlierKeys as string[] ?? earlierKeys.ToArray();
-#pragma warning disable S3236 // Caller information arguments should not be provided explicitly
-        ArgumentNullException.ThrowIfNull(earlierKeysArray, nameof(earlierKeys));
-#pragma warning restore S3236 // Caller information arguments should not be provided explicitly
 
         if (_logger.IsEnabled(LogLevel.Trace))
         {

src/Configuration/src/ConfigServer/ConfigServerClientOptions.cs

@@ -17,10 +17,14 @@ public sealed class ConfigServerClientOptions : IValidateCertificatesOptions
     private const char CommaDelimiter = ',';
     internal const string ConfigurationPrefix = "spring:cloud:config";
 
-    internal CertificateOptions ClientCertificate { get; } = new();
     internal TimeSpan HttpTimeout => TimeSpan.FromMilliseconds(Timeout);
     internal bool IsMultiServerConfiguration => Uri != null && Uri.Contains(CommaDelimiter);
 
+    /// <summary>
+    /// Gets or sets the client certificate used for mutual TLS authentication with the Config Server.
+    /// </summary>
+    internal CertificateOptions ClientCertificate { get; set; } = new();
+
     /// <summary>
     /// Gets or sets a value indicating whether the Config Server provider is enabled. Default value: true.
     /// </summary>
@@ -35,7 +39,7 @@ public sealed class ConfigServerClientOptions : IValidateCertificatesOptions
     /// Gets or sets a comma-separated list of environments used when accessing configuration data. Default value: "Production".
     /// </summary>
     [ConfigurationKeyName("Env")]
-    public string? Environment { get; set; } = "Production";
+    public string? Environment { get; set; }
 
     /// <summary>
     /// Gets or sets a comma-separated list of labels to request from the server.
@@ -96,17 +100,17 @@ public bool ValidateCertificatesAlt
     /// <summary>
     /// Gets retry settings.
     /// </summary>
-    public ConfigServerRetryOptions Retry { get; } = new();
+    public ConfigServerRetryOptions Retry { get; private set; } = new();
 
     /// <summary>
     /// Gets service discovery settings.
     /// </summary>
-    public ConfigServerDiscoveryOptions Discovery { get; } = new();
+    public ConfigServerDiscoveryOptions Discovery { get; private set; } = new();
 
     /// <summary>
     /// Gets health check settings.
     /// </summary>
-    public ConfigServerHealthOptions Health { get; } = new();
+    public ConfigServerHealthOptions Health { get; private set; } = new();
 
     /// <summary>
     /// Gets or sets the address used by the provider to obtain a OAuth Access Token.
@@ -129,7 +133,7 @@ public bool ValidateCertificatesAlt
     public int TokenTtl { get; set; } = 300_000;
 
     /// <summary>
-    /// Gets or sets the vault token renew rate (in milliseconds). Default value: 60_000 (1 minute).
+    /// Gets or sets the Vault token renew rate (in milliseconds). Default value: 60_000 (1 minute).
     /// </summary>
     public int TokenRenewRate { get; set; } = 60_000;
 
@@ -141,7 +145,52 @@ public bool ValidateCertificatesAlt
     /// <summary>
     /// Gets headers that will be added to the Config Server request.
     /// </summary>
-    public IDictionary<string, string> Headers { get; } = new Dictionary<string, string>();
+    public IDictionary<string, string> Headers { get; private set; } = new Dictionary<string, string>();
+
+    internal ConfigServerClientOptions Clone()
+    {
+        return new ConfigServerClientOptions
+        {
+            ClientCertificate = ClientCertificate.Clone(),
+            Enabled = Enabled,
+            FailFast = FailFast,
+            Environment = Environment,
+            Label = Label,
+            Name = Name,
+            Uri = Uri,
+            Username = Username,
+            Password = Password,
+            Token = Token,
+            Timeout = Timeout,
+            PollingInterval = PollingInterval,
+            ValidateCertificates = ValidateCertificates,
+            Retry = new ConfigServerRetryOptions
+            {
+                Enabled = Retry.Enabled,
+                InitialInterval = Retry.InitialInterval,
+                MaxInterval = Retry.MaxInterval,
+                Multiplier = Retry.Multiplier,
+                MaxAttempts = Retry.MaxAttempts
+            },
+            Discovery = new ConfigServerDiscoveryOptions
+            {
+                Enabled = Discovery.Enabled,
+                ServiceId = Discovery.ServiceId
+            },
+            Health = new ConfigServerHealthOptions
+            {
+                Enabled = Health.Enabled,
+                TimeToLive = Health.TimeToLive
+            },
+            AccessTokenUri = AccessTokenUri,
+            ClientSecret = ClientSecret,
+            ClientId = ClientId,
+            TokenTtl = TokenTtl,
+            TokenRenewRate = TokenRenewRate,
+            DisableTokenRenewal = DisableTokenRenewal,
+            Headers = new Dictionary<string, string>(Headers)
+        };
+    }
 
     internal List<Uri> GetUris()
     {