Spanner Outbox/Inbox tests fail against the emulator (missing EmulatorDetection on test SpannerConnection)

[iancooper]

Summary

When run against the Spanner emulator (per setup-spanner-emulator.sh), 60 Spanner Outbox and Inbox tests fail at fixture setup with System.InvalidOperationException : Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc. — the test helpers construct SpannerConnection directly from the connection string, which does not opt into EmulatorDetection, so the gRPC client tries to load ADC instead of routing to SPANNER_EMULATOR_HOST.

The fix is small and contained: route Outbox/Inbox test helpers through SpannerConnectionStringBuilder { EmulatorDetection = EmulatorDetection.EmulatorOrProduction } the way the BoxProvisioning Spanner tests already do.

Repro

docker compose -f docker-compose-spanner.yaml up -d
bash ./setup-spanner-emulator.sh
SPANNER_EMULATOR_HOST=localhost:9010 GOOGLE_CLOUD_PROJECT=brighter-tests \
  dotnet test tests/Paramore.Brighter.Gcp.Tests/Paramore.Brighter.Gcp.Tests.csproj -f net9.0 \
  --filter "FullyQualifiedName~Spanner"

Result on multi-tenancy@c6335eb00: Failed: 60, Passed: 33, Total: 93. The 33 passing are exactly the Spanner/BoxProvisioning/ tests.

Failure breakdown

Group	Count
Paramore.Brighter.Gcp.Tests.Outbox.SpannerBinary.{Sync,Async}	24
Paramore.Brighter.Gcp.Tests.Outbox.SpannerText.{Sync,Async}	24
Paramore.Brighter.Gcp.Tests.Spanner.Inbox.SpannerInbox{,Async}Test	12
Total	60

All fail with the identical "credentials were not found" message at fixture-init time.

Root cause (located)

The failing test helpers construct the connection without enabling emulator detection. Example — tests/Paramore.Brighter.Gcp.Tests/Spanner/Inbox/SpannerInboxAsyncTest.cs:23:

protected override async Task CreateInboxTableAsync(RelationalDatabaseConfiguration configuration)
{
    await using var connection = new SpannerConnection(configuration.ConnectionString);  // <-- no EmulatorDetection
    await connection.OpenAsync();
    ...
}

The Spanner BoxProvisioning tests (which all pass) use the pattern that works. Example — tests/Paramore.Brighter.Gcp.Tests/Spanner/BoxProvisioning/When_spanner_outbox_provisioner_runs_on_fresh_database_it_should_create_outbox_table.cs:

private SpannerConnection CreateConnection()
{
    var builder = new SpannerConnectionStringBuilder(_connectionString)
    {
        EmulatorDetection = EmulatorDetection.EmulatorOrProduction
    };
    return new SpannerConnection(builder);
}

EmulatorDetection.EmulatorOrProduction makes the SDK check SPANNER_EMULATOR_HOST and bypass ADC when set; without it, the SDK falls back to ADC and fails when no creds are configured.

Proposed fix

Apply the EmulatorDetection.EmulatorOrProduction connection-builder pattern to the Outbox/Inbox test setup helpers (test-only change). Candidate files:

• tests/Paramore.Brighter.Gcp.Tests/Spanner/Inbox/SpannerInboxTest.cs (sync helpers)
• tests/Paramore.Brighter.Gcp.Tests/Spanner/Inbox/SpannerInboxAsyncTest.cs (async helpers)
• The Outbox test base/builder helpers used by Outbox/Spanner{Binary,Text}/{Sync,Async}/ — likely tests/Paramore.Brighter.Gcp.Tests/Outbox/... shared fixture (worth a grep -rn 'new SpannerConnection' tests/Paramore.Brighter.Gcp.Tests).

Consider extracting the builder pattern into a tests/Paramore.Brighter.Gcp.Tests/Spanner/SpannerTestConnectionFactory.cs helper used by every test that constructs a SpannerConnection.

Out-of-scope question for a future ADR: should the production-side SpannerInbox / SpannerOutbox honour SPANNER_EMULATOR_HOST by default, or is that strictly a test concern? Today the test-only fix is the right scope.

Setup-script gap (not the root cause but worth noting)

setup-spanner-emulator.sh exports env vars in its own subshell. Running bash ./setup-spanner-emulator.sh does NOT export to the caller; you must either source ./setup-spanner-emulator.sh or prefix env inline. This is documented at the end of the script, but a reader may miss it. (The emulator detection fix above subsumes this — once EmulatorDetection is on the connection, the env var is the SDK's contract, not the test runner's.)

Out-of-scope context

Uncovered during the VERIFY phase of spec 0029 (multi-tenancy migration-history scope) on branch multi-tenancy (PR #4155). That spec touches only src/Paramore.Brighter.BoxProvisioning*/ — zero Spanner Outbox/Inbox code — so the failures are pre-existing and orthogonal. The Spanner/BoxProvisioning/ subset (--filter "FullyQualifiedName~Spanner.BoxProvisioning") is 33/33 on both TFMs on the same branch.

[iancooper]

See also #4131

[iancooper]

Hit again on replay_on_seen (spec 0027, Task 19f) — and a working reference for the fix

Encountered this exact failure again on branch replay_on_seen while implementing Task 19f of spec 0027 (Spanner inbox implements IAmACausationTrackingInbox). The pre-existing SpannerInboxTest / SpannerInboxAsyncTest are red here for the identical reason — new SpannerConnection(configuration.ConnectionString) with default EmulatorDetection=None → System.InvalidOperationException : Your default credentials were not found. at fixture setup, 1ms before any RPC. SPANNER_EMULATOR_HOST alone is not honoured.

The proposed fix works — confirmed

The new test added in Task 19f (SpannerCausationTrackingInboxTest : CausationTrackingInboxBaseTests) adopts exactly the connection-builder pattern this issue recommends, and passes 4/4 on net9.0 + net10.0 against the emulator:

var connectionString = new SpannerConnectionStringBuilder(Const.ConnectionString)
{
    EmulatorDetection = EmulatorDetection.EmulatorOrProduction
}.ConnectionString;

_configuration = new RelationalDatabaseConfiguration(
    connectionString,
    inboxTableName: $"{Const.TablePrefix}{Uuid.New():N}");

Note the variation: rather than building a SpannerConnection from the builder directly (as the BoxProvisioning tests do), I embed EmulatorDetection into the connection string itself. This matters because the value has to round-trip through RelationalDatabaseConfiguration so the inbox's internal connections (SpannerConnectionProvider → new SpannerConnection(connStr)) pick it up too, not just the DDL-setup connections the test controls directly. SpannerConnectionStringBuilder.EmulatorDetection does serialise into .ConnectionString, so this works. The proposed SpannerTestConnectionFactory helper should expose the connection-string form for cases where the connection is built downstream of a RelationalDatabaseConfiguration.

Note for whoever takes this

While the fix here is test-only (the framing of this issue), the production-side SpannerInbox/SpannerOutbox path surfaced a separate real gap I had to fix in prod code for Task 19f: Spanner rejects an untyped DBNull parameter (ArgumentException: SpannerDbType must be set...), so a null CausationId needs SpannerDbType.String set explicitly on the parameter. Unrelated to emulator detection, but lives in the same Spanner inbox code — worth being aware of if the eventual fix touches SpannerInboxAsync/SpannerOutbox parameter handling.