security: TestUseCerts failed

[github-actions]

security.TestUseCerts failed on master @ 34dec8d534fee21c1bff5d474a55649b53e574f1:

        syscall.read(0xc003a1fb60?, {0xc00482c800?, 0xc00b2fb000?, 0xc00b2fb050?})
        	GOROOT/src/syscall/zsyscall_linux_amd64.go:736 +0x38
        syscall.Read(...)
        	GOROOT/src/syscall/syscall_unix.go:183
        internal/poll.ignoringEINTRIO(...)
        	GOROOT/src/internal/poll/fd_unix.go:738
        internal/poll.(*FD).Read(0xc003a1fb60, {0xc00482c800, 0x3b5, 0x3b5})
        	GOROOT/src/internal/poll/fd_unix.go:161 +0x2ae
        os.(*File).read(...)
        	GOROOT/src/os/file_posix.go:29
        os.(*File).Read(0xc0102a00c0, {0xc00482c800?, 0x563d1a0?, 0xc0102a0001?})
        	GOROOT/src/os/file.go:144 +0x4f
        os.readFileContents(0x3b4, 0xc00b2fb1c8)
        	GOROOT/src/os/file.go:911 +0x90
        os.ReadFile({0xc00cc11bc0?, 0x6274743?})
        	GOROOT/src/os/file.go:874 +0xd3
        crypto/x509.loadSystemRoots()
        	GOROOT/src/crypto/x509/root_unix.go:70 +0x38a
        crypto/x509.initSystemRoots()
        	GOROOT/src/crypto/x509/root.go:40 +0x56
        sync.(*Once).doSlow(0xc00b2fb3d8?, 0x784cf1?)
        	GOROOT/src/sync/once.go:78 +0xac
        sync.(*Once).Do(...)
        	GOROOT/src/sync/once.go:69
        crypto/x509.systemRootsPool()
        	GOROOT/src/crypto/x509/root.go:31 +0x45
        crypto/x509.(*Certificate).Verify(0xc00c77a608, {{0xc00d667780, 0x9}, 0xc00f48a6c0, 0x0, {0xc266665e4f6b3a9e, 0x12fcead1d, 0xb14ba00}, {0x0, 0x0, ...}, ...})
        	GOROOT/src/crypto/x509/verify.go:857 +0x15d
        crypto/tls.(*Conn).verifyServerCertificate(0xc00b25bc08, {0xc00ed970b0, 0x1, 0x1})
        	GOROOT/src/crypto/tls/handshake_client.go:1171 +0x9fd
        crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate(0xc00b2fbc48)
        	GOROOT/src/crypto/tls/handshake_client_tls13.go:659 +0x272
        crypto/tls.(*clientHandshakeStateTLS13).handshake(0xc00b2fbc48)
        	GOROOT/src/crypto/tls/handshake_client_tls13.go:137 +0x76c
        crypto/tls.(*Conn).clientHandshake(0xc00b25bc08, {0x6fee560, 0xc00b277a40})
        	GOROOT/src/crypto/tls/handshake_client.go:367 +0x80e
        crypto/tls.(*Conn).handshakeContext(0xc00b25bc08, {0x6fee560, 0xc00b276910})
        	GOROOT/src/crypto/tls/conn.go:1575 +0x372
        crypto/tls.(*Conn).HandshakeContext(...)
        	GOROOT/src/crypto/tls/conn.go:1515
        net/http.(*persistConn).addTLS.func2()
        	GOROOT/src/net/http/transport.go:1706 +0x6e
        created by net/http.(*persistConn).addTLS in goroutine 4706
        	GOROOT/src/net/http/transport.go:1702 +0x309
        Leaked goroutine: goroutine 4710 [select]:
        crypto/tls.(*Conn).handshakeContext.func2()
        	GOROOT/src/crypto/tls/conn.go:1551 +0x86
        created by crypto/tls.(*Conn).handshakeContext in goroutine 4709
        	GOROOT/src/crypto/tls/conn.go:1550 +0x265
--- FAIL: TestUseCerts (16.46s)

Parameters:

• attempt=1
• run=1
• shard=1

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/security-engineering

This test on roachdash | Improve this report!

Jira issue: CRDB-61690

[msbutler]

/investigate

[github-actions]

Investigation: security.TestUseCerts

Investigated failure: #165944
Failure SHA: 34dec8d534fee21c1bff5d474a55649b53e574f1
Confidence: high

What This Test Does

TestUseCerts validates that a CockroachDB server correctly enforces TLS certificate authentication. It starts a server with generated certs, first makes an insecure HTTP request (expecting failure), then makes a properly-authenticated request (expecting success), and finally verifies a KV write.

Where the Failure Occurs

The test fails in the leaktest.AfterTest cleanup at certs_test.go:497 due to three leaked goroutines. The test logic itself passes — the failure is purely from goroutine leak detection.

The leaked goroutines are:

1. goroutine 4709 — stuck in syscall.read inside crypto/x509.loadSystemRoots() → os.ReadFile, reading system CA certificates from disk
2. goroutine 4706 — net/http.(*persistConn).addTLS blocked on chan receive, waiting for the TLS handshake goroutine (4709) to complete
3. goroutine 4710 — crypto/tls.(*Conn).handshakeContext.func2 in select, a context-cancellation watcher for the handshake

Analysis

Likely cause (high confidence): Premature removal of TLS handshake goroutine leak exclusion from leaktest, combined with slow system root CA loading in the CI environment.

The chain of events:

1. The test creates an "insecure" HTTP client at certs_test.go:442-452 and makes a request to the HTTPS server, expecting it to fail with a TLS error.

2. When Insecure: true, getUIClientTLSConfig() returns nil (tls.go:170-172), so the HTTP transport uses Go's default TLS config, which performs full certificate verification including loading system root CAs via crypto/x509.loadSystemRoots().

3. loadSystemRoots() reads system CA certificate files from disk (e.g., /etc/ssl/certs/ca-certificates.crt). In the EngFlow CI environment, this read can be slow — the goroutine gets stuck in syscall.read for longer than the HTTP client's 10-second timeout.

4. The HTTP request times out and returns an error (which the test expects). However, the TLS handshake goroutines are abandoned — loadSystemRoots is stuck in an uninterruptible syscall, and the goroutines waiting on it cannot be cancelled.

5. The leaktest.AfterTest cleanup detects these goroutines after waiting 5 seconds and fails the test.

The direct trigger was commit c921910f92f (PR #164427), merged on 2026-02-26, which removed the net/http.(*persistConn).addTLS and crypto/tls.(*Conn).handshakeContext goroutine leak exclusions from leaktest.go. The exclusion was removed under the assumption that the Go stdlib fix (golang/go#62227, merged in Go 1.23) resolved the underlying issue. However, the Go fix addressed a different aspect of the TLS handshake goroutine lifecycle — it did not fix the case where loadSystemRoots gets stuck in a slow/blocking file read, which is an environment-dependent issue.

Potential fixes:

1. Re-add the leaktest exclusion for net/http.(*persistConn).addTLS and crypto/tls.(*Conn).handshakeContext (reverting the removal), with an updated comment explaining the environment-specific root cause.

2. Close the HTTP client's transport after the insecure request fails, to actively clean up connections. Add httpClient.CloseIdleConnections() after line 462 in certs_test.go. However, this may not help since the goroutines are stuck in a syscall, not idle.

3. Set InsecureSkipVerify: true on the insecure client's TLS config instead of relying on a nil TLS config. This would skip certificate verification entirely (including loadSystemRoots) and still test the server's rejection of the unauthenticated client. This would be the most targeted fix.

4. Use SSL_CERT_FILE or SSL_CERT_DIR env vars to point to a small/empty CA bundle for the test, avoiding the slow system root loading entirely.

Related Issues and PRs

• #142448 (closed) — Same failure, same stack trace. Was closed by PR #164427 which removed the leaktest exclusion. Reopening as security: TestUseCerts failed #165944 shows this fix was premature.
• #119052 (closed) — Same failure pattern. A team member noted: "The failure is not consistently reproducible... We suspect that somehow the system is overloaded by other workloads and it just takes more than 30 secs to read files from disk." Fixed by PR #129802 which originally added the leaktest exclusion.
• #128214 (closed) — TestJWTAuthWithCustomCACert with the same loadSystemRoots goroutine leak pattern, also fixed by PR security,jwtauthccl: Fix test failures due to leaked goroutines #129802.
• #129073, #131498, #133994 (closed) — Earlier occurrences on release branches.

Timeline

Date	Event
2024-08-29	PR #129802 adds addTLS and handshakeContext exclusions to leaktest
2025-03-07	Issue #142448 opened — same failure reappears
2025-03-10	PR #164427 (commit 54ff465e9bd) re-adds exclusion for #142448
2026-02-26	PR #164427 (commit c921910f92f) removes the exclusion, believing Go 1.23+ fixed the underlying issue
2026-03-18	Issue #165944 opened — failure recurs 20 days after exclusion removal

Detailed evidence and investigation notes

Log Excerpts

From the EngFlow test log (shard 0), the test itself runs to completion but fails on leak detection:

=== RUN   TestUseCerts
    test_log_scope.go:172: test logs captured to: outputs.zip/logTestUseCerts1646180610
    test_log_scope.go:82: use -show-logs to present logs inline
    test_server_shim.go:95: cluster virtualization disabled due to issue: #109498 (expected label: C-bug)
    certs_test.go:497: -- test log scope end --
    certs_test.go:497:
        all stacks:
        ...
        goroutine 4709 [syscall]:
        syscall.Syscall(0x0, 0x8, 0xc00482c800, 0x3b5)
        ...
        os.ReadFile({0xc00cc11bc0?, 0x6274743?})
        	GOROOT/src/os/file.go:874 +0xd3
        crypto/x509.loadSystemRoots()
        	GOROOT/src/crypto/x509/root_unix.go:70 +0x38a
        ...
        Leaked goroutine: goroutine 4706 [chan receive]:
        net/http.(*persistConn).addTLS(...)
        ...
        Leaked goroutine: goroutine 4709 [syscall]:
        ...crypto/x509.loadSystemRoots()...
        Leaked goroutine: goroutine 4710 [select]:
        crypto/tls.(*Conn).handshakeContext.func2()
--- FAIL: TestUseCerts (16.46s)

Code References

The insecure client setup at certs_test.go:442-462:

• Creates an insecure SecurityContext
• Gets an HTTP client with a nil TLS config (Go default)
• Makes a request to the HTTPS server, which triggers full TLS verification including loadSystemRoots

The removed leaktest exclusion (commit ecc4d006d93):

// Ignore TLS handshake related goroutine.
strings.Contains(stack, "net/http.(*persistConn).addTLS") ||
strings.Contains(stack, "crypto/tls.(*Conn).handshakeContext") ||

Things Ruled Out

• Test logic bug — The test logic itself works correctly. The insecure request fails as expected, the secure request succeeds. The failure is purely from leaked goroutine detection.
• Go 1.23+ fix resolving the issue — The Go fix (net/http: remove persistConn reference from wantConn golang/go#62227) addressed lifecycle cleanup of the handshakeContext watcher goroutine. It did not address the case where loadSystemRoots gets stuck in a blocking syscall on slow disk, which is the root cause here.
• Code changes to certs_test.go — There have been no meaningful changes to TestUseCerts since the exclusion was removed.

Other Failure Occurrences

All failures across #165944, #142448, #119052, #129073, #131498, and #133994 show the same failure mode: goroutine leak in loadSystemRoots → os.ReadFile → syscall.read, with associated addTLS and handshakeContext goroutines. This is a single, consistent failure pattern that recurs in CI environments when disk I/O is slow.

Workflow run

Tooling Feedback

No issues with tooling. All necessary tools were available and functional.

---

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[github-actions]

Auto-solver has created a draft PR to address this issue: #166085

Please review the changes carefully before approving.

Workflow run