basalt-26.3: roachtest: activerecord failed

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.activerecord failed with artifacts on basalt-26.3 @ 410866d934770553b0ff008caa721a72632dbaaa:

Failed with:

(orm_helpers.go:232).summarizeFailed: 
Tests run on Cockroach v26.3.2-dev-410866d934770553b0ff008caa721a72632dbaaa
Tests run against activerecord 8.1.0
8905 Total Tests Run
8901 tests passed
4 tests failed
45 tests skipped
60 tests ignored
0 tests passed unexpectedly
3 tests failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- PASS: SerializedAttributeTestWithYamlSafeLoad#test_unexpected_serialized_type (expected)
--- FAIL: CalculationsTest#test_should_calculate_grouped_by_function_with_table_alias (unexpected)
--- FAIL: CalculationsTest#test_should_calculate_grouped_by_function (unexpected)
--- PASS: SerializedAttributeTest#test_unexpected_serialized_type (expected)
--- FAIL: CalculationsTest#test_pluck_loaded_relation_sql_fragment (unexpected)
For a full summary look at the activerecord artifacts 
An updated blocklist (activeRecordBlocklist) is available in the artifacts' activerecord log
test artifacts and logs in: /artifacts/activerecord/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21524604-1789884520-14-n1cpu4-0001		10.91.100.11

Parameters:

• MetamorphicBasaltMultiAZ=false
• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=true
• metamorphicDRPC=true
• metamorphicLeases=default
• metamorphicWriteBuffering=true
• runtimeAssertionsBuild=true
• storageNodesVersion=v0.1.20-nightly.20260918

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Logs were not uploaded to Datadog

/cc @cockroachlabs/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-68696

[celiala]

/investigate

[github-actions]

Investigation: roachtest/activerecord (CalculationsTest)

Investigated failure: issue #175682 body
Failure SHA: 410866d934770553b0ff008caa721a72632dbaaa (basalt-26.3)
TeamCity build: 21524604
Confidence: moderate

What This Test Does

The activerecord roachtest runs the upstream Rails ActiveRecord ORM test
suite (v8.1.0) against a single-node CockroachDB cluster and compares the
results to a curated blocklist. A test that fails but is not on the blocklist
is reported as "FAIL - unexpected" and fails the roachtest. Here, 8901/8905
tests passed and 3 tests failed unexpectedly.

Where the Failure Occurs

The roachtest itself does not crash — it fails in
orm_helpers.go:232
(summarizeFailed) because 3 CalculationsTest cases failed unexpectedly. All
three are client-side assertion mismatches on result data, not SQL errors:

Test (calculations_test.rb)	Expected	Actual
test_pluck_loaded_relation_sql_fragment (:1375)	["37signals", "Apex", "Ex Nihilo"]	["37signals", "Acme Co.", "Apex"]
test_should_calculate_grouped_by_function_with_table_alias (:568)	3	4
test_should_calculate_grouped_by_function (:560)	3	4

The common thread: an extra companies row was present during these tests.
The pluck test orders companies by name and takes the first 3; the actual result
contains "Acme Co." (which sorts between "37signals" and "Apex") while the
expected fixtures do not, meaning a "Acme Co."-named company existed that
should not have. The two grouped_by_function tests count companies grouped by
UPPER(type) and got 4 of a group instead of 3 — again consistent with one
extra committed company row.

Analysis

Hypothesis 1 — Shared-fixture pollution / test isolation flake (moderate confidence).
All three failures are explained by a single extra committed row in the shared
companies table that leaked from an earlier, non-transactional test in the
same suite run. Supporting evidence:

• The failures are data-content assertions (extra row / higher count), not
  CockroachDB errors — the queries executed fine, they just saw polluted data.
• In the same run, CockroachDB::TransactionsTest#test_concurrent_insert_with_processes
  hit a Timeout::Error: Test took over 5 minutes to finish and then
  RuntimeError: Found 1 leaked connection at teardown. This test forks
  child processes that perform concurrent inserts and runs outside a
  transaction (test/support/exclude_from_transactional_tests.rb). A leaked
  connection / abandoned child process can commit rows that outlive the test and
  are visible to later tests, which is exactly the pollution signature above.
  (Its interleaved WARNING: there is no transaction in progress output appears
  early in the log, while the failing CalculationsTest cases run later.)
• Most tests roll back via transactional fixtures; the process/concurrency tests
  are the known exception where committed state can escape.

Hypothesis 2 — Generic low-frequency ORM-suite flake (moderate confidence, overlaps H1).
Independent of the exact pollution mechanism, this is the classic activerecord
flake signature. Three sibling issues opened the same day (2026-09-21) on
other branches fail on completely different subtests:

• release-26.3: roachtest: activerecord failed #175678 (release-26.3): HasAndBelongsToManyAssociationsTest (2 tests)
• roachtest: activerecord failed #174829 (release-25.2.24-rc): SerializedAttributeTestWithYamlSafeLoad (4 tests)
• roachtest: activerecord failed #174826 (release-25.4.17-rc)

Different subtests failing per run across branches is the hallmark of suite
flakiness rather than a deterministic product regression. Prior closed
investigations (#174371, #172779) reached the same conclusion and noted the
test_concurrent_insert_with_processes 5-minute timeout + leaked connection as
a recurring, separate known issue.

This is very likely a test-suite isolation flake, not a CockroachDB
correctness regression. No PG:: / ActiveRecord::StatementInvalid error is
attached to any of the 3 unexpected failures.

Recommendations:

• Short-term (unblock the release-blocker): add the 3 CalculationsTest
  entries to activeRecordBlocklist (or, since they are flaky rather than
  consistently broken, to activeRecordIgnoreList) in
  activerecord_blocklist.go.
  The run already emitted an updated blocklist ("An updated blocklist ... is
  available in the artifacts' activerecord log").
• Root cause: the more durable fix is to prevent
  test_concurrent_insert_with_processes from leaking connections/rows on
  timeout (e.g. ensure child processes are reaped and their inserts rolled
  back), since that is the most plausible pollution source. This test is slow
  and times out more readily under runtime-assertions builds (this build has
  runtimeAssertionsBuild=true), increasing pollution frequency.
• Before permanently blocklisting, a re-run would likely pass or fail on
  different tests — treat as flake if so.

Related Issues and PRs

• release-26.3: roachtest: activerecord failed #175678 — release-26.3, same day, different failing subtests (same suite, likely same flake class).
• roachtest: activerecord failed #174829 / roachtest: activerecord failed #174826 — release-25.2/25.4 rc, same day, different subtests.
• roachtest: activerecord failed #174371, roachtest: activerecord failed #172779 (CLOSED) — prior investigations concluding this suite flakes
  per-run and is quarantined via blocklist; both noted the concurrent-insert
  5-min timeout + leaked connection.
• Long history of deflake activerecord commits on this file (e.g. 4a8a5e7295d,
  15f4b87416f, efcc6c92ff7) confirming chronic per-test flakiness.

Timeline

• Failure observed on basalt-26.3 @ 410866d9347, TeamCity build 21524604,
  run started 2026-09-20 ~06:16 UTC; suite finished after ~3266s.
• Sibling failures on release branches all filed 2026-09-21, each on different
  subtests — indicates a suite-wide flake window, not a single regression.

Detailed evidence and investigation notes

Log Excerpts

From run_062455.909205740_n1_cd-mntdata1activerec.log (extracted from
artifacts.zip):

19) Failure:
CalculationsTest#test_pluck_loaded_relation_sql_fragment [.../calculations_test.rb:1375]:
--- expected
+++ actual
@@ -1 +1 @@
-["37signals", "Apex", "Ex Nihilo"]
+["37signals", "Acme Co.", "Apex"]

22) Failure:
CalculationsTest#test_should_calculate_grouped_by_function_with_table_alias [.../calculations_test.rb:568]:
Expected: 3
  Actual: 4

23) Failure:
CalculationsTest#test_should_calculate_grouped_by_function [.../calculations_test.rb:560]:
Expected: 3
  Actual: 4

Co-occurring non-transactional test failure (plausible pollution source):

2) Error:
CockroachDB::TransactionsTest#test_concurrent_insert_with_processes:
Timeout::Error: Test took over 5 minutes to finish
    test/cases/transactions_test.rb:43:in `join'

Error:
CockroachDB::TransactionsTest#test_concurrent_insert_with_processes:
RuntimeError: Found 1 leaked connection after CockroachDB::TransactionsTest#test_concurrent_insert_with_processes
    .../test_case.rb:82:in `check_connection_leaks'
    test/support/exclude_from_transactional_tests.rb:29:in `after_teardown'

Summary (from failure_1.log):

8905 Total Tests Run / 8901 passed / 4 failed / 3 tests failed unexpectedly
--- FAIL: CalculationsTest#test_should_calculate_grouped_by_function_with_table_alias (unexpected)
--- FAIL: CalculationsTest#test_should_calculate_grouped_by_function (unexpected)
--- FAIL: CalculationsTest#test_pluck_loaded_relation_sql_fragment (unexpected)

Code References

• orm_helpers.go:232 — summarizeFailed t.Fatalf that fails the roachtest on unexpected failures.
• activerecord.go — test registration / result summarization (summarizeAll @ line ~264).
• activerecord_blocklist.go — blocklist/ignore-list; none of the 3 failing tests are currently listed.

Things Ruled Out

• CockroachDB SQL error / crash: No PG::* error, panic, or
  StatementInvalid is attached to any of the 3 unexpected failures — they are
  pure client-side data assertions. Node logs show no correlated fatal error.
• Deterministic query regression: ruled down because sibling issues the same
  day fail on entirely different subtests; a real regression would reproduce the
  same failing test.
• OOM / resource exhaustion: not indicated; the failure is a result mismatch,
  not a node death. (dmesg/heap profiles present but not implicated.)

Other Failure Occurrences

Only one failure in this issue thread (the body). Cross-branch comparison shows
the same suite failing on different subtests on 2026-09-21 (#175678, #174829,
#174826), reinforcing the flake interpretation over a single root cause.

Tooling Feedback

• No blockers. All needed data (issue bodies via gh, artifacts.zip via
  fetch-url, unzip) was accessible.
• Minor: the detailed per-test assertion diffs live only inside the large
  ~1MB run_*.log command output within artifacts.zip; there is no compact
  test.log-style summary of why each unexpected test failed. Surfacing the
  failing-assertion diffs directly in the issue body (beyond the pass/fail
  tally) would speed triage.
• Could not fetch the upstream Rails fork test source (calculations_test.rb at
  the pinned commit 845165d954e2) to confirm exact query text; reasoning above
  is inferred from the assertion output and standard Rails fixtures.

---

Investigation workflow run: https://github.com/cockroachdb/cockroach/actions/runs/35599752286

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

---

Generated by claude-opus-4-8 at high effort. Use /investigate fable-5 max to go deeper.

Workflow run · Was this investigation helpful? Leave a 👍 or 👎 on this comment.

[spilchen]

We have seen this mismatch once before #166981. I feel like it's a test flake somehow. Removing the release blocker.

[spilchen]

Root-caused this — it's cross-test pollution from the adapter's own suite, not a CRDB bug.

The polluter. CockroachDB::InheritanceComputeTypeTest#test_inheritance_new_with_subclass_as_default (inheritance_test.rb:25):

ActiveRecord::Base.lease_connection.change_column_default :companies, :type, "Firm"
Company.reset_column_information
Company.create!(name: "Acme Co.", firm_name: "Shri Hans Plastic")
ensure
  ActiveRecord::Base.lease_connection.change_column_default :companies, :type, original_type

"Acme Co." appears nowhere in Rails — only here.

Why the row escapes. The class uses transactional tests, so that row should roll back. But the roachtest forces autocommit_before_ddl: true by sed-ing the adapter's test/config.yml (in pkg/cmd/roachtest/tests/activerecord.go, added in be574aaf40 for #157123). Each change_column_default therefore commits and closes the fixture transaction, the create! lands in an implicit transaction and commits for real, and teardown has nothing left to undo. The run log shows it directly — the test passes while emitting three warnings:

CockroachDB::InheritanceComputeTypeTest#test_inheritance_new_with_subclass_as_default = WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
WARNING:  there is no transaction in progress
0.38 s = .

Why these three fail. companies.yml has exactly 3 Firms (ids 1, 4, 15). The leaked row picks up type = 'Firm' from the changed default, so assert_equal 3, c["FIRM"] sees 4, and Company.order(:name).limit(3) returns ["37signals", "Acme Co.", "Apex"] — both reported failures, exactly.

Why it's intermittent. Ordering, but more narrowly than just "polluter runs first". Rails re-runs create_fixtures (DELETE FROM companies plus re-insert) whenever a new test class declaring those fixtures starts, which scrubs the leaked row. The failure therefore needs CalculationsTest to be the next companies-fixture class after the polluter. In the failing run (seed 58358) the polluter finished at log line 2954 and CalculationsTest started at 3000, with zero companies-fixture classes in between. A clean run under seed 49012 put the polluter even earlier, at line 523, but with 1071 lines of companies-fixture classes in the gap — the row was long gone by the time CalculationsTest ran, and all three tests passed. The suite is single-process and minitest picks a random seed per run (Run options: -v --seed 58358), so landing in that window is luck. It explains why this has only surfaced here and in #166981.

This isn't isolated — the run has 693 there is no transaction in progress warnings, and the ignorelist already carries five entries tagged "affected by autocommit_before_ddl". Any test doing DDL inside a transactional test case can leak committed rows.

Fix: pin the minitest seed. TESTOPTS passes through to minitest, so adding --seed=<N> next to the existing -v makes ordering deterministic; the suite is single-process with no parallelize, so the seed fully determines order.

This doesn't stop the leak, it makes the consequences reproducible instead of random. Whatever then fails consistently gets an ignorelist entry with an accurate reason, rather than being re-triaged as a fresh flake every few weeks. The trade-off is that we stop exercising other orderings — which today mostly produces triage noise rather than fixes.

Fixing the adapter test to clean up its own row would address the root cause, but that's a change in activerecord-cockroachdb-adapter and out of scope here.