Skip to content

Fix the C extension build on Ruby 4.1 and a GC crash on Ruby 3.3+ - #63

Open
CyberNativeAI wants to merge 2 commits into
kanwei:masterfrom
CyberNative-AI:ruby-4.1-typed-data
Open

CyberNativeAI wants to merge 2 commits into
kanwei:masterfrom
CyberNative-AI:ruby-4.1-typed-data

Conversation

@CyberNativeAI

Copy link
Copy Markdown

This fixes two problems in the C extensions:

  1. CRBTreeMap and CSplayTreeMap allocate during GC marking; starting with CRuby 3.3.0, this can abort the process. With the extensions built, they are what Containers::RBTreeMap and Containers::SplayTreeMap point to.
  2. On Ruby 4.1, the extensions no longer compile, so gem install algorithms fails.

GC crash on Ruby 3.3+

require "algorithms"
t = Containers::RBTreeMap.new
p t.class
1000.times { |i| t[i] = i.to_s }
GC.start
puts "ok #{t.size}"

With algorithms 1.1.0 on Ruby 4.0.7:

Containers::CRBTreeMap
<internal:gc>:44: [BUG] Cannot malloc during GC
ruby 4.0.7 (2026-09-15 revision 229531a6cf) +PRISM [x86_64-linux]

With this PR it prints ok 1000.

rbtree_mark and splaytree_mark build a queue with ALLOC while the GC is marking. Ruby 3.3.0's check_malloc_not_in_gc rejects Ruby-managed allocation through ALLOC/xmalloc during marking in the normal single-Ractor GC path. Ruby 3.2.2 warns on that path and calls rb_bug when RGENGC_CHECK_MODE or RUBY_DEBUG is enabled; the warning does not prevent a nested-GC crash. On Ruby 3.1.7 a normal GC passes, but under GC.stress the same code aborts with [BUG] during_gc != 0. #62 reports a different GC [BUG] from CRBTreeMap on Ruby 3.2.2. It may have the same cause, but we haven't run 3.2.

The second commit marks the nodes without allocating. A red-black tree is balanced, so plain recursion is enough. A splay tree can be a long chain (after sorted inserts, for example), so mark_nodes recurses into the smaller subtree and loops on the larger one. That keeps the C stack depth logarithmic. The same keys and values are marked as before.

Build failure on Ruby 4.1

Ruby 4.1 removes the untyped Data API, including Data_Wrap_Struct and Data_Get_Struct (Feature #21768, https://bugs.ruby-lang.org/issues/21768). On Ruby 4.1.0-preview1:

bst.c:23:9: error: implicit declaration of function 'Data_Get_Struct'; did you mean 'TypedData_Get_Struct'? [-Wimplicit-function-declaration]
bst.c:195:16: error: implicit declaration of function 'Data_Wrap_Struct'; did you mean 'TypedData_Wrap_Struct'? [-Wimplicit-function-declaration]

The first commit moves bst, deque, rbtree_map and splaytree_map to TypedData_Wrap_Struct and TypedData_Get_Struct, with one rb_data_type_t per class. The existing mark and free functions are reused, with no size function and flags 0, so objects are freed as before. The C accessors now use each class's TypedData descriptor. A CDeque subclass also passes the probe's push/read check. The TypedData API has been in Ruby since 1.9.2.

Testing

Official ruby Docker images, linux/amd64. For each Ruby: gem build, gem install of the built gem, then from outside the source tree:

  • a check that Containers::Deque, RBTreeMap and SplayTreeMap are the C classes, then 8,000 values reachable only through the four C containers, three full GCs, GC.compact and GC.verify_compaction_references; the probe observes all 8,000 values alive, and checks RBTreeMap key order, SplayTreeMap minimum/maximum keys and CBst size after compaction;
  • the algorithm from [Containers::CRBTreeMap] GC bug happened in Leetcode 3321 #62 on CRBTreeMap under GC.stress, compared with a run without it;
  • 200,000 sorted inserts into both maps, then full GCs;
  • bundle exec rake, and bundle exec rspec with the compiled extensions on the load path.
Ruby master this PR
4.1.0-preview1 gem install fails (errors above) all pass
4.0.7 GC check, GC.stress run and sorted inserts abort: Cannot malloc during GC all pass
3.1.7 GC.stress run aborts: during_gc != 0 all pass

Both commits together add no compiler warnings. On 4.0.7 they remove master's 9 -Wdeprecated-declarations. Not tested: Ruby 2.3 to 3.0 and 3.2 to 3.4, macOS, Windows.

The specs don't run the C classes

  • CI runs bundle exec rspec without compiling, so it tests only the pure-Ruby classes. On 4.1.0-preview1 it passes (115 examples, 0 failures) even though gem install fails.
  • bundle exec rake compiles, but rake-compiler puts the .so files in lib/containers and lib/algorithms, and lib/ requires them by bare name (require 'CDeque'). The specs still load the Ruby classes.
  • The guard in spec/map_gc_mark_spec.rb, !(defined? RUBY_ENGINE && ...), parses as !defined?(RUBY_ENGINE && ...), which is always false, so that spec never runs. With the guard fixed and the C classes loaded, it aborts with Cannot malloc during GC on master on Ruby 4.0.7, and passes with this PR.

We have a small commit for the last two (ext.lib_dir = "lib" in the Rakefile and the guard fix). With it, bundle exec rake runs 157 examples on the C classes, 0 failures, on all three Rubies. We can add it here or send it as a separate PR, and CI could then run bundle exec rake instead of bundle exec rspec.

This PR doesn't touch the CRBTreeMap#delete crash on a missing key; #55 fixes that, and the two merge cleanly.


Prepared by CyberNative AI LLC, an AI-run company, using AI agents. We checked each result above by running it.

Data_Wrap_Struct and Data_Get_Struct are not available in Ruby 4.1
(4.1.0-preview1), so the bst, deque, rbtree_map and splaytree_map
extensions fail to compile and gem install fails there.

Switch them to TypedData_Wrap_Struct and TypedData_Get_Struct with one
rb_data_type_t per class. The existing mark/free implementations are
reused, with bst's callback signatures changed to void pointers. Flags
are 0, preserving deferred custom-free behavior. TypedData exists in
Ruby 1.9.2, and the positional initializer matches Ruby 2.3.8's header
layout. Builds were tested on Ruby 3.1.7, 4.0.7 and 4.1.0-preview1;
Ruby 2.3 through 3.0 and 3.2 through 3.4 were not run for this
revision.
rbtree_mark and splaytree_mark built a linked-list queue with ALLOC
while the GC was marking. Starting with CRuby 3.3.0, ALLOC/xmalloc in
these mark callbacks is fatal in the normal single-Ractor GC path, with
"[BUG] Cannot malloc during GC". On Ruby 3.1.7, allocation in the mark
callback can re-enter GC under GC.stress and abort with
"[BUG] during_gc != 0".

Mark the nodes without allocating. The red-black tree is balanced, so
plain recursion is enough. A splay tree can be a long chain (for
example after inserting sorted keys), so mark_nodes recurses into the
smaller subtree and loops on the larger one, which keeps the stack depth
logarithmic. The same keys and values are marked as before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant