Allocate RootValues more efficiently#16114
Open
edolstra wants to merge 6 commits into
Open
Conversation
Previously every RootValue was created with std::allocate_shared<Value *>(traceable_allocator<Value *>()), which costs a GC_MALLOC_UNCOLLECTABLE() / GC_FREE() pair per root value. Uncollectable allocations always take the global GC allocation lock, making this a significant source of mutex contention during parallel evaluation: the eval cache allocates a root value per AttrCursor (i.e. per attribute visited by 'nix search'), and the parallel evaluator allocates one per queued work item. Instead, carve root slots out of large uncollectable slabs (which are permanently part of the GC root set) and recycle them through a free list protected by a plain mutex, whose critical section is a few instructions rather than the entire GC allocator. Slots are cleared on release so they don't keep values alive; liveness semantics are unchanged. The slabs are never freed, so pool memory is bounded by the peak number of simultaneously live root values (8 bytes per slot). At 16 eval cores (with GC disabled via GC_INITIAL_HEAP_SIZE=40G), this cuts kernel time from ~8-9s to ~7s, context switches from 1.46M to 0.99M, and elapsed time from ~4.9s to ~4.3-4.5s for 'nix search nixpkgs --no-eval-cache fizzbuzz'. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
RootValue (std::shared_ptr<Value *>) adds avoidable overhead per GC
root: a malloc'd control block, atomic reference count updates on
copies (a measured source of cache line contention during parallel
evaluation), and 16 bytes per handle. Most users never copy their
roots, so introduce UniqueRootValue, a move-only RAII wrapper around
a pointer to a root slot that returns the slot to the pool on
destruction.
The pool internals move from allocRootValue() into
allocRootValueSlot() / freeRootValueSlot(), shared by both handle
types, and the whole root value machinery moves into new files
root-value.{hh,cc}. The non-Boehm fallback now also goes through the
slot functions (new/delete) instead of a make_shared special case.
Converted to UniqueRootValue: ValMap, fileEvalCache, internalPrimOps,
genericClosure's work/result lists, EvalCache::value,
AttrCursor::_value, InstallableAttrPath::v, and the JSON parser
state. RootValue remains (documented as such) for roots captured in
std::function-backed lambdas, which require copyability.
Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Replace the Sync<std::vector<Value **>> free slot pool with an intrusive linked list: each slot is a union of Value * (in use) and a pointer to the next free slot, with a Sync<Slot *> head. Allocation pops the head; freeing pushes in O(1). This avoids the auxiliary vector memory (8 bytes per free slot), the 4096-iteration push_back loop when carving a new slab, and most importantly the possibility of a vector reallocation (a malloc) while holding the pool lock on the free path. GC safety is unchanged: slabs remain uncollectable and conservatively scanned. In-use slots contain a Value * and root the value; free slots contain a pointer into a slab (or null), which is scanned harmlessly since slabs are never freed anyway. Writing the next pointer on free overwrites the Value *, which doubles as the "stop rooting the value" step. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
the
repl
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Taken from DeterminateSystems#546. This is mostly useful for parallel eval but may also improve single-threaded eval slightly by avoiding a lot of
std::shared_ptrallocations and calls totraceable_allocator(23.60s -> 23.49s onnix search nixpkgs fizzbuzz --no-eval-cache).RootValues were expensive for parallel eval, since eachRootValuecaused a Boehm GC call to allocate traceable-but-uncollectable memory while holding the global GC root. We now allocate slabs of uncollectable memory from which theRootValues are allocated.Also,
RootValueis not astd::shared_ptranymore but a pointer directly into thetraceable_allocatorslab, so that saves a heap allocation per root value.Context
Add 👍 to pull requests you find important.
The Nix maintainer team uses a GitHub project board to schedule and track reviews.