primops: feat listToAttrsNull

[Malix-Labs]

Motivation

Constructing an attribute set from a list of strings (where all values are null) is a common pattern in Nix to represent sets for fast membership checks. This PR introduces a new primop builtins.listToAttrsNull to optimize this conversion.

Initially, this primop was implemented as listToSet. However, since Nix does not have a native "Set" type (using attribute sets with null values as a convention), the name listToSet is misleading. Renaming it to listToAttrsNull makes the return type and value behavior explicit, remaining consistent with listToAttrs.

Context

This PR contains:

1. The implementation of builtins.listToAttrsNull (originally listToSet).
2. Refactoring for code sharing in primops.cc.
3. Renaming of the primop to listToAttrsNull and corresponding unit tests in src/libexpr-tests/primops.cc.

---

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[llakala]

Where would this be useful? I've never seen this being done in nixpkgs, and I don't expect listToAttrs (map (name: { inherit name; value = null; }) names) to be much slower.

[Malix-Labs]

This pattern is sometimes used in nixpkgs to construct sets (attribute sets with null or true values) for builtins.intersectAttrs or membership tests:

• https://github.com/NixOS/nixpkgs/blob/3af24d1a5fc8a15e91c15da889ceefe21fc1b3b5/pkgs/applications/editors/vscode/extensions/vscode-utils.nix#L134
• https://github.com/NixOS/nixpkgs/blob/3af24d1a5fc8a15e91c15da889ceefe21fc1b3b5/pkgs/development/compilers/dotnet/source/default.nix#L154
• https://github.com/NixOS/nixpkgs/blob/3af24d1a5fc8a15e91c15da889ceefe21fc1b3b5/nixos/modules/system/boot/kernel.nix#L31

About performances, listToAttrs (map ...) requires invoking the map lambda N times and allocating N intermediate { name, value } attribute sets and a list of size N. listToAttrsNull does this directly in C++ with zero intermediate allocations, which is ~3x speedup and reduces heap usage and garbage to be collected

[xokdvium]

for builtins.intersectAttrs or membership tests:

But usage without a handful of packages might not really be performance sensitive though. Is this pattern used somewhere in commonly used code throughout nixpkgs (in lib eg.).

Ideally of course we'd make nix smart enough to do such matters matching optimisations without making a new primop each time someone finds a new pattern. But that of course relies on the grim future of making nix do non-trivial transformations on the interpreted code, so I'll shut up.

[Malix-Labs]

I've ran a clanker to benchmark and find more use cases potentially:

---

lib.lists.uniqueStrings is a prime candidate for this. It is currently implemented as attrNames (groupBy id list) using builtins.groupBy.

Benchmarking uniqueStrings on a list of 50k strings:

• Current (groupBy): ~0.30s
• With listToAttrsNull: ~0.08s (~4x speedup, avoiding intermediate grouping list allocations)

Benchmark commands:

# Current (groupBy)
nix eval --impure --expr '
  let
    uniqueStrings = list: builtins.attrNames (builtins.groupBy (x: x) list);
    names = map (x: "attr-" + toString (x / 2)) (builtins.genList (x: x) 50000);
  in
    builtins.seq (uniqueStrings names) 42
'

# With listToAttrsNull
nix eval --impure --expr '
  let
    uniqueStrings = list: builtins.attrNames (builtins.listToAttrsNull list);
    names = map (x: "attr-" + toString (x / 2)) (builtins.genList (x: x) 50000);
  in
    builtins.seq (uniqueStrings names) 42
'

While interpreter-level pattern-matching optimization would be ideal in theory, Nix's simple design makes targeted primops (much like builtins.groupBy itself) the pragmatic way to optimize these common patterns.

[roberth]

This doesn't feel like the right solution for the three examples

vscode-utils

It's a constant attrset, so may as well use an attrset literal.
Alternatively, intersectAttrs could be generalized to support a name list in the arg where it's ignoring the attribute values.

dotnet

This would benefit more from concatMapAttrs if the original list was actually an attrset.
That said, folding over pnames is a significant lack of laziness small, so I would try to tackle that first.

Otherwise, we might also consider a primop concatMapAttrList such that:
concatMapAttrs f attrs == concatMapAttrList (pair: f pair.name pair.value) (attrsToList attrs)

These two functions can represent a significant portion of whole-attrset consumption in practice, eliminating large intermediate list or attrset allocations when the processing can be moved into the callback.
(Also these are just very powerful and nice functions overall, subjectively, but also because of being like monadic bind fwiw)

kernel

Needs true, not null, so you'd still need to mapAttrs to correct that. That probably largely cancels any gain.

[Malix-Labs]

Well you are right that it doesn't need to default to null anyway, what matters is that the val is static

I've generalized the implementation to listToAttrsWithValue in 77d48a1, which takes the constant value as the first argument, and the list of keys as the second such as

builtins.listToAttrsWithValue value list

Which also would benefits in two of the examples I provided earlier

• kernel can now do builtins.listToAttrsWithValue true ... directly and removes the need for mapAttrs
• dotnet can use builtins.listToAttrsWithValue null (map (p: p.pname) a) which avoids allocating intermediate { name, value } attribute sets (but yeah for that specific example a architectural change to avoid list package collections would be more optimal)

vscode-utils would be okay with a static attrset because even the keys are constant, but otherwise listToAttrsWithValue remains useful for any dynamic list->set conversions

[llakala]

Otherwise, we might also consider a primop concatMapAttrList such that:
concatMapAttrs f attrs == concatMapAttrList (pair: f pair.name pair.value) (attrsToList attrs)

As someone who recently cleaned up concatMapAttrs in nixpkgs to not be ridiculously slow, this would definitely be nice. Curious as to the benefit it would make over the new definition, though, since it doesn't use foldl' mergeAttrs {} anymore.