Skip to content

Fix rehash(0) producing a 100%-full table - #303

Merged
greg7mdp merged 1 commit into
greg7mdp:masterfrom
wenyekui:fix-rehash-zero-full-table
Sep 10, 2026
Merged

greg7mdp merged 1 commit into
greg7mdp:masterfrom
wenyekui:fix-rehash-zero-full-table

Conversation

@wenyekui

Copy link
Copy Markdown
Contributor

Fixes #302.

rehash(n) computes the new capacity as NormalizeCapacity(std::max(n, size())), which omits the growth→capacity conversion. When size() is exactly 2^k-1, NormalizeCapacity(size()) == size(), so rehash(0) resizes to a capacity equal to the element count — a table at 100% load. Two consequences:

  • ctrl_ is left with no kEmpty byte, so the probe loop in find_impl() has no reachable exit: any lookup of an absent key spins forever at 100% CPU.
  • reset_growth_left() computes CapacityToGrowth(capacity_) - size_ < 0, which underflows, so prepare_insert()'s growth_left() == 0 test never fires again and the table can no longer grow.

Besides rehash(0), the same path is reached by reserve(0) (because GrowthToLowerboundCapacity(0) == 0), by the resize(0) compatibility alias on the flat/node map/set wrappers, and by parallel_hash_set::rehash(n) with n < num_tables (integer division truncates to 0).

The change

-        auto m = NormalizeCapacity((std::max)(n, size()));
+        auto m = NormalizeCapacity(n | GrowthToLowerboundCapacity(size()));

This restores the Abseil form, i.e. it reverts the rehash() half of 1aeeff1 ("reserve and capacity not matching #18"). The parallel_hash_set::reserve() half of that commit is left intact — that half is what actually reduces over-allocation, and it also happens to keep nn >= 1, which is why parallel_hash_set::reserve() never hit this bug.

Why this is safe

bucket_count() after reserve(n), 12 cases × flat_hash_map and parallel_flat_hash_map — identical before and after:

 pre-size   reserve(n) |  flat   parallel
        0         1000 |  2047       2032
        0       100000 | 131071     131056
     1000         2000 |  4095       4080
   100000       200000 | 262143     262128
     1000         1000 |  2047       2032      <- n == size()
   100000       100000 | 131071     131056
     1000          500 |  2047       1776      <- n < size()
   100000        50000 | 131071     131056

Structurally: reserve(n) passes GrowthToLowerboundCapacity(n) as rehash's n, and GrowthToLowerboundCapacity(n) >= n >= size(), so that argument dominates either expression. The GrowthToLowerboundCapacity(size()) term can only matter when n < GrowthToLowerboundCapacity(size()), and there the m > capacity_ guard prevents any resize — except for n == 0, which is the unconditional path this fixes.

The loop from #18 (reserve(std::max(capacity(), 800000))) also behaves identically with and without this change — the capacity still doubles per iteration in both, matching your closing comment on that issue.

The only behaviour that changes:

rehash(0), flat_hash_map        before                      after
  size=15    cap  31 ->  15  100% full, find() hangs     31 ->  31
  size=28    cap  31 ->  31  ok                          31 ->  31   (same)
  size=31    cap  63 ->  31  100% full, find() hangs     63 ->  63
  size=63    cap 127 ->  63  100% full, find() hangs    127 -> 127
  size=100   cap 127 -> 127  ok                         127 -> 127   (same)

Sizes that were already safe shrink exactly as before; the ones that would have overflowed stop one tier earlier.

Tests

Two regression tests in tests/raw_hash_set_test.cc, next to the existing RehashZero* tests: RehashZeroPreservesMaxLoadFactor and ReserveZeroPreservesMaxLoadFactor. They sweep size() from 1 to 256 and assert size() <= CapacityToGrowth(bucket_count()) plus IsValidCapacity(bucket_count()).

They deliberately assert on capacities rather than calling find(): on a regressed build a lookup would spin forever, so a find()-based test would hang CI instead of failing. Verified both ways — with the fix reverted, both fail cleanly at n = 7 (on a Group::kWidth == 8 build; n = 15 where kWidth == 16).

Full suite, macOS / clang, -std=c++11:

test_raw_hash_set              74   test_parallel_flat_hash_map   297
test_flat_hash_map            288   test_parallel_flat_hash_set   173
test_flat_hash_set            167   test_parallel_node_hash_map   296
test_node_hash_map             98   test_parallel_node_hash_set   166
test_node_hash_set            166   test_btree                     69
test_erase_if                   3   test_dump_load                  3
                                    ------------------------------
                                    1800 tests, 0 failures

I did not add a separate test for the parallel_hash_set::rehash(n < num_tables) entry point: it funnels into raw_hash_set::rehash(0), so it is covered by construction, and a direct test would have to either reach into private state or risk hanging. The standalone repro for it is in #302 if you want it as a test as well — happy to add it.

rehash(n) computed the new capacity as NormalizeCapacity(max(n, size())),
which omits the growth->capacity conversion. When size() is exactly 2^k-1,
NormalizeCapacity(size()) == size(), so rehash(0) resizes to a capacity equal
to the element count -- a table at 100% load.

Two consequences:

 - ctrl_ is left with no kEmpty byte, so the probe loop in find_impl() has no
   reachable exit and any lookup of an absent key spins forever at 100% CPU.

 - reset_growth_left() computes CapacityToGrowth(capacity_) - size_ < 0, which
   underflows, so prepare_insert()'s `growth_left() == 0` test never fires
   again and the table can never grow.

Besides rehash(0), the same path is reached by reserve(0) (because
GrowthToLowerboundCapacity(0) == 0), by the resize(0) compatibility alias on
flat/node hash map/set, and by parallel_hash_set::rehash(n) with
n < num_tables (n / num_tables truncates to 0).

Restore the Abseil form, which converts size() to the capacity needed to hold
it under the 7/8 max load before taking the maximum. This reverts the rehash()
half of 1aeeff1; the parallel_hash_set::reserve() half of that commit is left
intact, and is what actually reduces over-allocation.

The change only affects the n == 0 path: for reserve(n) the
GrowthToLowerboundCapacity(n) argument already dominates, and for n != 0 the
`m > capacity_` guard means the table only ever grows. bucket_count() after
reserve(n) is unchanged across flat and parallel maps.

Add two regression tests. They assert on capacities rather than calling find(),
so a regressed build fails cleanly instead of hanging.

Fixes greg7mdp#302
@wenyekui
wenyekui force-pushed the fix-rehash-zero-full-table branch from f253c29 to f861fca Compare September 10, 2026 06:25
@greg7mdp
greg7mdp merged commit 55bf3c0 into greg7mdp:master Sep 10, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rehash(0) can produce a 100%-full table, causing an infinite loop in find()

2 participants