Add affine MinHash permutation schemes; refresh benchmarks and docs

[ekzhu]

Introduce affine multiply-shift permutation schemes for MinHash — affine32 (the new default) and affine64 — which remove the legacy scheme's large-set similarity bias, run faster, and use less memory. A legacy mode keeps backward compatibility with sketches from earlier versions.

Also includes an optional CuPy GPU path for update_batch, and refreshed synthetic benchmarks, figures, and documentation across the MinHash, LSH, LSH Forest, and LSH Ensemble pages.

[gemini-code-assist]

Code Review

This pull request bumps the package version to 2.0.0 and introduces new affine permutation schemes ("affine32" and "affine64") alongside the "legacy" scheme to fix similarity over-estimation bias on large sets. The default scheme is now "affine32", which halves memory usage and significantly speeds up updates. The changes update serialization and deserialization formats for MinHash, LeanMinHash, and bBitMinHash, add scheme consistency checks across LSH indexes, introduce extensive benchmark scripts, update documentation, and expand test coverage. Since there are no review comments, I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[Copilot]

Pull request overview

This PR introduces new MinHash permutation schemes (“affine32” default, plus “affine64” and backward-compatible “legacy”), adds scheme-consistency guards across MinHash-based indexes/serializations, and refreshes related tests, benchmarks, figures, and documentation for the 2.0.0 release.

Changes:

• Add affine multiply-shift permutation schemes (with fmix pre-mix), make affine32 the default, and preserve bit-exact compatibility via scheme="legacy".
• Enforce scheme consistency across MinHash comparisons and LSH-family indexes; extend LeanMinHash and b-bit MinHash serialization formats to encode scheme (while keeping legacy bytes unchanged).
• Refresh/expand tests, GPU-path coverage, benchmarks, and docs to reflect new schemes and performance/accuracy claims.

Reviewed changes

Copilot reviewed 37 out of 64 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
uv.lock	Updates locked project version to 2.0.0.
pyproject.toml	Bumps package version to 2.0.0.
README.rst	Adds upgrade note about the new default MinHash scheme and migration guidance.
docs/minhash.rst	Documents permutation schemes, migration, and updated benchmark/GPU guidance.
docs/lsh.rst	Rewrites/expands MinHash LSH documentation and adds new benchmark discussion.
docs/lshforest.rst	Rewrites/expands LSH Forest documentation and adds benchmark/characteristics sections.
docs/lshensemble.rst	Reworks LSH Ensemble benchmark explanation and figures narrative.
docs/documentation.rst	Adds API-doc entry for b-bit MinHash.
docs/b-bit-minhash.rst	Adds new b-bit MinHash documentation page.
docs/_static/minhash/affine_scheme.svg	Adds new diagram illustrating the affine scheme pipeline.
datasketch/minhash.py	Implements schemes, fmix pre-mix, scheme guards, GPU branching per scheme, and legacy compatibility behavior.
datasketch/lsh.py	Learns/enforces MinHash scheme per index and prevents mixed-scheme merges/queries.
datasketch/lshforest.py	Learns/enforces scheme on add/query; updates hashvalue reconstruction to support 32/64-bit storage.
datasketch/lshensemble.py	Clarifies where scheme checks occur in the ensemble query path.
datasketch/lsh_bloom.py	Adds scheme learning/guarding and makes band hashing deterministic across dtypes.
datasketch/lean_minhash.py	Adds scheme-aware construction, scheme-tagged serialization for affine schemes, and legacy byte compatibility.
datasketch/b_bit_minhash.py	Adds scheme tracking/guards and scheme-tagged pickle payloads for non-legacy schemes.
datasketch/aio/lsh.py	Adds scheme learning/guarding in async LSH inserts/queries.
datasketch/init.py	Exposes sha1_hash64 at the package top level.
test/test_minhash.py	Updates byte-token expectations for the new default scheme and adds affine64 coverage.
test/test_minhash_schemes.py	Adds comprehensive scheme, serialization, compatibility, and statistical-behavior tests.
test/test_minhash_gpu.py	Expands GPU tests across schemes and adds a CuPy shim to exercise GPU branch on CPU.
test/test_lshforest.py	Updates reconstruction test to pass scheme explicitly when rebuilding from raw hashvalues.
test/test_lshbloom.py	Skips bloom tests when optional dep missing (but fails in CI) and adds scheme-mismatch tests.
test/test_lean_minhash.py	Requires scheme when constructing from raw hashvalues and adds affine bytesize assertions.
benchmark/sketches/similarity_benchmark.py	Ports benchmark off Python2/pyhash to Python3/mmh3 seeded hash function.
benchmark/sketches/inclusion_benchmark.py	Ports benchmark off Python2/pyhash to Python3/mmh3 seeded hash function.
benchmark/sketches/cardinality_benchmark.py	Ports benchmark off Python2/pyhash to Python3/mmh3 seeded hash function.
benchmark/sketches/minhash_benchmark.py	Refreshes benchmark to compare multiple hash functions and output updated figure.
benchmark/sketches/minhash_scheme_bias_benchmark.py	Adds new large-scale bias experiment reproducer and plotting utility.
benchmark/sketches/b_bit_minhash_benchmark.py	Reworks b-bit benchmark to focus on estimator spread vs exact similarity.
benchmark/indexes/synthetic_data.py	Adds shared synthetic corpora + exact metrics helpers for index benchmarks.
benchmark/indexes/lsh_synthetic_benchmark.py	Adds synthetic accuracy/speed benchmark for MinHashLSH.
benchmark/indexes/lsh_similarity_benchmark.py	Adds planted-neighbor retrieval-characteristics benchmark for MinHashLSH.
benchmark/indexes/lshforest_synthetic_benchmark.py	Adds synthetic top-k benchmark for MinHashLSHForest.
benchmark/indexes/lshforest_similarity_benchmark.py	Adds retrieval-characteristics benchmark for MinHashLSHForest.
benchmark/indexes/lshensemble_synthetic_benchmark.py	Adds synthetic partitioning-effect benchmark for MinHashLSHEnsemble.
benchmark/indexes/lsh_20newsgroups_benchmark.py	Adds 20 newsgroups benchmark recreating the docs LSH figure.
benchmark/indexes/containment/lshensemble_opendata_benchmark.py	Adds (legacy-style) open-data benchmark script under containment benchmarks.

Comments suppressed due to low confidence (1)

datasketch/b_bit_minhash.py:98

• This ValueError message uses a backslash line-continuation inside the string literal, which includes indentation spaces in the actual error text. Using a single-line message avoids surprising whitespace in user-facing errors.

        if self.seed != other.seed:
            raise ValueError(
                "Cannot compare two b-bit MinHashes with different\
                    set of permutations"
            )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.