feat(pyramid,sindi,warp): add MARK_REMOVE support

[LHT129]

Summary

Add Remove() method to Pyramid, SINDI, and WARP indexes to support MARK_REMOVE mode, enabling marking vectors as deleted without physical removal.

Changes

WARP Index

• Implement Remove() method using existing delete_count_ member
• Support RemoveMode::MARK_REMOVE only

Pyramid Index

• Add std::atomic<int64_t> delete_count_ member variable
• Modify GetNumElements() to return cur_element_count_ - delete_count_
• Add GetNumberRemoved() method
• Implement Remove() method

SINDI Index

• Add std::atomic<int64_t> delete_count_ member variable
• Modify GetNumElements() to return cur_element_count_ - delete_count_
• Add GetNumberRemoved() method
• Implement Remove() method

Implementation Details

All three indexes:

• Support Remove(ids, RemoveMode::MARK_REMOVE) operation
• Return correct active element count via GetNumElements()
• Return correct deleted element count via GetNumberRemoved()
• Throw VsagException for unsupported FORCE_REMOVE mode
• Use label_table_->MarkRemove(ids) for marking deletions
• Thread-safe with std::scoped_lock protection

Test Plan

• Build: ✅ Release build successful
• Format: ✅ Code formatted with make fmt
• Existing tests: Pyramid, SINDI, WARP tests pass

Related Issue

Closes #2135

Reference

• HGraph Remove: src/algorithm/hgraph/hgraph_modify.cpp:22-45
• BruteForce Remove: src/algorithm/bruteforce/bruteforce.cpp:149-187

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Require kind label

Wonderful, this rule succeeded.

• label~=^kind/

🟢 Require version label

Wonderful, this rule succeeded.

• label~=^version/

[gemini-code-assist]

Code Review

This pull request implements the Remove method and tracks deleted elements via delete_count_ across the Pyramid, SINDI, and WARP algorithms. However, several critical issues were identified: first, delete_count_ is not restored during deserialization in Pyramid and SINDI, leading to incorrect element counts after loading; second, there are concurrency synchronization mismatches in Pyramid (mismatched locks between Add and Remove) and SINDI (using label_lookup_mutex_ instead of global_mutex_); and third, WARP's delete_count_ is not atomic, introducing a data race during concurrent reads and writes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[Copilot]

Pull request overview

Adds MARK_REMOVE-style deletion support to the Pyramid, SINDI, and WARP inner indexes by introducing Remove() implementations and exposing removed/active element counts via GetNumberRemoved() and updated GetNumElements() behavior.

Changes:

• Implement Remove(ids, RemoveMode::MARK_REMOVE) for WARP, Pyramid, and SINDI using label_table_->MarkRemove(ids).
• Track removed counts (new delete_count_ atomics for Pyramid/SINDI) and update GetNumElements() to return active elements.
• Add GetNumberRemoved() for Pyramid/SINDI (WARP already had it).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/algorithm/warp/warp.h	Declares Remove() for WARP.
src/algorithm/warp/warp.cpp	Implements WARP::Remove() (MARK_REMOVE only) and updates delete counter.
src/algorithm/sindi/sindi.h	Updates element counting, adds GetNumberRemoved(), and declares Remove() plus delete_count_.
src/algorithm/sindi/sindi.cpp	Implements SINDI::Remove() (MARK_REMOVE only) and updates delete counter.
src/algorithm/pyramid/pyramid.h	Declares GetNumberRemoved()/Remove() and adds delete_count_.
src/algorithm/pyramid/pyramid.cpp	Updates GetNumElements(), implements GetNumberRemoved()/Remove() for MARK_REMOVE.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.

[LHT129]

Thanks for the careful reviews! Addressed in 044304a:

Correctness (gemini)

• Pyramid/SINDI Deserialize now restores delete_count_ from label_table_->GetAllDeletedIds().size() (WARP already done). Note: tombstone persistence requires support_tombstone_=true on the label table; left as opt-in to match HGraph defaults.
• Pyramid::Remove now acquires both label_lookup_mutex_ and cur_element_count_mutex_ to match Add.
• SINDI::Remove now locks global_mutex_ to align with Add/Search.
• WARP delete_count_ is now std::atomic<uint64_t>.

Perf / fast-path (copilot)

• Pyramid/SINDI KnnSearch+RangeSearch now pass nullptr instead of an empty CombinedFilter so the unfiltered fast-path is preserved when there are no deletions or external filter.

Feature flags (copilot)

• All three indexes now set SUPPORT_DELETE_BY_ID in InitFeatures().

Tests (copilot)

• Added [ft][remove] test cases for Pyramid, SINDI and WARP covering: initial counts, FORCE_REMOVE rejection, mark-remove count, duplicate-remove (zero deletions), and search exclusion of removed ids. Persistence of tombstones across serialize/deserialize is out of scope here (consistent with HGraph), and would require enabling the existing support_tombstone_ option.

All 22050 assertions in the 3 new test cases pass locally.

[LHT129]

All review comments addressed and rebased onto latest main (a1b501f). Key fixes:

1. Deserialization: All 3 indexes now restore delete_count_ from label_table_->GetAllDeletedIds().size()
2. Thread safety: Pyramid uses scoped_lock(label_lookup_mutex_, cur_element_count_mutex_); SINDI uses global_mutex_; WARP delete_count_ changed to std::atomic<uint64_t>
3. Search filtering: Uses base class create_search_filter() which automatically includes deleted IDs filter and returns nullptr for fast path when no filters needed
4. InitFeatures: SUPPORT_DELETE_BY_ID added to all 3 indexes
5. Tests: Added Mark Remove functional tests for Pyramid, SINDI, and WARP - all passing

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.