developers-guide.md

Developers guide

This guide collects day-to-day practices for contributors working on the Chutoro codebase. It complements the more specialized documents in docs/ and keeps operational guidance in one place.

CPU HNSW public APIs

chutoro-core exposes CpuHnsw as the public CPU-resident Hierarchical Navigable Small World (HNSW) index. The primary insertion and query entry points are:

pub fn build<D: DataSource + Sync>(source: &D, params: HnswParams)
    -> Result<CpuHnsw, HnswError>;
pub fn build_with_edges<D: DataSource + Sync>(source: &D, params: HnswParams)
    -> Result<(CpuHnsw, EdgeHarvest), HnswError>;
pub fn with_capacity(params: HnswParams, capacity: usize)
    -> Result<CpuHnsw, HnswError>;
pub fn insert<D: DataSource + Sync>(&self, node: usize, source: &D)
    -> Result<(), HnswError>;
pub fn insert_harvesting<D: DataSource + Sync>(&self, node: usize, source: &D)
    -> Result<Vec<CandidateEdge>, HnswError>;
pub fn search<D: DataSource + Sync>(
    &self,
    source: &D,
    query: usize,
    ef: NonZeroUsize,
) -> Result<Vec<Neighbour>, HnswError>;

build and build_with_edges seed the entry point from node 0 and insert the remaining nodes in parallel. build_with_edges is the preferred path when the caller needs the deterministic EdgeHarvest used by the MST stage. with_capacity is the entry point for manual or incremental index population.

insert mutates the graph without allocating harvested edge storage. insert_harvesting performs the same planning and commit sequence, but also returns the candidate edges identified during the read-phase search. The first insertion into an empty index returns an empty vector, while later insertions return edges ordered by insertion sequence. DuplicateNode indicates a repeat insertion, and other HnswError variants cover invalid parameters, poisoned locks, non-finite distances, graph invariant failures, and wrapped DataSourceError values.

search is the observable way to compare graph state after different insertion paths. The HNSW edge-harvesting tests treat equivalent search results as the behavioural contract between insert and insert_harvesting.

Design rationale and deeper implementation notes live in the design document, the completed edge-harvesting ExecPlan, and the completed incremental core-distance ExecPlan.

Session public APIs

The public session surface is CPU-only. build_session constructs an empty ClusteringSession without seeding HNSW or running the batch bootstrap path. The architectural rationale for that split lives in the design document.

// ChutoroBuilder (cpu feature required for session APIs)
pub fn with_hnsw_params(self, params: HnswParams) -> Self;
pub fn hnsw_params(&self) -> &HnswParams;
pub fn with_session_refresh_policy(self, policy: SessionRefreshPolicy) -> Self;
pub fn session_refresh_policy(&self) -> &SessionRefreshPolicy;
pub fn build_session<D: DataSource + Send + Sync>(self, source: Arc<D>)
    -> Result<ClusteringSession<D>>;

// SessionRefreshPolicy
pub fn manual() -> Self;
pub fn with_refresh_every_n(self, refresh_every_n: Option<NonZeroUsize>) -> Self;
pub fn refresh_every_n(&self) -> Option<NonZeroUsize>;

// SessionConfig
pub fn min_cluster_size(&self) -> NonZeroUsize;
pub fn hnsw_params(&self) -> &HnswParams;
pub fn refresh_policy(&self) -> &SessionRefreshPolicy;

// ClusteringSession<D: DataSource + Send + Sync>
pub fn config(&self) -> &SessionConfig;
pub fn append(&mut self, indices: &[usize]) -> Result<()>;
pub fn recompute_core_distances(&mut self) -> Result<()>;
pub fn recompute_core_distances_full(&mut self) -> Result<()>;
pub fn core_distance(&self, point: usize) -> Option<f32>;
pub fn point_count(&self) -> usize;
pub fn snapshot_version(&self) -> u64;

build_session validates min_cluster_size > 0, rejects ExecutionStrategy::GpuPreferred, accepts empty and undersized sources, and returns an inert session whose initial observable state is point_count() == 0 and snapshot_version() == 0.

append inserts source indices into the live HNSW index by calling CpuHnsw::insert_harvesting for each index. It must not duplicate HNSW insertion logic or inspect private HNSW adapter internals. The session stores all returned CandidateEdge values in its internal pending_edges buffer for future refresh work. The method is fail-fast and preserves partial progress: insertions completed before the first error remain in the index, and their harvested edges remain pending.

recompute_core_distances computes core distances for dirty newly inserted source indices and for existing points that appear in those new points' non-self HNSW neighbour lists. recompute_core_distances_full searches every inserted point and mirrors the batch CPU core-distance loop. The core_distance(i) accessor returns None for cells that are dirty, unset, or outside the source-indexed storage, and it also treats non-finite cells as unavailable. Callers that need to distinguish an out-of-range read from a dirty inserted point should check their own source index bookkeeping; point_count() reports the number of inserted points, not a maximum source index.

Session construction allocates HNSW capacity from source.len().max(1) while still leaving the index empty. append prevalidates each requested index against source.len() before insertion so early bootstrap cases return a ChutoroError::DataSource for out-of-bounds indices even when HNSW would not need a distance query for the first inserted node.

The v1 incremental clustering surface has these limitations:

• Ingestion is append-only; deletions and updates are not supported.
• Cluster identity is not stable across snapshots, and cluster IDs may change.
• Refreshes are micro-batched rather than applied per point.
• Existing points may be relabelled after a refresh.

Session internal architecture

Keep the session responsibility split intact when adding behaviour. Domain state should stay on ClusteringSession; configuration-only changes belong with session configuration; append construction work belongs with the append implementation; and core-distance work should stay in the core-distance subsystem until it grows a clearer sub-boundary.

Core-distance storage is indexed by source index, not by dense insertion ordinal. core_distances: Vec<f32> stores finite values after recompute and uses f32::INFINITY only as an internal unset sentinel. dirty_core_distances: Vec<bool> is authoritative: true means the corresponding cell is stale or never computed, and false means a finite cell may be read through core_distance(i). The dirty state uses Vec<bool> because the workspace does not already depend on fixedbitset, and roadmap item 11.1.4 forbids new production dependencies.

ClusteringSession::append emits tracing through #[tracing::instrument] and structured warn!/debug! events, but it must not install tracing subscribers. Library code may emit metrics and tracing; application boundaries remain responsible for recorder and subscriber installation.

Metrics support is entirely feature-gated behind metrics. Production builds without that feature must not allocate the clock field or compile metric emission code. When metrics is enabled, construction describes the append error counter, per-point latency histogram, and harvested-edge counter. The append path records:

• chutoro.session.append.errors_total, labelled by low-cardinality failure reason.
• chutoro.session.append.point_seconds, one histogram sample per inserted point.
• chutoro.session.harvested_edges, counting buffered candidate edges.

Core-distance recompute helpers record:

• chutoro.session.core_distance.queries_total, counting HNSW searches used for core-distance recompute.
• chutoro.session.core_distance.recomputed_existing, counting existing points recomputed after incremental neighbour discovery.
• chutoro.session.core_distance.appends_left_dirty_total, counting recompute calls that began with dirty core distances.
• chutoro.session.core_distance.touched_existing_per_recompute, recording incremental recompute fan-out.
• chutoro.session.core_distance.recompute_seconds, recording recompute duration.

The latency histogram reads time through the internal MonotonicClock trait. StdMonotonicClock is the production implementation. Tests may replace it via with_clock_for_test with FixedMonotonicClock, which is available only under #[cfg(all(feature = "metrics", test))]. Do not expose this seam through the public constructor or builder API; it exists solely to make metrics assertions deterministic while preserving the public session contract.

Continuous integration

Property-test CI jobs (property-tests-pr and property-tests-weekly) run on ubicloud-standard-8, an 8-core Ubicloud runner, rather than ubuntu-latest.

The PR job has a timeout-minutes: 20 budget, sized to exceed the longest nextest slow-timeout (600 s for HNSW idempotency) so earlier setup and property phases do not consume the full budget. The weekly job retains a timeout-minutes: 120 budget for deeper test runs.

All test runs use the nextest CI profile (--profile ci). Benchmark targets require threads-required = 8; see .config/nextest.toml.

Use .github/workflows/property-tests.yml and .config/nextest.toml for the authoritative configuration, and docs/property-testing-design.md for the architectural rationale.

Dense SIMD parity suite

Dense Euclidean backend parity tests live in chutoro-providers/dense/src/simd/tests/parity/. The suite compares each compiled and runtime-supported backend against the scalar oracle defined by the test-only DistanceSemantics value object.

When adding a new dense SIMD backend or Euclidean kernel:

1. Add the backend to dispatch.rs::enabled_backends so tests can discover it only when it is both compiled and runtime-supported.
2. Wire pairwise and query-to-points entry functions in the test-only helpers in kernels.rs. Keep backend implementation modules private unless a production caller needs wider visibility.
3. Extend tests/parity/strategies.rs only when the new backend has a new layout, lane width, or input hazard that the existing generators do not cover.
4. Run cargo nextest run -p chutoro-providers-dense simd::tests::parity:: before the full make check-fmt, make lint, make typecheck, and make test gates.

If proptest records a regression, keep the generated file under the relevant proptest-regressions/ directory. That file is the shrunk counterexample and should be treated as a regression guard, not as disposable local output.

Dense SIMD Kani harnesses

Dense SIMD Kani harnesses live in chutoro-providers/dense/src/simd/kani_proofs.rs and are compiled only under #[cfg(kani)]. They prove boundary policy for the safe SIMD seams, not raw architecture intrinsics.

When changing dense SIMD tail padding, lane batching, or runtime backend selection:

1. Keep reusable arithmetic in production-used helpers, then prove those helpers or their immediate call boundary. Avoid proof-only arithmetic that can drift away from the kernels.

2. Keep selector policy in dispatch.rs::choose_euclidean_backend; the Kani harness proves every compile-time and runtime support-mask combination.

3. Use rstest unit tests for concrete storage behaviour, especially DensePointView<'a> alignment, 16-lane padding, and zero-filled unused lanes.

4. Run the practical Kani suite before requesting review:

   set -o pipefail
   make kani 2>&1 | tee /tmp/kani-chutoro-$(git branch --show-current).out

make kani-full runs every Kani harness in chutoro-core and chutoro-providers-dense. Keep new dense harnesses small enough for make kani unless they are intentionally slow-lane proofs.

Benchmarks

The chutoro-benches crate provides Criterion benchmarks for the four CPU pipeline stages: Hierarchical Navigable Small World (HNSW) index construction, edge harvest, minimum spanning tree (MST) computation, and hierarchy extraction.

Running benchmarks

Run all benchmarks from the repository root:

make bench

Criterion writes HTML reports to target/criterion/. Open the report for a specific group (for example target/criterion/hnsw_build/report/index.html) to view timing distributions and comparisons against previous runs.

Benchmark regression workflow

Benchmark regression detection follows a two-tier strategy:

• Pull request (PR) workflows run a fast benchmark smoke check using discovery mode (--list) to confirm that benchmark binaries still compile and enumerate benchmark cases.
• A scheduled weekly workflow (plus manual workflow_dispatch) runs Criterion baseline comparison by saving a reference baseline from HEAD^ and comparing the current revision with --baseline.

Run the local baseline workflow for one benchmark from the repository root:

set -o pipefail
CHUTORO_BENCH_HNSW_MEMORY_PROFILE=0 \
CHUTORO_BENCH_HNSW_RECALL_REPORT=0 \
CHUTORO_BENCH_HNSW_CLUSTER_QUALITY_REPORT=0 \
cargo bench -p chutoro-benches --bench hnsw_ef_sweep -- \
  --save-baseline local-reference \
  --noplot \
  2>&1 | tee /tmp/bench-hnsw-ef-sweep-save.log

set -o pipefail
CHUTORO_BENCH_HNSW_MEMORY_PROFILE=0 \
CHUTORO_BENCH_HNSW_RECALL_REPORT=0 \
CHUTORO_BENCH_HNSW_CLUSTER_QUALITY_REPORT=0 \
cargo bench -p chutoro-benches --bench hnsw_ef_sweep -- \
  --baseline local-reference \
  --noplot \
  2>&1 | tee /tmp/bench-hnsw-ef-sweep-compare.log

Use --list for a quick discovery check only:

set -o pipefail
cargo bench -p chutoro-benches --bench hnsw_ef_sweep -- --list \
  2>&1 | tee /tmp/bench-hnsw-ef-sweep-list.log

Benchmark architecture

Benchmarks live in chutoro-benches/benches/ as separate Criterion binaries. Shared support code — the SyntheticSource data source, benchmark parameter types, and the BenchSetupError error type — lives in the chutoro-benches library (chutoro-benches/src/).

Each benchmark file follows this pattern:

1. A fallible _impl function performs all setup (data generation, HNSW build, MST computation) using ? to propagate errors via BenchSetupError.
2. A thin wrapper registered with criterion_group! calls the _impl function and panics on failure.
3. The measured closure uses b.iter() or b.iter_batched() to time only the target operation.

Lint policy for benchmarks

The chutoro-benches crate does not inherit workspace lints. Criterion's macro expansions (criterion_group!, criterion_main!, bench_with_input closures) trigger several of the strict workspace denials — most notably missing_docs, shadow_reuse, and excessive_nesting. A crate-local [lints] section in chutoro-benches/Cargo.toml mirrors the workspace strictness for handwritten code. Benchmark source files use tightly scoped #![expect(lint, reason = "…")] attributes for the lints that Criterion's macro expansions unavoidably trigger. The rationale is documented in a comment at the top of the [lints.clippy] section in chutoro-benches/Cargo.toml.

Adding a new benchmark

1. Create a new file in chutoro-benches/benches/.
2. Add a corresponding [[bench]] entry in chutoro-benches/Cargo.toml.
3. Follow the fallible-setup pattern described above.
4. Use #![expect(…)] (not #![allow(…)]) for any Criterion-triggered lint suppressions, with a reason string.
5. Run make bench to verify the new benchmark appears in the output.

Benchmark dataset recipes

The chutoro-bench-datasets crate defines the shared recipe surface for benchmark dataset preparation. A DatasetRecipe fetches source inputs, validates them, prepares benchmark-ready bytes, and publishes a manifest handle through typed phase handoffs. The type returned by each phase is consumed by the next one, so callers cannot publish fetched data that has not passed validation and preparation.

Recipes receive infrastructure through RecipeContext. Fetcher reads source bytes with a mandatory max_bytes cap, Storage is a mutable cache with overwrite semantics, and Publisher is the final write-once sink. Network download, archive extraction, object-store adapters, lockfiles, checksum verification, and licence gates are deferred to later roadmap items.

Enable the crate's testing feature for adapter contract tests. It exposes InMemoryFetcher, InMemoryStorage, InMemoryPublisher, FilesystemFetcher, and StubRecipe. The behavioural tests in chutoro-bench-datasets/tests/recipe_bdd.rs run the same fetcher contract against both in-memory and filesystem-backed adapters.

The design rationale is recorded in ADR-003. The broader dataset pipeline is described in benchmark-dataset-retrieval.md §3.1.

Verus proofs

Verus is used for formal verification of edge harvest primitives. Run proofs via make verus, which is idempotent and installs the pinned Verus release and required Rust toolchain as needed.

Quantifier trigger annotations

Verus prints warnings when it selects quantifier triggers automatically. Do not ignore these warnings. Prefer explicit annotations so the prover behaviour is stable and predictable:

• Use #[trigger] when a specific term should control instantiation.
• Use #![auto] only when the automatically chosen trigger is acceptable and the quantifier is straightforward.
• Avoid --triggers-mode silent in continuous integration (CI) because it hides trigger-selection changes.

Example:

assert forall|i: int| #![auto] 0 <= i < edges.len() implies edges[i].source == source_node;