diff --git a/.github/workflows/workspace.yml b/.github/workflows/workspace.yml index fe3e985c..6954e1b3 100644 --- a/.github/workflows/workspace.yml +++ b/.github/workflows/workspace.yml @@ -8,6 +8,12 @@ on: permissions: contents: read +# A newer push to the same pull request supersedes the run before it. Pushes to +# main are never cancelled: each one is a commit someone may need a verdict on. +concurrency: + group: workspace-${{ github.event.pull_request.number || github.sha }} + cancel-in-progress: true + env: CARGO_TERM_COLOR: always @@ -43,8 +49,22 @@ jobs: - name: Graph benchmark Clippy run: cargo clippy --locked --manifest-path benchmarks/lsqb/Cargo.toml --all-targets -- -D warnings + # grust-ladybug is tested on its own. LadybugDB's prebuilt static library + # bundles zstd, and a whole-workspace --all-features build unifies + # features so that zstd-sys (LanceDB, Arrow IPC compression) lands in a + # crate Ladybug links as well. rust-lld then rejects the duplicate ZSTD_* + # symbols in Ladybug's test binary. Built alone, Ladybug never pulls + # zstd-sys, so its tests link and still run with its default features. - name: Workspace tests - run: cargo test --locked --workspace --all-features + run: cargo test --locked --workspace --all-features --exclude grust-ladybug + + - name: Ladybug tests + run: cargo test --locked -p grust-ladybug + + # The offline benchmark tests read the pinned LSQB example dataset, which + # is fetched and verified rather than checked in. + - name: Pinned LSQB sources + run: benchmarks/lsqb/fetch-upstream.sh - name: Graph benchmark tests run: cargo test --locked --manifest-path benchmarks/lsqb/Cargo.toml diff --git a/CHANGELOG.md b/CHANGELOG.md index 8dafaf2c..84067445 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,184 @@ reconstructed from Git history, release commits, and the shipped docs. ## Unreleased +- Add **FastRP node embeddings**: `fast_rp` and `grust.algorithms.fastRP`, with + `embeddingDimension`, `iterationWeights`, `nodeSelfInfluence`, + `normalizationStrength` and `seed`. Each node draws a very sparse random + vector (`±√3` with probability 1/6 each), every round replaces it with the + weighted mean of the vectors its arcs reach, and the rounds are summed at unit + length with their weights. Entry `j` of node `v` is a pure function of + `(seed, v, j)`, so the embedding is identical at any pool width. It is the + first vector-valued result: result tables gain a fixed-length `Float32` column, + emitted as Arrow `FixedSizeList` and as a float list to Cypher. The + test rebuilds the random vectors from a round-zero run and repeats the whole + computation densely in double precision. No claim is made about embedding + quality. +- Add **maximum flow and minimum cut**: `max_flow`, and + `grust.algorithms.maxFlow(source, target)` and `.minCut(source, target)`. + Capacities are the projection's weights, parallel edges are separate + capacities, and self-loops and zero capacities carry nothing. `maxFlow` + returns one row per edge carrying flow, oriented the way the flow runs, so an + undirected edge reports its net flow; `minCut` returns each node's side, the + source side being what the source still reaches once the flow is in place. + Dinic's algorithm, iterative, sequential. With integral capacities every value + is exact; with fractional ones the flow is feasible and maximal up to + rounding. The test compares the value with the cheapest of every possible cut + on 7,500 random networks and checks capacity and conservation on the reported + rows. Procedures can now declare a second positional node argument, `target`. +- Add **Leiden** community detection: `leiden` and `grust.algorithms.leiden`, + with Louvain's options and result shape. Between moving and coarsening it + refines each community from singletons, merging a node only while it is still + alone, only into a neighbour's group, and only when modularity does not fall, + so **every community is connected** — weakly, on a directed projection, and + never through a zero-weight arc. The result is always the refined partition, + so the guarantee holds however the run ends, `maxLevels` included. Refinement + is greedy, the zero-temperature limit of the paper's randomised rule: it + keeps connectivity and reproducibility and gives up the paper's asymptotic + optimality guarantee; there is no `theta`. Tests assert connectivity and + honest modularity over 4,500 random runs with zero weights, loops and parallel + edges, never above the brute-force optimum, and exact recovery of planted + cliques. +- Add **eigenvector centrality**, **Katz centrality** and **HITS**: + `eigenvector`, `katz`, `hits` and the procedures of the same names, each + reporting `iterations`, `converged` and `residual` as `pagerank` does, never + asserting convergence silently. Influence flows along the projection's arcs. + Eigenvector iterates `(A + I)x` rather than `Ax`: same eigenvectors, but it + converges on bipartite graphs where plain power iteration oscillates. Katz + does not refuse an `alpha` above the cheap `1 / (largest in-strength)` bound, + which would reject many valid values; a run that does not settle says so, and + one that overflows fails and names `alpha`. Each node pulls from its + neighbours over fixed chunks of 4,096 nodes on the caller's rayon pool, so + scores, residuals and charged work are bit-identical at any width. Tests check + the defining equations with dense linear algebra: `Ax = λx`, `MMᵀa = λa`, + and Katz against Gaussian elimination. +- Add **minimum and maximum spanning forests**: `spanning_tree` and + `grust.algorithms.spanningTree`, with `objective` and `sourceNode`. Kruskal + with union by size and path halving, on an undirected projection. Ties go to + the smaller edge ordinal, so edges are in a total order and the forest is the + greedy one for it: the same projection always gives the same edges, not one + of several equally light ones. `sourceNode` keeps the tree of that node's + component, which is what a Prim run from it returns. Only the sort runs on + the rayon pool, and its order is total, so nothing depends on the pool. The + test finds the expected forest by exhaustive search over edge subsets, with + heavy ties, for both objectives. +- Add **bridges**, **articulation points** and **biconnected components**: + `biconnectivity`, and `grust.algorithms.bridges`, `.articulationPoints` and + `.biconnectedComponents`, from one iterative low-link pass on an undirected + projection. The pass tracks the edge it arrived by, not the node, so parallel + edges are a cycle of two: neither is a bridge and both share a component. + Self-loops disconnect nothing and lie in no component. Edge rows carry + `sourceNodeId`, `targetNodeId` and `edgeOrdinal`; a component is named by the + smallest edge ordinal in it. Each output is checked against its definition — + remove the edge or node and recount, or enumerate every simple cycle — on + every simple graph of up to five nodes and 4,000 random multigraphs, and a + million-node path shows the pass does not recurse. +- Add **node similarity**: `node_similarity` and + `grust.algorithms.nodeSimilarity`, with `metric` (`jaccard`, `overlap`, + `cosine`), `topK`, `topN`, `similarityCutoff`, `degreeCutoff` and + `upperDegreeCutoff`. Nodes are compared by the *distinct* nodes their arcs + reach: parallel edges collapse into one neighbour, present or not without + weights and carrying their summed weight with them, which differs from the + degree-style kernels. A pair sharing nothing is never emitted; a kept pair + appears from both sides because `topK` is per `node1`. Cost is the number of + (node, shared neighbour, node) triples, every one charged, so the work budget + stops the quadratic hub case. Rows and charged work are identical at any pool + width, and the two directions of a pair carry the same bits. Checked row for + row, order and bits included, against an O(n²) recomputation over 3,600 + random runs. +- Result tables can key their own rows, so a kernel can answer with node pairs + (`node1`, `node2`, ...) through the same Arrow and row adapters. A table with + no rows now yields one empty Arrow batch carrying its schema instead of no + batch at all. +- Add **label propagation**: `label_propagation` and + `grust.algorithms.labelPropagation`, with `maxIterations` and `seed`. A node + adopts the label carrying the most weight among the nodes with an arc into it, + so labels flow along the projection's arcs. Updates are asynchronous and + sequential, in row order or a seeded order per pass, which makes a run + reproducible; an asynchronous parallel schedule is not. A node keeps its label + when it is among the heaviest and otherwise takes the smallest of them, which + guarantees convergence on an undirected projection; a directed run reports + `converged` honestly. Tests check that every converged run is a fixed point, + straight from the edge list, over 9,000 random runs. +- Add **closeness** and **harmonic centrality**: `closeness`, `harmonic`, + `grust.algorithms.closeness` (`useWassermanFaust`) and + `grust.algorithms.harmonic` (`normalized`, on by default). Closeness is the + per-component form, `r / Σd` over the `r` nodes reached and zero when none + are, with the Wasserman-Faust `r / (n-1)` correction on request; harmonic sums + `1/d` and needs no convention for disconnected graphs. Distances run from the + node along the projection's arcs, by hop count or by weight; project with the + opposite orientation for distances to the node. Zero weights are rejected. + One sweep per node on the caller's rayon pool, bit-identical at any width, + checked against Floyd-Warshall on every graph of up to five nodes and 600 + random multigraphs of up to forty. +- Add **betweenness centrality**: `betweenness` and + `grust.algorithms.betweenness`, with `samplingSize`, `seed` and `normalized`. + Brandes' algorithm, by hop count on an unweighted projection and by Dijkstra + on a weighted one, in every orientation; undirected sums are halved. Parallel + edges are distinct shortest paths and each counts. Weighted ties are exact + `f64` equality, which is exact for integral weights and approximate otherwise, + and a zero weight is rejected because it makes the path count ill-defined. + `samplingSize` draws that many sources without replacement and scales the sum + by `n / samplingSize`; a sample of every node is the exact result bit for bit. + Sources run on the caller's rayon pool in fixed blocks of 64 merged in block + order, so scores and charged work are bit-identical at any pool width; memory + held at once grows with the pool, one workspace per running block. The oracle + is an all-pairs definition with no traversal in it, checked on every graph of + up to five nodes and 1,500 random multigraphs, weighted and not, in all three + orientations. +- `GraphProjection` can build in-arcs with weights and edge slots once per + projection (crate-internal; label propagation uses them, and the eigenvector, + Katz and HITS kernels will). +- Add **Louvain** community detection: `louvain` and `grust.algorithms.louvain`, + with `resolution`, `maxLevels`, `maxIterations`, `tolerance` and `seed`. It + works in every orientation: Newman's modularity on an undirected projection, + Leicht and Newman's directed modularity otherwise, from one formula. Parallel + edges add their weights; an undirected self-loop counts twice in its node's + strength, the convention that the two-node test pins. Communities are named by + their smallest member, as `wcc` names components, and the reported modularity + is recomputed from the projection rather than accumulated from the moves. + Moves are applied one at a time in row order or a seeded order, so the result + is reproducible and independent of the rayon pool; asynchronous parallel moves + are not. The oracle computes modularity from a dense matrix and the true + optimum by enumerating every partition: the result is never above the optimum, + never below its starting point, and equals the optimum on planted cliques. +- Add **triangle count** and **local clustering coefficient**: `triangles`, + `grust.algorithms.triangleCount` and `grust.algorithms.localClusteringCoefficient`. + Triangles belong to the simple graph, so parallel edges count once and + self-loops are ignored, unlike `degree` and `kCore`. A coefficient is null, + not zero, where it is undefined. `maxDegree` leaves hubs out: such a node + reports `-1` and triangles through it are counted for no one. The + degree-ordered forward algorithm finds each triangle once and runs on the + caller's rayon pool over blocks of equal estimated work; the counts and the + work charged are identical at any pool width, which a test checks at 1, 2, 3 + and 8 threads. The oracle is the O(n^3) definition over every simple graph on + six nodes and every four-node multigraph with loops. +- `grust-algorithms` gains a default `parallel` feature (rayon). A kernel runs on + whatever pool the caller installs; without the feature it runs sequentially + with the same results. +- Add **k-core decomposition**, `k_core` and `grust.algorithms.kCore`: bucket + peeling in O(V + A), returning `coreValue` per node and the graph's + `degeneracy`. Degree counts parallel edges with multiplicity and ignores + self-loops. It is defined on undirected graphs and rejects any other + orientation instead of symmetrizing silently. The test oracle applies the + definition by brute force to all 1,458 four-node multigraphs with up to two + edges per pair and to every simple graph on five nodes. +- Add `NodeTable`, a node-aligned result with typed named columns and repeated + whole-result scalars, with one Arrow adapter and one row adapter. Kernels that + answer "a value per node" hand it their buffers without copying, so the rest of + the analytics catalog needs no adapter code of its own. +- The workspace CI job tests `grust-ladybug` on its own. It had failed on every + push: LadybugDB's prebuilt library bundles zstd, and the unified all-features + build linked `zstd-sys` into the same test binary, which `rust-lld` rejects. +- `grust-algorithm-procedures` gains `run_on_projection(name, &GraphProjection, + &ValidatedArguments)`, `projection_options(&ValidatedArguments)` and + `projection_kernel_names()`, behind a new `arrow` feature the facade's `arrow` + feature forwards. A caller that already holds a projection runs any registered + kernel by name and receives its typed Arrow results, so an embedding such as + Nutmeg needs no per-algorithm dispatch and serves a kernel the day it is + registered. Registration and direct execution read one catalog, and a test + requires every kernel's Arrow columns to equal its declared outputs. + `projectionStats` and `estimateCsr` are refused there: they are not kernels + over a projection. ## 0.21.0 — Tadpole — 2026-09-18 ### Cypher language diff --git a/Cargo.lock b/Cargo.lock index 4b24d01b..878f5d69 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4284,6 +4284,7 @@ dependencies = [ name = "grust-algorithm-procedures" version = "0.21.0" dependencies = [ + "arrow-array 59.3.0", "grust-algorithms", "grust-core", "grust-cypher", @@ -4300,6 +4301,7 @@ dependencies = [ "grust-arrow", "grust-core", "grust-procedures", + "rayon", ] [[package]] diff --git a/PUBLISH.md b/PUBLISH.md index da5feb42..7069fd7a 100644 --- a/PUBLISH.md +++ b/PUBLISH.md @@ -65,6 +65,18 @@ scripts/verify-package-attribution.sh git diff --check ``` +On Linux, test `grust-ladybug` separately, as CI does: `cargo test --workspace +--all-features --exclude grust-ladybug` and then `cargo test -p grust-ladybug`. +LadybugDB's prebuilt library bundles zstd, and the unified all-features build +links `zstd-sys` into the same test binary, which `rust-lld` rejects as +duplicate symbols. macOS's linker accepts it, so the single command works there. + +The benchmark runner's offline tests (`cargo test --manifest-path +benchmarks/lsqb/Cargo.toml`) read the pinned LSQB example dataset from +`benchmarks/lsqb/upstream/lsqb`, which is not checked in. Run +`benchmarks/lsqb/fetch-upstream.sh` first, as CI does: it downloads the pinned +commit's archive and refuses it unless its size and SHA-256 match. + Run [`scripts/integration-test.sh`](scripts/integration-test.sh) for affected live backends. A release that changes backend behavior should record the exact profile, service/image revisions, and any intentionally unrun backend. An diff --git a/benchmarks/lsqb/fetch-upstream.sh b/benchmarks/lsqb/fetch-upstream.sh new file mode 100755 index 00000000..cb8de152 --- /dev/null +++ b/benchmarks/lsqb/fetch-upstream.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Fetch the pinned LSQB source into upstream/lsqb, which is not checked in. The +# offline tests read its example dataset, Cypher sources and expected output. +# The commit, size and digest are the ones README.md and run-upstream.sh pin. +set -euo pipefail + +root=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P) +readonly commit=242cb2fd31340ca688954cb94794d74c0d5b6f92 +readonly url="https://codeload.github.com/ldbc/lsqb/tar.gz/${commit}" +readonly sha256=db17ee8b0a8559d6cb7c06e1388e6d89cee2ac924779473ac847965c0c0d37bb +readonly bytes=2861380 +readonly target="${root}/upstream/lsqb" + +if [[ -d "${target}/data/social-network-sfexample-projected-fk" ]]; then + echo "fetch-upstream.sh: ${target} is already present" + exit 0 +fi + +work=$(mktemp -d) +trap 'rm -rf -- "$work"' EXIT +curl --fail --silent --show-error --location --retry 3 --output "${work}/lsqb.tar.gz" "$url" + +actual_bytes=$(wc -c < "${work}/lsqb.tar.gz" | tr -d ' ') +if command -v sha256sum >/dev/null; then + actual_sha256=$(sha256sum "${work}/lsqb.tar.gz" | cut -d' ' -f1) +else + actual_sha256=$(shasum -a 256 "${work}/lsqb.tar.gz" | cut -d' ' -f1) +fi +if [[ "$actual_bytes" != "$bytes" || "$actual_sha256" != "$sha256" ]]; then + echo "fetch-upstream.sh: archive is ${actual_bytes} bytes, ${actual_sha256};" \ + "expected ${bytes} bytes, ${sha256}" >&2 + exit 1 +fi + +tar -xzf "${work}/lsqb.tar.gz" -C "$work" +mkdir -p -- "${root}/upstream" +mv -- "${work}/lsqb-${commit}" "$target" +echo "fetch-upstream.sh: ${target} is LSQB ${commit}" diff --git a/benchmarks/lsqb/src/observation_process/cleanup_tests.rs b/benchmarks/lsqb/src/observation_process/cleanup_tests.rs index 306b8739..7609c39e 100644 --- a/benchmarks/lsqb/src/observation_process/cleanup_tests.rs +++ b/benchmarks/lsqb/src/observation_process/cleanup_tests.rs @@ -27,13 +27,16 @@ fn post_result_cleanup_is_recovery_not_query_time() { #[test] fn failed_cleanup_invalidates_an_already_emitted_result() { let error = run(&mut worker("exit 1"), "cleanup", 500, 10, 500, 500).unwrap_err(); - assert!(error.contains("failed after writing its result")); + assert!(error.contains("failed after writing its result"), "{error}"); } #[test] fn hung_cleanup_cannot_hold_the_coordinator_indefinitely() { let started = Instant::now(); let error = run(&mut worker("sleep 10"), "cleanup", 500, 10, 50, 500).unwrap_err(); - assert!(error.contains("did not exit within the reap grace")); + assert!( + error.contains("did not exit within the reap grace"), + "{error}" + ); assert!(started.elapsed() < Duration::from_secs(2)); } diff --git a/crates/grust-algorithm-procedures/Cargo.toml b/crates/grust-algorithm-procedures/Cargo.toml index 4a1f404b..c0cf9b6a 100644 --- a/crates/grust-algorithm-procedures/Cargo.toml +++ b/crates/grust-algorithm-procedures/Cargo.toml @@ -9,6 +9,11 @@ description = "Registry adapters for Grust's reusable property-graph algorithms. keywords.workspace = true categories.workspace = true +[features] +default = [] +# Typed Arrow results for callers that hold a projection (`run_on_projection`). +arrow = ["grust-algorithms/arrow"] + [dependencies] grust-algorithms = { version = "0.21.0", path = "../grust-algorithms" } grust-procedures = { version = "0.21.0", path = "../grust-procedures" } @@ -16,7 +21,12 @@ grust-core = { version = "0.21.0", path = "../grust-core" } serde_json.workspace = true [dev-dependencies] +arrow-array = "59.3.0" grust-cypher = { version = "0.21.0", path = "../grust-cypher" } [lints.rust] unsafe_code = "forbid" + +[[test]] +name = "run_on_projection" +required-features = ["arrow"] diff --git a/crates/grust-algorithm-procedures/src/lib.rs b/crates/grust-algorithm-procedures/src/lib.rs index e14fee45..cdccacab 100644 --- a/crates/grust-algorithm-procedures/src/lib.rs +++ b/crates/grust-algorithm-procedures/src/lib.rs @@ -16,8 +16,17 @@ use output::{AlgorithmCursor, AlgorithmOutput}; /// hardcoded name dispatcher are needed when applications register more providers. pub fn register_algorithms(builder: &mut RegistryBuilder) -> Result<()> { statistics::register(builder)?; - register( - builder, + for spec in catalog() { + register(builder, spec)?; + } + Ok(()) +} + +/// Every projection kernel, once: registration and direct execution both read +/// this list, so a kernel cannot be registered without being runnable. +fn catalog() -> Vec { + let mut specs = Vec::new(); + specs.push(Spec::new( "degree", None, vec![ @@ -27,9 +36,8 @@ pub fn register_algorithms(builder: &mut RegistryBuilder) -> Result<()> { ], vec![], |graph, _| Ok(AlgorithmOutput::Degrees(algorithms::degree(graph)?)), - )?; - register( - builder, + )); + specs.push(Spec::new( "bfs", Some(ValueType::String), vec![ @@ -43,9 +51,8 @@ pub fn register_algorithms(builder: &mut RegistryBuilder) -> Result<()> { source(args)?, )?)) }, - )?; - register( - builder, + )); + specs.push(Spec::new( "dijkstra", Some(ValueType::String), vec![ @@ -59,9 +66,8 @@ pub fn register_algorithms(builder: &mut RegistryBuilder) -> Result<()> { source(args)?, )?)) }, - )?; - register( - builder, + )); + specs.push(Spec::new( "shortestPaths", Some(ValueType::String), vec![ @@ -74,13 +80,13 @@ pub fn register_algorithms(builder: &mut RegistryBuilder) -> Result<()> { ], vec![], |graph, args| { - Ok(AlgorithmOutput::Paths( - algorithms::shortest_paths(graph, source(args)?)?.into_cursor()?, - )) + Ok(AlgorithmOutput::Paths(algorithms::shortest_paths( + graph, + source(args)?, + )?)) }, - )?; - register( - builder, + )); + specs.push(Spec::new( "wcc", None, vec![ @@ -93,9 +99,8 @@ pub fn register_algorithms(builder: &mut RegistryBuilder) -> Result<()> { algorithms::weakly_connected_components(graph)?, )) }, - )?; - register( - builder, + )); + specs.push(Spec::new( "scc", None, vec![ @@ -108,9 +113,8 @@ pub fn register_algorithms(builder: &mut RegistryBuilder) -> Result<()> { algorithms::strongly_connected_components(graph)?, )) }, - )?; - register( - builder, + )); + specs.push(Spec::new( "pagerank", None, vec![ @@ -127,9 +131,8 @@ pub fn register_algorithms(builder: &mut RegistryBuilder) -> Result<()> { options::pagerank(args)?, )?)) }, - )?; - register( - builder, + )); + specs.push(Spec::new( "dfs", Some(ValueType::String), vec![ @@ -143,9 +146,8 @@ pub fn register_algorithms(builder: &mut RegistryBuilder) -> Result<()> { source(args)?, )?)) }, - )?; - register( - builder, + )); + specs.push(Spec::new( "multiSourceBfs", Some(ValueType::Strings), vec![ @@ -163,9 +165,8 @@ pub fn register_algorithms(builder: &mut RegistryBuilder) -> Result<()> { graph, sources, )?)) }, - )?; - register( - builder, + )); + specs.push(Spec::new( "topologicalSort", None, vec![ @@ -179,7 +180,327 @@ pub fn register_algorithms(builder: &mut RegistryBuilder) -> Result<()> { graph, )?)) }, - ) + )); + specs.push(Spec::new( + "kCore", + None, + vec![ + field("nodeId", ValueType::String), + field("coreValue", ValueType::Integer), + field("degeneracy", ValueType::Integer), + ], + vec![], + |graph, _| { + Ok(AlgorithmOutput::Table( + algorithms::k_core(graph)?.into_table(), + )) + }, + )); + specs.push(Spec::new( + "triangleCount", + None, + vec![ + field("nodeId", ValueType::String), + field("triangles", ValueType::Integer), + field("triangleCount", ValueType::Integer), + ], + options::triangle_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::triangles(graph, options::triangles(args)?)?.into_triangle_table()?, + )) + }, + )); + specs.push(Spec::new( + "localClusteringCoefficient", + None, + vec![ + field("nodeId", ValueType::String), + nullable("coefficient", ValueType::Number), + field("triangles", ValueType::Integer), + field("averageCoefficient", ValueType::Number), + ], + options::triangle_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::triangles(graph, options::triangles(args)?)? + .into_coefficient_table()?, + )) + }, + )); + specs.push(Spec::new( + "betweenness", + None, + vec![ + field("nodeId", ValueType::String), + field("score", ValueType::Number), + ], + options::betweenness_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::betweenness(graph, options::betweenness(args)?)?.into_table()?, + )) + }, + )); + specs.push(Spec::new( + "closeness", + None, + vec![ + field("nodeId", ValueType::String), + field("score", ValueType::Number), + ], + options::closeness_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::closeness(graph, options::closeness(args)?)?.into_table()?, + )) + }, + )); + specs.push(Spec::new( + "harmonic", + None, + vec![ + field("nodeId", ValueType::String), + field("score", ValueType::Number), + ], + options::harmonic_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::harmonic(graph, options::harmonic(args)?)?.into_table()?, + )) + }, + )); + specs.push(Spec::new( + "labelPropagation", + None, + vec![ + field("nodeId", ValueType::String), + field("communityId", ValueType::String), + field("iterations", ValueType::Integer), + field("converged", ValueType::Boolean), + ], + options::label_propagation_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::label_propagation(graph, options::label_propagation(args)?)? + .into_table()?, + )) + }, + )); + specs.push(Spec::new( + "nodeSimilarity", + None, + vec![ + field("node1", ValueType::String), + field("node2", ValueType::String), + field("similarity", ValueType::Number), + ], + options::node_similarity_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::node_similarity(graph, options::node_similarity(args)?)? + .into_table()?, + )) + }, + )); + specs.push(Spec::new( + "bridges", + None, + vec![ + field("sourceNodeId", ValueType::String), + field("targetNodeId", ValueType::String), + field("edgeOrdinal", ValueType::Integer), + ], + vec![], + |graph, _| { + Ok(AlgorithmOutput::Table( + algorithms::biconnectivity(graph)?.into_bridge_table()?, + )) + }, + )); + specs.push(Spec::new( + "articulationPoints", + None, + vec![field("nodeId", ValueType::String)], + vec![], + |graph, _| { + Ok(AlgorithmOutput::Table( + algorithms::biconnectivity(graph)?.into_articulation_table()?, + )) + }, + )); + specs.push(Spec::new( + "biconnectedComponents", + None, + vec![ + field("sourceNodeId", ValueType::String), + field("targetNodeId", ValueType::String), + field("edgeOrdinal", ValueType::Integer), + field("componentId", ValueType::Integer), + ], + vec![], + |graph, _| { + Ok(AlgorithmOutput::Table( + algorithms::biconnectivity(graph)?.into_component_table()?, + )) + }, + )); + specs.push(Spec::new( + "spanningTree", + None, + vec![ + field("sourceNodeId", ValueType::String), + field("targetNodeId", ValueType::String), + field("edgeOrdinal", ValueType::Integer), + field("weight", ValueType::Number), + field("totalWeight", ValueType::Number), + ], + options::spanning_tree_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::spanning_tree(graph, options::spanning_tree(args)?)?.into_table()?, + )) + }, + )); + specs.push(Spec::new( + "eigenvector", + None, + vec![ + field("nodeId", ValueType::String), + field("score", ValueType::Number), + field("iterations", ValueType::Integer), + field("converged", ValueType::Boolean), + field("residual", ValueType::Number), + ], + options::iteration_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::eigenvector(graph, options::iteration(args)?)?.into_table()?, + )) + }, + )); + specs.push(Spec::new( + "katz", + None, + vec![ + field("nodeId", ValueType::String), + field("score", ValueType::Number), + field("iterations", ValueType::Integer), + field("converged", ValueType::Boolean), + field("residual", ValueType::Number), + ], + options::katz_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::katz(graph, options::katz(args)?)?.into_table()?, + )) + }, + )); + specs.push(Spec::new( + "hits", + None, + vec![ + field("nodeId", ValueType::String), + field("hub", ValueType::Number), + field("authority", ValueType::Number), + field("iterations", ValueType::Integer), + field("converged", ValueType::Boolean), + field("residual", ValueType::Number), + ], + options::iteration_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::hits(graph, options::iteration(args)?)?.into_table()?, + )) + }, + )); + specs.push( + Spec::new( + "maxFlow", + Some(ValueType::String), + vec![ + field("sourceNodeId", ValueType::String), + field("targetNodeId", ValueType::String), + field("edgeOrdinal", ValueType::Integer), + field("flow", ValueType::Number), + field("maxFlow", ValueType::Number), + ], + vec![], + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::max_flow(graph, source(args)?, target(args)?)?.into_flow_table()?, + )) + }, + ) + .with_target(), + ); + specs.push( + Spec::new( + "minCut", + Some(ValueType::String), + vec![ + field("nodeId", ValueType::String), + field("sourceSide", ValueType::Boolean), + field("maxFlow", ValueType::Number), + ], + vec![], + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::max_flow(graph, source(args)?, target(args)?)?.into_cut_table()?, + )) + }, + ) + .with_target(), + ); + specs.push(Spec::new( + "fastRP", + None, + vec![ + field("nodeId", ValueType::String), + field("embedding", ValueType::Numbers), + ], + options::fast_rp_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::fast_rp(graph, options::fast_rp(args)?)?.into_table()?, + )) + }, + )); + specs.push(Spec::new( + "leiden", + None, + vec![ + field("nodeId", ValueType::String), + field("communityId", ValueType::String), + field("modularity", ValueType::Number), + field("levels", ValueType::Integer), + field("converged", ValueType::Boolean), + ], + options::louvain_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::leiden(graph, options::louvain(args)?)?.into_table()?, + )) + }, + )); + specs.push(Spec::new( + "louvain", + None, + vec![ + field("nodeId", ValueType::String), + field("communityId", ValueType::String), + field("modularity", ValueType::Number), + field("levels", ValueType::Integer), + field("converged", ValueType::Boolean), + ], + options::louvain_fields(), + |graph, args| { + Ok(AlgorithmOutput::Table( + algorithms::louvain(graph, options::louvain(args)?)?.into_table()?, + )) + }, + )); + specs } fn field(name: &str, value_type: ValueType) -> Field { @@ -201,14 +522,91 @@ struct Provider { kernel: Kernel, } -fn register( - builder: &mut RegistryBuilder, - name: &str, +struct Spec { + name: &'static str, source: Option, + /// A second positional node id, `target`, after `source`. + target: bool, outputs: Vec, extra_options: Vec, kernel: Kernel, -) -> Result<()> { +} + +impl Spec { + fn new( + name: &'static str, + source: Option, + outputs: Vec, + extra_options: Vec, + kernel: Kernel, + ) -> Self { + Self { + name, + source, + target: false, + outputs, + extra_options, + kernel, + } + } +} + +impl Spec { + fn with_target(mut self) -> Self { + self.target = true; + self + } +} + +const PREFIX: &str = "grust.algorithms."; + +/// The projection options a validated call carries: orientation, label and +/// relationship-type selection, and the weight property with its default. A +/// caller that builds its own `GraphProjection` reads them with this, so its +/// projection means what the registered procedure's would. +pub fn projection_options(args: &ValidatedArguments) -> Result> { + options::projection(args) +} + +/// Run a registered kernel on a projection the caller already holds, and hand +/// back its typed Arrow results. `name` is the registered name, in any ASCII +/// case, with or without the `grust.algorithms.` prefix. `args` must have been validated against that +/// procedure's definition, exactly as the registry validates a `CALL`. +/// +/// `projectionStats` and `estimateCsr` are not kernels over a projection: one +/// reads projection metadata and the other sizes a graph before it is built. +/// They are refused here; call `GraphProjection::statistics` or +/// `CsrEstimate::upper_bound`. +#[cfg(feature = "arrow")] +pub fn run_on_projection( + name: &str, + graph: &algorithms::GraphProjection, + args: &ValidatedArguments, +) -> Result { + let short = name.strip_prefix(PREFIX).unwrap_or(name); + let spec = catalog() + .into_iter() + .find(|spec| spec.name.eq_ignore_ascii_case(short)) + .ok_or_else(|| { + ProcedureError::Unsupported(format!("`{name}` is not a registered projection kernel")) + })?; + (spec.kernel)(graph, args)?.into_arrow_results() +} + +/// Names `run_on_projection` serves, without the prefix, in registration order. +pub fn projection_kernel_names() -> Vec<&'static str> { + catalog().into_iter().map(|spec| spec.name).collect() +} + +fn register(builder: &mut RegistryBuilder, spec: Spec) -> Result<()> { + let Spec { + name, + source, + target, + outputs, + extra_options, + kernel, + } = spec; let mut arguments = Vec::new(); if let Some(source_type) = source { arguments.push(Argument { @@ -223,6 +621,12 @@ fn register( default: None, }); } + if target { + arguments.push(Argument { + field: field("target", ValueType::String), + default: None, + }); + } let options_argument = arguments.len(); arguments.push(Argument { field: field("configuration", ValueType::Map), @@ -232,7 +636,7 @@ fn register( options.extend(extra_options); builder.register( ProcedureDefinition { - name: format!("grust.algorithms.{name}"), + name: format!("{PREFIX}{name}"), aliases: vec![], version: 1, provider: "grust.algorithms".into(), @@ -261,7 +665,16 @@ impl ProcedureProvider for Provider { })?; let graph = preparation::prepare(snapshot, options::projection(&args)?, &invocation)?; let output = (self.kernel)(&graph, &args)?; - Ok(Box::new(AlgorithmCursor::new(graph, output))) + Ok(Box::new(AlgorithmCursor::new(graph, output)?)) + } +} + +fn target(args: &ValidatedArguments) -> Result<&str> { + match args.positional().get(1) { + Some(Value::String(target)) => Ok(target), + _ => Err(ProcedureError::InvalidArguments( + "target must be an external node ID string".into(), + )), } } diff --git a/crates/grust-algorithm-procedures/src/options.rs b/crates/grust-algorithm-procedures/src/options.rs index 4222ab6e..581e2b23 100644 --- a/crates/grust-algorithm-procedures/src/options.rs +++ b/crates/grust-algorithm-procedures/src/options.rs @@ -1,5 +1,10 @@ use super::*; -use algorithms::{MissingWeight, Orientation, PageRankOptions, ProjectionOptions, WeightSelection}; +use algorithms::{ + BetweennessOptions, ClosenessOptions, FastRpOptions, HarmonicOptions, IterationOptions, + KatzOptions, LabelPropagationOptions, LouvainOptions, MissingWeight, NodeSimilarityOptions, + Orientation, PageRankOptions, ProjectionOptions, SimilarityMetric, SpanningObjective, + SpanningTreeOptions, TriangleOptions, WeightSelection, +}; fn option(name: &str, value_type: ValueType, default: Value, nullable: bool) -> OptionField { OptionField { @@ -130,3 +135,305 @@ pub(super) fn pagerank(args: &ValidatedArguments) -> Result> personalization, }) } + +pub(super) fn triangle_fields() -> Vec { + vec![option("maxDegree", ValueType::Integer, Value::Null, true)] +} + +pub(super) fn triangles(args: &ValidatedArguments) -> Result { + let max_degree = match value(args, "maxDegree")? { + Value::Null => None, + Value::Int(value) => Some(usize::try_from(*value).map_err(|_| { + ProcedureError::InvalidArguments("maxDegree must not be negative".into()) + })?), + _ => { + return Err(ProcedureError::InvalidArguments( + "maxDegree must be an integer".into(), + )); + } + }; + Ok(TriangleOptions { max_degree }) +} + +pub(super) fn louvain_fields() -> Vec { + vec![ + option("resolution", ValueType::Number, Value::Float(1.0), false), + option("maxLevels", ValueType::Integer, Value::Int(10), false), + option("maxIterations", ValueType::Integer, Value::Int(10), false), + option("tolerance", ValueType::Number, Value::Float(1e-4), false), + option("seed", ValueType::Integer, Value::Null, true), + ] +} + +fn positive(args: &ValidatedArguments, key: &str) -> Result { + match value(args, key)? { + Value::Int(value) if *value > 0 => usize::try_from(*value) + .map_err(|_| ProcedureError::InvalidArguments(format!("{key} is too large"))), + _ => Err(ProcedureError::InvalidArguments(format!( + "{key} must be a positive integer" + ))), + } +} + +/// `null` is no seed; any integer is one, and its bits are what matter. +fn seed(args: &ValidatedArguments) -> Result> { + match value(args, "seed")? { + Value::Null => Ok(None), + Value::Int(value) => Ok(Some(*value as u64)), + _ => Err(ProcedureError::InvalidArguments( + "seed must be an integer".into(), + )), + } +} + +pub(super) fn betweenness_fields() -> Vec { + vec![ + option("samplingSize", ValueType::Integer, Value::Null, true), + option("seed", ValueType::Integer, Value::Null, true), + option("normalized", ValueType::Boolean, Value::Bool(false), false), + ] +} + +pub(super) fn betweenness(args: &ValidatedArguments) -> Result { + Ok(BetweennessOptions { + sampling_size: match value(args, "samplingSize")? { + Value::Null => None, + _ => Some(positive(args, "samplingSize")?), + }, + seed: seed(args)?.unwrap_or(0), + normalized: matches!(value(args, "normalized")?, Value::Bool(true)), + }) +} + +pub(super) fn closeness_fields() -> Vec { + vec![option( + "useWassermanFaust", + ValueType::Boolean, + Value::Bool(false), + false, + )] +} + +pub(super) fn closeness(args: &ValidatedArguments) -> Result { + Ok(ClosenessOptions { + wasserman_faust: matches!(value(args, "useWassermanFaust")?, Value::Bool(true)), + }) +} + +pub(super) fn harmonic_fields() -> Vec { + vec![option( + "normalized", + ValueType::Boolean, + Value::Bool(true), + false, + )] +} + +pub(super) fn harmonic(args: &ValidatedArguments) -> Result { + Ok(HarmonicOptions { + normalized: matches!(value(args, "normalized")?, Value::Bool(true)), + }) +} + +pub(super) fn label_propagation_fields() -> Vec { + vec![ + option("maxIterations", ValueType::Integer, Value::Int(10), false), + option("seed", ValueType::Integer, Value::Null, true), + ] +} + +pub(super) fn label_propagation(args: &ValidatedArguments) -> Result { + Ok(LabelPropagationOptions { + max_iterations: positive(args, "maxIterations")?, + seed: seed(args)?, + }) +} + +pub(super) fn node_similarity_fields() -> Vec { + vec![ + option( + "metric", + ValueType::String, + Value::String("jaccard".into()), + false, + ), + option("topK", ValueType::Integer, Value::Int(10), false), + option("topN", ValueType::Integer, Value::Int(0), false), + option( + "similarityCutoff", + ValueType::Number, + Value::Float(0.0), + false, + ), + option("degreeCutoff", ValueType::Integer, Value::Int(1), false), + option("upperDegreeCutoff", ValueType::Integer, Value::Null, true), + ] +} + +pub(super) fn node_similarity(args: &ValidatedArguments) -> Result { + let metric = match value(args, "metric")? { + Value::String(value) => match value.to_ascii_lowercase().as_str() { + "jaccard" => SimilarityMetric::Jaccard, + "overlap" => SimilarityMetric::Overlap, + "cosine" => SimilarityMetric::Cosine, + _ => { + return Err(ProcedureError::InvalidArguments( + "metric must be jaccard, overlap or cosine".into(), + )); + } + }, + _ => { + return Err(ProcedureError::InvalidArguments( + "metric must be a string".into(), + )); + } + }; + let top_n = match value(args, "topN")? { + Value::Int(0) => 0, + _ => positive(args, "topN")?, + }; + Ok(NodeSimilarityOptions { + metric, + top_k: positive(args, "topK")?, + top_n, + similarity_cutoff: number(value(args, "similarityCutoff")?)?, + degree_cutoff: positive(args, "degreeCutoff")?, + upper_degree_cutoff: match value(args, "upperDegreeCutoff")? { + Value::Null => None, + _ => Some(positive(args, "upperDegreeCutoff")?), + }, + }) +} + +pub(super) fn spanning_tree_fields() -> Vec { + vec![ + option( + "objective", + ValueType::String, + Value::String("minimum".into()), + false, + ), + option("sourceNode", ValueType::String, Value::Null, true), + ] +} + +pub(super) fn spanning_tree(args: &ValidatedArguments) -> Result> { + let objective = match value(args, "objective")? { + Value::String(value) => match value.to_ascii_lowercase().as_str() { + "minimum" => SpanningObjective::Minimum, + "maximum" => SpanningObjective::Maximum, + _ => { + return Err(ProcedureError::InvalidArguments( + "objective must be minimum or maximum".into(), + )); + } + }, + _ => { + return Err(ProcedureError::InvalidArguments( + "objective must be a string".into(), + )); + } + }; + let source = match value(args, "sourceNode")? { + Value::Null => None, + Value::String(id) => Some(id.as_str()), + _ => { + return Err(ProcedureError::InvalidArguments( + "sourceNode must be a node id".into(), + )); + } + }; + Ok(SpanningTreeOptions { objective, source }) +} + +pub(super) fn iteration_fields() -> Vec { + vec![ + option("tolerance", ValueType::Number, Value::Float(1e-8), false), + option("maxIterations", ValueType::Integer, Value::Int(1000), false), + ] +} + +pub(super) fn iteration(args: &ValidatedArguments) -> Result { + Ok(IterationOptions { + tolerance: number(value(args, "tolerance")?)?, + max_iterations: positive(args, "maxIterations")?, + }) +} + +pub(super) fn katz_fields() -> Vec { + let mut fields = vec![ + option("alpha", ValueType::Number, Value::Float(0.1), false), + option("beta", ValueType::Number, Value::Float(1.0), false), + option("normalized", ValueType::Boolean, Value::Bool(false), false), + ]; + fields.extend(iteration_fields()); + fields +} + +pub(super) fn katz(args: &ValidatedArguments) -> Result { + Ok(KatzOptions { + alpha: number(value(args, "alpha")?)?, + beta: number(value(args, "beta")?)?, + normalized: matches!(value(args, "normalized")?, Value::Bool(true)), + iteration: iteration(args)?, + }) +} + +pub(super) fn fast_rp_fields() -> Vec { + vec![ + option( + "embeddingDimension", + ValueType::Integer, + Value::Int(128), + false, + ), + option( + "iterationWeights", + ValueType::Numbers, + Value::FloatArray(vec![0.0, 1.0, 1.0]), + false, + ), + option( + "nodeSelfInfluence", + ValueType::Number, + Value::Float(0.0), + false, + ), + option( + "normalizationStrength", + ValueType::Number, + Value::Float(0.0), + false, + ), + option("seed", ValueType::Integer, Value::Null, true), + ] +} + +pub(super) fn fast_rp(args: &ValidatedArguments) -> Result> { + let iteration_weights = match value(args, "iterationWeights")? { + Value::FloatArray(values) => values.as_slice(), + _ => { + return Err(ProcedureError::InvalidArguments( + "iterationWeights must be a numeric array".into(), + )); + } + }; + Ok(FastRpOptions { + dimension: positive(args, "embeddingDimension")?, + iteration_weights, + self_influence: number(value(args, "nodeSelfInfluence")?)?, + normalization_strength: number(value(args, "normalizationStrength")?)?, + seed: seed(args)?.unwrap_or(0), + }) +} + +pub(super) fn louvain(args: &ValidatedArguments) -> Result { + let seed = seed(args)?; + Ok(LouvainOptions { + resolution: number(value(args, "resolution")?)?, + max_levels: positive(args, "maxLevels")?, + max_iterations: positive(args, "maxIterations")?, + tolerance: number(value(args, "tolerance")?)?, + seed, + }) +} diff --git a/crates/grust-algorithm-procedures/src/output.rs b/crates/grust-algorithm-procedures/src/output.rs index dd0be5cf..033c634f 100644 --- a/crates/grust-algorithm-procedures/src/output.rs +++ b/crates/grust-algorithm-procedures/src/output.rs @@ -7,9 +7,35 @@ pub(super) enum AlgorithmOutput { Components(algorithms::Components), PageRank(algorithms::PageRank), Degrees(algorithms::Degrees), - Paths(algorithms::PathCursor), + Paths(algorithms::ShortestPaths), + /// Paths being streamed as rows; never produced by a kernel. + PathRows(algorithms::PathCursor), Order(algorithms::NodeOrder), Topology(algorithms::TopologicalOrder), + /// A value per node: the shape most of the catalog returns. + Table(algorithms::NodeTable), +} + +#[cfg(feature = "arrow")] +impl AlgorithmOutput { + /// The same result as typed Arrow, for callers that hold a projection. + pub(super) fn into_arrow_results(self) -> Result { + Ok(match self { + Self::Distances(result) => result.into_arrow_results(), + Self::Components(result) => result.into_arrow_results(), + Self::PageRank(result) => result.into_arrow_results(), + Self::Degrees(result) => result.into_arrow_results(), + Self::Paths(result) => result.into_arrow_results()?, + Self::Order(result) => result.into_arrow_results(), + Self::Topology(result) => result.into_arrow_results(), + Self::Table(result) => result.into_arrow_results(), + Self::PathRows(_) => { + return Err(ProcedureError::OutputContract( + "a kernel returned a row cursor instead of its result".into(), + )); + } + }) + } } pub(super) struct AlgorithmCursor { @@ -19,12 +45,16 @@ pub(super) struct AlgorithmCursor { } impl AlgorithmCursor { - pub(super) fn new(graph: algorithms::GraphProjection, output: AlgorithmOutput) -> Self { - Self { + pub(super) fn new(graph: algorithms::GraphProjection, output: AlgorithmOutput) -> Result { + let output = match output { + AlgorithmOutput::Paths(paths) => AlgorithmOutput::PathRows(paths.into_cursor()?), + other => other, + }; + Ok(Self { graph, output, next: 0, - } + }) } } @@ -32,7 +62,7 @@ impl ProcedureCursor for AlgorithmCursor { fn next_batch(&mut self) -> Result> { let context = self.graph.execution(); context.checkpoint()?; - if let AlgorithmOutput::Paths(cursor) = &mut self.output { + if let AlgorithmOutput::PathRows(cursor) = &mut self.output { return match cursor.next_path()? { Some(path) => path_batch(&self.graph, path).map(Some), None => Ok(None), @@ -48,6 +78,7 @@ impl ProcedureCursor for AlgorithmCursor { } let count = match &self.output { AlgorithmOutput::Order(result) => result.values().len(), + AlgorithmOutput::Table(table) => table.rows(), _ => self.graph.node_count(), }; if self.next == count { @@ -57,6 +88,7 @@ impl ProcedureCursor for AlgorithmCursor { context.charge_work(1)?; let node = match &self.output { AlgorithmOutput::Order(result) => result.values()[index], + AlgorithmOutput::Table(table) => table.key(index), _ => index, }; let id = self.graph.node_ids()[node].as_str(); @@ -68,7 +100,21 @@ impl ProcedureCursor for AlgorithmCursor { ), AlgorithmOutput::PageRank(_) => (5, 0), AlgorithmOutput::Degrees(_) => (3, 0), - AlgorithmOutput::Paths(_) | AlgorithmOutput::Topology(_) => { + AlgorithmOutput::Table(table) => ( + 1 + table.width(), + (0..table.width()) + .map(|column| match table.value(column, index) { + algorithms::TableValue::Node(id) => id.as_str().len(), + algorithms::TableValue::Vector(vector) => { + vector.len().saturating_mul(size_of::()) + } + _ => 0, + }) + .sum(), + ), + AlgorithmOutput::Paths(_) + | AlgorithmOutput::PathRows(_) + | AlgorithmOutput::Topology(_) => { return Err(ProcedureError::OutputContract( "path output reached scalar adapter".into(), )); @@ -105,13 +151,32 @@ impl ProcedureCursor for AlgorithmCursor { .map_or(Value::Null, |values| Value::Float(values[index])), ); } + AlgorithmOutput::Table(table) => { + for column in 0..table.width() { + row.push(match table.value(column, index) { + algorithms::TableValue::Integer(value) => Value::Int(value), + algorithms::TableValue::Number(value) => Value::Float(value), + algorithms::TableValue::Boolean(value) => Value::Bool(value), + algorithms::TableValue::Node(id) => Value::String(id.as_str().into()), + algorithms::TableValue::Vector(vector) => { + let mut values = Vec::new(); + values.try_reserve_exact(vector.len())?; + values.extend(vector.iter().map(|&value| f64::from(value))); + Value::FloatArray(values) + } + algorithms::TableValue::Null => Value::Null, + }); + } + } AlgorithmOutput::PageRank(result) => { row.push(Value::Float(result.values()[index])); row.push(Value::Int(integer(result.iterations())?)); row.push(Value::Bool(result.converged())); row.push(Value::Float(result.residual())); } - AlgorithmOutput::Paths(_) | AlgorithmOutput::Topology(_) => { + AlgorithmOutput::Paths(_) + | AlgorithmOutput::PathRows(_) + | AlgorithmOutput::Topology(_) => { return Err(ProcedureError::OutputContract( "path output reached scalar adapter".into(), )); diff --git a/crates/grust-algorithm-procedures/tests/cypher.rs b/crates/grust-algorithm-procedures/tests/cypher.rs index f3db7303..86545a90 100644 --- a/crates/grust-algorithm-procedures/tests/cypher.rs +++ b/crates/grust-algorithm-procedures/tests/cypher.rs @@ -261,3 +261,421 @@ fn degree_counts_and_strengths_use_projection_orientation() { ] ); } + +#[test] +fn k_core_is_an_ordinary_procedure_with_explicit_orientation() { + // The fixture's b<->c pair is two edges, so undirected b and c form a + // 2-core; a hangs off it, and the isolate has core 0. + assert_eq!( + run( + "CALL grust.algorithms.kCore({orientation: 'undirected'}) YIELD nodeId, coreValue, degeneracy RETURN nodeId, coreValue, degeneracy" + ), + vec![ + vec![Value::String("a".into()), Value::Int(1), Value::Int(2)], + vec![Value::String("b".into()), Value::Int(2), Value::Int(2)], + vec![Value::String("c".into()), Value::Int(2), Value::Int(2)], + vec![ + Value::String("isolate".into()), + Value::Int(0), + Value::Int(2) + ], + ] + ); + // Aggregation over the result is ordinary Cypher. + assert_eq!( + run( + "CALL grust.algorithms.kCore({orientation: 'undirected'}) YIELD coreValue RETURN max(coreValue), count(coreValue)" + ), + vec![vec![Value::Int(2), Value::Int(4)]] + ); +} + +#[test] +fn triangles_and_clustering_are_ordinary_procedures_on_the_simple_graph() { + // Undirected, the fixture is the path a-b-c with b-c doubled: no triangle, + // and the doubled edge does not invent one. + assert_eq!( + run( + "CALL grust.algorithms.triangleCount({orientation: 'undirected'}) YIELD nodeId, triangles, triangleCount RETURN nodeId, triangles, triangleCount" + ), + vec![ + vec![Value::String("a".into()), Value::Int(0), Value::Int(0)], + vec![Value::String("b".into()), Value::Int(0), Value::Int(0)], + vec![Value::String("c".into()), Value::Int(0), Value::Int(0)], + vec![ + Value::String("isolate".into()), + Value::Int(0), + Value::Int(0) + ], + ] + ); + // b has two distinct neighbours and no triangle: coefficient 0. The others + // have fewer than two neighbours: no coefficient, which is null, not 0. + assert_eq!( + run( + "CALL grust.algorithms.localClusteringCoefficient({orientation: 'undirected'}) YIELD nodeId, coefficient RETURN nodeId, coefficient" + ), + vec![ + vec![Value::String("a".into()), Value::Null], + vec![Value::String("b".into()), Value::Float(0.0)], + vec![Value::String("c".into()), Value::Null], + vec![Value::String("isolate".into()), Value::Null], + ] + ); + // maxDegree 1 leaves b out: -1 triangles, no coefficient. + assert_eq!( + run( + "CALL grust.algorithms.triangleCount({orientation: 'undirected', maxDegree: 1}) YIELD nodeId, triangles RETURN nodeId, triangles" + )[1], + vec![Value::String("b".into()), Value::Int(-1)] + ); +} + +#[test] +fn louvain_is_an_ordinary_procedure_in_every_orientation() { + // Undirected, the fixture is a-b with b-c doubled: the heavy pair b,c is one + // community and a joins it or not by modularity; the isolate stays alone. + let rows = run( + "CALL grust.algorithms.louvain({orientation: 'undirected'}) YIELD nodeId, communityId, modularity, converged RETURN nodeId, communityId, modularity, converged", + ); + assert_eq!(rows.len(), 4); + assert_eq!(rows[1][1], rows[2][1], "b and c share a community"); + assert_eq!(rows[3][1], Value::String("isolate".into())); + assert!(rows.iter().all(|row| row[3] == Value::Bool(true))); + // The default orientation is directed: Leicht-Newman modularity, not an error. + let directed = + run("CALL grust.algorithms.louvain() YIELD communityId RETURN count(communityId)"); + assert_eq!(directed, vec![vec![Value::Int(4)]]); + // Options are named and typed; a huge resolution leaves every node alone. + assert_eq!( + run( + "CALL grust.algorithms.louvain({orientation: 'undirected', resolution: 1000000.0, seed: 7}) YIELD nodeId, communityId WHERE nodeId <> communityId RETURN count(nodeId)" + ), + vec![vec![Value::Int(0)]] + ); +} + +#[test] +fn betweenness_is_an_ordinary_procedure_exact_or_sampled() { + // a-b, b-c doubled, and an isolate: only b lies between two other nodes. + let expected = vec![ + vec![Value::String("a".into()), Value::Float(0.0)], + vec![Value::String("b".into()), Value::Float(1.0)], + vec![Value::String("c".into()), Value::Float(0.0)], + vec![Value::String("isolate".into()), Value::Float(0.0)], + ]; + let query = |options: &str| { + run(&format!( + "CALL grust.algorithms.betweenness({options}) YIELD nodeId, score RETURN nodeId, score" + )) + }; + assert_eq!(query("{orientation: 'undirected'}"), expected); + // A sample of every node is the exact answer. + assert_eq!( + query("{orientation: 'undirected', samplingSize: 4, seed: 3}"), + expected + ); + // One pair of three possible: a-c, normalised by (n-1)(n-2)/2 = 3. + assert_eq!( + query("{orientation: 'undirected', normalized: true}")[1][1], + Value::Float(1.0 / 3.0) + ); + assert!( + run_read_query_with_registry( + &graph(), + "default", + "CALL grust.algorithms.betweenness({samplingSize: 0}) YIELD score RETURN score", + &CypherParameters::new(), + ®istry(), + ) + .is_err() + ); +} + +#[test] +fn closeness_and_harmonic_are_ordinary_procedures() { + // Undirected a-b-c plus an isolate, so n-1 = 3. + let scores = |call: &str| { + run(&format!( + "CALL grust.algorithms.{call} YIELD nodeId, score RETURN nodeId, score" + )) + .into_iter() + .map(|row| row[1].clone()) + .collect::>() + }; + let floats = |values: [f64; 4]| values.map(Value::Float).to_vec(); + assert_eq!( + scores("closeness({orientation: 'undirected'})"), + floats([2.0 / 3.0, 1.0, 2.0 / 3.0, 0.0]) + ); + assert_eq!( + scores("closeness({orientation: 'undirected', useWassermanFaust: true})"), + floats([4.0 / 9.0, 2.0 / 3.0, 4.0 / 9.0, 0.0]) + ); + assert_eq!( + scores("harmonic({orientation: 'undirected', normalized: false})"), + floats([1.5, 2.0, 1.5, 0.0]) + ); + assert_eq!( + scores("harmonic({orientation: 'undirected'})"), + floats([0.5, 2.0 / 3.0, 0.5, 0.0]) + ); +} + +#[test] +fn label_propagation_is_an_ordinary_procedure() { + let rows = run( + "CALL grust.algorithms.labelPropagation({orientation: 'undirected', maxIterations: 20, seed: 5}) YIELD nodeId, communityId, iterations, converged RETURN nodeId, communityId, converged", + ); + // a-b-c is one connected piece and ends as one community; the isolate is its own. + let community = |row: usize| rows[row][1].clone(); + assert_eq!(community(0), community(1)); + assert_eq!(community(1), community(2)); + assert_eq!(community(3), Value::String("isolate".into())); + assert!(rows.iter().all(|row| row[2] == Value::Bool(true))); + // Directed by default, and still an answer rather than an error. + assert_eq!( + run("CALL grust.algorithms.labelPropagation() YIELD communityId RETURN count(communityId)"), + vec![vec![Value::Int(4)]] + ); +} + +#[test] +fn node_similarity_returns_pair_rows_through_ordinary_cypher() { + // Undirected a-b-c: a and c share their only neighbour, b. + let pairs = vec![ + vec![ + Value::String("a".into()), + Value::String("c".into()), + Value::Float(1.0), + ], + vec![ + Value::String("c".into()), + Value::String("a".into()), + Value::Float(1.0), + ], + ]; + assert_eq!( + run( + "CALL grust.algorithms.nodeSimilarity({orientation: 'undirected'}) YIELD node1, node2, similarity RETURN node1, node2, similarity" + ), + pairs + ); + assert_eq!( + run( + "CALL grust.algorithms.nodeSimilarity({orientation: 'undirected', metric: 'cosine', topN: 1}) YIELD node1, node2, similarity RETURN node1, node2, similarity" + ), + pairs[..1] + ); + // Directed, a and c still both point at b, and b shares nothing with them. + assert_eq!( + run( + "CALL grust.algorithms.nodeSimilarity() YIELD node1, node2, similarity RETURN node1, node2, similarity" + ), + pairs + ); + // A degree floor of two leaves no one to compare. + assert_eq!( + run( + "CALL grust.algorithms.nodeSimilarity({degreeCutoff: 2}) YIELD node1 RETURN count(node1)" + ), + vec![vec![Value::Int(0)]] + ); + assert!( + run_read_query_with_registry( + &graph(), + "default", + "CALL grust.algorithms.nodeSimilarity({metric: 'pearson'}) YIELD node1 RETURN node1", + &CypherParameters::new(), + ®istry(), + ) + .is_err() + ); +} + +#[test] +fn bridges_articulation_points_and_components_are_ordinary_procedures() { + // Undirected: a-b is edge 0; b-c and c-b are edges 1 and 2, a cycle of two. + let text = |value: &str| Value::String(value.into()); + assert_eq!( + run( + "CALL grust.algorithms.bridges({orientation: 'undirected'}) YIELD sourceNodeId, targetNodeId, edgeOrdinal RETURN sourceNodeId, targetNodeId, edgeOrdinal" + ), + vec![vec![text("a"), text("b"), Value::Int(0)]] + ); + assert_eq!( + run( + "CALL grust.algorithms.articulationPoints({orientation: 'undirected'}) YIELD nodeId RETURN nodeId" + ), + vec![vec![text("b")]] + ); + assert_eq!( + run( + "CALL grust.algorithms.biconnectedComponents({orientation: 'undirected'}) YIELD edgeOrdinal, componentId RETURN edgeOrdinal, componentId" + ), + vec![ + vec![Value::Int(0), Value::Int(0)], + vec![Value::Int(1), Value::Int(1)], + vec![Value::Int(2), Value::Int(1)], + ] + ); + assert!( + run_read_query_with_registry( + &graph(), + "default", + "CALL grust.algorithms.bridges() YIELD edgeOrdinal RETURN edgeOrdinal", + &CypherParameters::new(), + ®istry(), + ) + .unwrap_err() + .to_string() + .contains("undirected") + ); +} + +#[test] +fn spanning_tree_is_an_ordinary_procedure() { + // Undirected with costs: a-b 2.0, b-c 0.5, c-b 0.0. The lighter of the + // parallel pair is taken. + let query = |options: &str| { + run(&format!( + "CALL grust.algorithms.spanningTree({{orientation: 'undirected', weightProperty: 'cost'{options}}}) YIELD edgeOrdinal, weight, totalWeight RETURN edgeOrdinal, weight, totalWeight" + )) + }; + assert_eq!( + query(""), + vec![ + vec![Value::Int(0), Value::Float(2.0), Value::Float(2.0)], + vec![Value::Int(2), Value::Float(0.0), Value::Float(2.0)], + ] + ); + assert_eq!( + query(", objective: 'maximum'"), + vec![ + vec![Value::Int(0), Value::Float(2.0), Value::Float(2.5)], + vec![Value::Int(1), Value::Float(0.5), Value::Float(2.5)], + ] + ); + // The isolate's component has no edges. + assert!(query(", sourceNode: 'isolate'").is_empty()); + assert_eq!(query(", sourceNode: 'c'").len(), 2); +} + +#[test] +fn eigenvector_katz_and_hits_are_ordinary_procedures_with_convergence_evidence() { + // Directed a->b, b->c, c->b. Nothing points at a, so a = 1; b and c solve + // b = 1 + 0.25(a + c) and c = 1 + 0.25 b: 1.6 and 1.4. + let rows = run( + "CALL grust.algorithms.katz({alpha: 0.25, tolerance: 0.000000000001}) YIELD nodeId, score, converged RETURN nodeId, score, converged", + ); + let score = |row: usize| match rows[row][1] { + Value::Float(value) => value, + ref other => panic!("{other:?}"), + }; + assert!(rows.iter().all(|row| row[2] == Value::Bool(true))); + assert_eq!(score(0), 1.0); + assert!((score(1) - 1.6).abs() < 1e-9 && (score(2) - 1.4).abs() < 1e-9); + assert_eq!(score(3), 1.0); + + // HITS: b is pointed at by a and c, so it is the authority. + let rows = run( + "CALL grust.algorithms.hits() YIELD nodeId, hub, authority, converged WHERE authority > 0.9 RETURN nodeId, converged", + ); + assert_eq!( + rows, + vec![vec![Value::String("b".into()), Value::Bool(true)]] + ); + + // Eigenvector: a limit of one iteration is reported, not hidden. + let rows = run( + "CALL grust.algorithms.eigenvector({orientation: 'undirected', maxIterations: 1}) YIELD iterations, converged RETURN DISTINCT iterations, converged", + ); + assert_eq!(rows, vec![vec![Value::Int(1), Value::Bool(false)]]); +} + +#[test] +fn leiden_is_an_ordinary_procedure_with_louvains_shape() { + let rows = run( + "CALL grust.algorithms.leiden({orientation: 'undirected', seed: 7}) YIELD nodeId, communityId, modularity, converged RETURN nodeId, communityId, modularity, converged", + ); + assert_eq!(rows.len(), 4); + assert_eq!(rows[1][1], rows[2][1], "b and c share a community"); + assert_eq!(rows[3][1], Value::String("isolate".into())); + assert!(rows.iter().all(|row| row[3] == Value::Bool(true))); + // On this graph the two searches agree. + let plain = run( + "CALL grust.algorithms.louvain({orientation: 'undirected', seed: 7}) YIELD nodeId, communityId, modularity, converged RETURN nodeId, communityId, modularity, converged", + ); + assert_eq!(rows, plain); +} + +#[test] +fn max_flow_and_min_cut_take_a_source_and_a_target() { + // Directed with costs as capacities: a->b 2.0, b->c 0.5, c->b 0.0. + let text = |value: &str| Value::String(value.into()); + assert_eq!( + run( + "CALL grust.algorithms.maxFlow('a', 'c', {weightProperty: 'cost'}) YIELD sourceNodeId, targetNodeId, edgeOrdinal, flow, maxFlow RETURN sourceNodeId, targetNodeId, edgeOrdinal, flow, maxFlow" + ), + vec![ + vec![ + text("a"), + text("b"), + Value::Int(0), + Value::Float(0.5), + Value::Float(0.5) + ], + vec![ + text("b"), + text("c"), + Value::Int(1), + Value::Float(0.5), + Value::Float(0.5) + ], + ] + ); + // The bottleneck b->c is the cut: a and b stay with the source. + assert_eq!( + run( + "CALL grust.algorithms.minCut('a', 'c', {weightProperty: 'cost'}) YIELD nodeId, sourceSide WHERE sourceSide RETURN nodeId" + ), + vec![vec![text("a")], vec![text("b")]] + ); + // Nothing reaches the isolate: no rows, and that is an answer. + assert!(run("CALL grust.algorithms.maxFlow('a', 'isolate') YIELD flow RETURN flow").is_empty()); + assert!( + run_read_query_with_registry( + &graph(), + "default", + "CALL grust.algorithms.maxFlow('a', 'a') YIELD flow RETURN flow", + &CypherParameters::new(), + ®istry(), + ) + .is_err() + ); +} + +#[test] +fn fast_rp_returns_an_embedding_list_per_node() { + let rows = run( + "CALL grust.algorithms.fastRP({orientation: 'undirected', embeddingDimension: 16, seed: 4}) YIELD nodeId, embedding RETURN nodeId, embedding", + ); + assert_eq!(rows.len(), 4); + let vector = |row: usize| match &rows[row][1] { + Value::FloatArray(values) => values.clone(), + other => panic!("{other:?}"), + }; + assert!((0..4).all(|row| vector(row).len() == 16)); + // The isolate averages nothing, so with no self-influence it is zero. + assert!(vector(3).iter().all(|&value| value == 0.0)); + assert!(vector(1).iter().any(|&value| value != 0.0)); + // Seeded: the same call gives the same list; another seed does not. + let again = run( + "CALL grust.algorithms.fastRP({orientation: 'undirected', embeddingDimension: 16, seed: 4}) YIELD nodeId, embedding RETURN nodeId, embedding", + ); + assert_eq!(rows, again); + let other = run( + "CALL grust.algorithms.fastRP({orientation: 'undirected', embeddingDimension: 16, seed: 5, iterationWeights: [1.0]}) YIELD embedding RETURN embedding", + ); + assert_ne!(other[1][0], rows[1][1]); +} diff --git a/crates/grust-algorithm-procedures/tests/run_on_projection.rs b/crates/grust-algorithm-procedures/tests/run_on_projection.rs new file mode 100644 index 00000000..3af6599d --- /dev/null +++ b/crates/grust-algorithm-procedures/tests/run_on_projection.rs @@ -0,0 +1,166 @@ +//! A caller that already holds a projection runs any registered kernel by name +//! and gets typed Arrow back. Nutmeg is that caller: with this it needs no +//! per-algorithm code, so a kernel is served the day it is registered. + +use arrow_array::RecordBatch; +use grust_algorithm_procedures::{ + projection_kernel_names, projection_options, register_algorithms, run_on_projection, +}; +use grust_algorithms::{GraphProjection, Orientation, ProjectionEdge, WeightSelection}; +use grust_core::Value; +use grust_procedures::{ + ExecutionContext, ExecutionLimits, ProcedureError, ProcedureRegistry, RegistryBuilder, + SnapshotIdentity, ValueType, +}; + +fn registry() -> ProcedureRegistry { + let mut builder = RegistryBuilder::default(); + register_algorithms(&mut builder).unwrap(); + builder.build() +} + +fn context() -> ExecutionContext { + ExecutionContext::new(ExecutionLimits { + memory_bytes: 1024 * 1024, + work_units: 1_000_000, + batch_rows: 2, + deadline: None, + }) + .unwrap() +} + +/// a -> b -> c, plus an isolate: acyclic, so every kernel has a plain answer. +fn projection(context: &ExecutionContext, orientation: Orientation) -> GraphProjection { + let edge = |source, target, ordinal| ProjectionEdge { + source, + target, + ordinal, + id: None, + }; + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r1".into(), "reader".into()).unwrap(), + vec!["a".into(), "b".into(), "c".into(), "isolate".into()], + vec![edge(0, 1, 0), edge(1, 2, 1)], + Some(vec![2.0, 0.5]), + orientation, + context, + ) + .unwrap() +} + +fn drain(mut cursor: grust_algorithms::ArrowResultCursor) -> Vec { + let mut batches = Vec::new(); + while let Some(batch) = cursor.next_batch().unwrap() { + batches.push(batch.record_batch().clone()); + } + batches +} + +#[test] +fn every_registered_projection_kernel_runs_by_name_with_its_declared_columns() { + let registry = registry(); + let names = projection_kernel_names(); + assert!(names.contains(&"pagerank") && names.contains(&"shortestPaths")); + + for name in names { + let resolved = registry + .resolve(&format!("grust.algorithms.{name}")) + .unwrap_or_else(|error| panic!("{name} is in the catalog but not registered: {error}")); + let definition = resolved.definition(); + // The configuration map is always last; a source argument, if any, precedes it. + let mut args = Vec::new(); + // Successive node arguments name different nodes: a flow needs two ends. + let mut ids = ["a", "b", "c"].into_iter(); + for argument in &definition.arguments { + args.push(match argument.field.value_type { + ValueType::String => Value::from(ids.next().unwrap()), + ValueType::Strings => Value::StringArray(vec!["a".into()]), + ValueType::Map => Value::Json(serde_json::json!({})), + ref other => panic!("{name}: unexpected argument type {other:?}"), + }); + } + let args = resolved.validate_arguments(args).unwrap(); + + // Kernels defined on undirected graphs refuse a directed projection; + // that refusal is theirs to state, and they then run on an undirected one. + let context = context(); + let mut graph = projection(&context, Orientation::Outgoing); + if let Err(ProcedureError::InvalidArguments(message)) = + run_on_projection(name, &graph, &args) + { + assert!(message.contains("undirected"), "{name}: {message}"); + graph = projection(&context, Orientation::Undirected); + } + let batches = drain(run_on_projection(name, &graph, &args).unwrap()); + assert!(!batches.is_empty(), "{name} produced no batches"); + + let declared: Vec<&str> = definition.outputs.iter().map(|f| f.name.as_str()).collect(); + let schema = batches[0].schema(); + let columns: Vec<&str> = schema.fields().iter().map(|f| f.name().as_str()).collect(); + assert_eq!( + columns, declared, + "{name}: Arrow columns differ from the registry" + ); + assert!( + batches.iter().all(|batch| batch.num_rows() <= 2), + "{name} ignored the batch row bound" + ); + + // The prefixed name is the same kernel. + let prefixed = + drain(run_on_projection(&format!("grust.algorithms.{name}"), &graph, &args).unwrap()); + assert_eq!(prefixed, batches, "{name}"); + // Case is not significant, as in the registry. + let lowered = drain(run_on_projection(&name.to_ascii_lowercase(), &graph, &args).unwrap()); + assert_eq!(lowered, batches, "{name}"); + } +} + +#[test] +fn metadata_procedures_and_unknown_names_are_refused() { + let registry = registry(); + let context = context(); + let graph = projection(&context, Orientation::Outgoing); + let args = registry + .resolve("grust.algorithms.wcc") + .unwrap() + .validate_arguments(vec![Value::Json(serde_json::json!({}))]) + .unwrap(); + for name in ["projectionStats", "estimateCsr", "louvain-not-yet"] { + assert!( + matches!( + run_on_projection(name, &graph, &args), + Err(ProcedureError::Unsupported(_)) + ), + "{name}" + ); + } +} + +#[test] +fn projection_options_read_what_the_procedure_would() { + let registry = registry(); + let resolved = registry.resolve("grust.algorithms.degree").unwrap(); + let args = resolved + .validate_arguments(vec![Value::Json(serde_json::json!({ + "orientation": "undirected", + "weightProperty": "cost", + "defaultWeight": 1.5, + "nodeLabels": ["Person"], + }))]) + .unwrap(); + let options = projection_options(&args).unwrap(); + assert_eq!(options.orientation, Orientation::Undirected); + assert_eq!(options.node_labels, Some(&["Person".to_string()][..])); + assert!(matches!( + options.weight, + WeightSelection::Property { key: "cost", .. } + )); + + let defaults = resolved + .validate_arguments(vec![Value::Json(serde_json::json!({}))]) + .unwrap(); + let options = projection_options(&defaults).unwrap(); + assert_eq!(options.orientation, Orientation::Outgoing); + assert!(matches!(options.weight, WeightSelection::Unit)); +} diff --git a/crates/grust-algorithms/Cargo.toml b/crates/grust-algorithms/Cargo.toml index e260f7d7..f0875d64 100644 --- a/crates/grust-algorithms/Cargo.toml +++ b/crates/grust-algorithms/Cargo.toml @@ -10,19 +10,23 @@ keywords.workspace = true categories.workspace = true [features] -default = [] +default = ["parallel"] arrow = ["dep:arrow-array", "dep:grust-arrow"] +# Run kernels on the caller's rayon pool. Results are identical at any width. +parallel = ["dep:rayon"] [dependencies] grust-arrow = { version = "0.21.0", path = "../grust-arrow", optional = true } grust-core = { version = "0.21.0", path = "../grust-core" } grust-procedures = { version = "0.21.0", path = "../grust-procedures" } arrow-array = { version = "59.3.0", optional = true } +rayon = { version = "1.11", optional = true } [lints.rust] unsafe_code = "forbid" [dev-dependencies] +rayon = "1.11" criterion.workspace = true [[bench]] diff --git a/crates/grust-algorithms/src/arrow_output.rs b/crates/grust-algorithms/src/arrow_output.rs index c597e80f..a89ec385 100644 --- a/crates/grust-algorithms/src/arrow_output.rs +++ b/crates/grust-algorithms/src/arrow_output.rs @@ -4,13 +4,16 @@ use std::sync::Arc; use arrow_array::{ ArrayRef, RecordBatch, - builder::{BooleanBuilder, Float64Builder, LargeListBuilder, StringBuilder, UInt64Builder}, + builder::{ + BooleanBuilder, FixedSizeListBuilder, Float32Builder, Float64Builder, Int64Builder, + LargeListBuilder, StringBuilder, UInt64Builder, + }, }; use grust_procedures::MemoryReservation; use crate::{ - AlgorithmError, Components, Degrees, Distances, GraphProjection, NodeOrder, PageRank, - PathCursor, PathView, Result, ShortestPaths, TopologicalOrder, + AlgorithmError, Components, Degrees, Distances, GraphProjection, NodeOrder, NodeTable, + PageRank, PathCursor, PathView, Result, ShortestPaths, TableType, TableValue, TopologicalOrder, }; /// A bounded Arrow batch retaining its query memory reservation. Clone this @@ -44,6 +47,7 @@ enum Output { PageRank(PageRank), Degrees(Degrees), Paths(PathCursor), + Table(NodeTable), } /// Pull Arrow results in projection row order. Scalar batches honor the query's @@ -52,6 +56,7 @@ pub struct ArrowResultCursor { graph: Option, output: Option, next: usize, + emitted: bool, failed: bool, } @@ -89,6 +94,14 @@ impl PageRank { ArrowResultCursor::new(self.projection().clone(), Output::PageRank(self)) } } +impl NodeTable { + /// Convert without copying the result buffers; each batch is admitted. + /// Columns are `nodeId`, then the table's columns in declared order. + pub fn into_arrow_results(self) -> ArrowResultCursor { + ArrowResultCursor::new(self.projection().clone(), Output::Table(self)) + } +} + impl Degrees { /// Transfer exact UInt64 `degree` and nullable Float64 `strength` columns. pub fn into_arrow_results(self) -> ArrowResultCursor { @@ -113,6 +126,7 @@ impl ArrowResultCursor { graph: Some(graph), output: Some(output), next: 0, + emitted: false, failed: false, } } @@ -154,9 +168,13 @@ impl ArrowResultCursor { } let total = match output { Output::Order(result) => result.values().len(), + Output::Table(table) => table.rows(), _ => graph.node_count(), }; - if self.next == total { + // A table that found nothing still says what its columns are: one empty + // batch, so a consumer reading the schema off the first batch has one. + let empty_table = total == 0 && !self.emitted && matches!(output, Output::Table(_)); + if self.next == total && !empty_table { return Ok(None); } let start = self.next; @@ -168,6 +186,7 @@ impl ArrowResultCursor { context.charge_work(1)?; let node = match output { Output::Order(result) => result.values()[index], + Output::Table(table) => table.key(index), _ => index, }; id_bytes = id_bytes.saturating_add(graph.node_ids()[node].as_str().len()); @@ -175,11 +194,30 @@ impl ArrowResultCursor { component_bytes = component_bytes .saturating_add(graph.node_ids()[result.values()[index]].as_str().len()); } + if let Output::Table(table) = output { + for column in 0..table.width() { + if let TableValue::Node(id) = table.value(column, index) { + component_bytes = component_bytes.saturating_add(id.as_str().len()); + } + } + } } + // The fixed overhead covers six arrays; a wider table admits its own. + let width_bytes = match output { + Output::Table(table) => (0..table.width()) + .map(|column| match table.field(column).1 { + TableType::Vector(dimension) => 16 + 4 * dimension, + _ => 16, + }) + .fold(0usize, usize::saturating_add) + .saturating_mul(count), + _ => 0, + }; let reservation = context.reserve( overhead(count) .saturating_add(id_bytes) - .saturating_add(component_bytes), + .saturating_add(component_bytes) + .saturating_add(width_bytes), )?; utf8_size(id_bytes)?; utf8_size(component_bytes)?; @@ -188,11 +226,16 @@ impl ArrowResultCursor { context.charge_work(1)?; let node = match output { Output::Order(result) => result.values()[index], + Output::Table(table) => table.key(index), _ => index, }; ids.append_value(graph.node_ids()[node].as_str()); } - let mut columns = vec![("nodeId", Arc::new(ids.finish()) as ArrayRef)]; + let key_name = match output { + Output::Table(table) => table.key_name(), + _ => "nodeId", + }; + let mut columns = vec![(key_name, Arc::new(ids.finish()) as ArrayRef)]; match output { Output::Order(_) => { let mut values = UInt64Builder::with_capacity(count); @@ -251,6 +294,80 @@ impl ArrowResultCursor { ("residual", Arc::new(residual.finish())), ]); } + Output::Table(table) => { + for column in 0..table.width() { + context.charge_work(count)?; + let (name, kind) = table.field(column); + let array: ArrayRef = match kind { + TableType::Integer => { + let mut values = Int64Builder::with_capacity(count); + for row in start..end { + match table.value(column, row) { + TableValue::Integer(value) => values.append_value(value), + _ => values.append_null(), + } + } + Arc::new(values.finish()) + } + TableType::Number => { + let mut values = Float64Builder::with_capacity(count); + for row in start..end { + match table.value(column, row) { + TableValue::Number(value) => values.append_value(value), + _ => values.append_null(), + } + } + Arc::new(values.finish()) + } + TableType::Boolean => { + let mut values = BooleanBuilder::with_capacity(count); + for row in start..end { + match table.value(column, row) { + TableValue::Boolean(value) => values.append_value(value), + _ => values.append_null(), + } + } + Arc::new(values.finish()) + } + TableType::Vector(dimension) => { + let width = i32::try_from(dimension).map_err(|_| { + AlgorithmError::OutputContract( + "vector dimension exceeds Arrow's list size".into(), + ) + })?; + let mut values = FixedSizeListBuilder::with_capacity( + Float32Builder::with_capacity(count * dimension), + width, + count, + ); + for row in start..end { + match table.value(column, row) { + TableValue::Vector(vector) => { + values.values().append_slice(vector); + values.append(true); + } + _ => { + values.values().append_nulls(dimension); + values.append(false); + } + } + } + Arc::new(values.finish()) + } + TableType::Node => { + let mut values = StringBuilder::with_capacity(count, component_bytes); + for row in start..end { + match table.value(column, row) { + TableValue::Node(id) => values.append_value(id.as_str()), + _ => values.append_null(), + } + } + Arc::new(values.finish()) + } + }; + columns.push((name, array)); + } + } Output::Paths(_) | Output::Topology(_) => { return Err(AlgorithmError::OutputContract( "path output reached scalar Arrow adapter".into(), @@ -258,6 +375,7 @@ impl ArrowResultCursor { } } self.next = end; + self.emitted = true; finish(columns, reservation).map(Some) } } diff --git a/crates/grust-algorithms/src/betweenness.rs b/crates/grust-algorithms/src/betweenness.rs new file mode 100644 index 00000000..58704157 --- /dev/null +++ b/crates/grust-algorithms/src/betweenness.rs @@ -0,0 +1,260 @@ +//! Betweenness centrality by Brandes' algorithm, exact or from sampled sources. + +use crate::{ + AlgorithmError, GraphProjection, Orientation, Result, + buffer::Buffer, + meter::Meter, + parallel, random, + shortest::MinHeap, + table::{NodeColumn, NodeTable}, +}; +use grust_procedures::ExecutionContext; + +/// Betweenness controls. +#[derive(Clone, Copy, Debug, Default)] +pub struct BetweennessOptions { + /// Use this many source nodes, drawn without replacement, and scale the sum + /// by `n / samplingSize` so it estimates the exact score. `None`, or a size + /// of at least `n`, is exact. + pub sampling_size: Option, + /// Seed for the draw. The same seed draws the same sources. + pub seed: u64, + /// Divide by the number of ordered pairs a node could lie between: + /// `(n-1)(n-2)`, or half that on an undirected projection. + pub normalized: bool, +} + +/// A betweenness score per node. +pub struct Betweenness { + graph: GraphProjection, + scores: Buffer, +} + +impl Betweenness { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + /// Score per node, in projection row order. + pub fn values(&self) -> &[f64] { + &self.scores.values + } + /// The result as a node table with a `score` column. + pub fn into_table(self) -> Result { + NodeTable::new(&self.graph).column("score", NodeColumn::Number(self.scores)) + } +} + +/// Sources per block. Fixed, so which sums are formed, and in what order, does +/// not depend on the pool width. +const BLOCK: usize = 64; + +/// For every node `v`, the sum over ordered pairs `s != v != t` of the share of +/// shortest `s`-`t` paths that pass through `v`. +/// +/// **Orientation.** Paths follow the projection's arcs. On an undirected +/// projection each unordered pair is reached from both ends, so the sum is +/// halved, as Brandes specifies. +/// +/// **Multigraphs.** Parallel edges are distinct shortest paths and each counts. +/// Self-loops lie on no shortest path. +/// +/// **Weights.** An unweighted projection uses hop counts. A weighted one uses +/// Dijkstra and treats two path costs as tied when they are equal as `f64`, +/// which is exact for integral and dyadic weights and only approximate +/// otherwise. A zero weight makes the number of shortest paths ill-defined and +/// is rejected. +/// +/// Sources run in blocks of 64 on the caller's rayon pool and are merged in +/// block order; scores, and the work charged, are identical at any pool width. +/// Memory held at once does grow with the pool: one workspace per running block. +pub fn betweenness(graph: &GraphProjection, options: BetweennessOptions) -> Result { + let context = graph.execution(); + context.checkpoint()?; + let n = graph.node_count(); + let adjacency = graph.outgoing(); + if options.sampling_size == Some(0) { + return Err(AlgorithmError::InvalidArguments( + "samplingSize must be positive".into(), + )); + } + let weighted = adjacency.weights.is_some(); + if let Some(weights) = &adjacency.weights { + let mut meter = Meter::new(context); + for &weight in &weights.values { + meter.tick(1)?; + if weight == 0.0 { + return Err(AlgorithmError::InvalidArguments( + "betweenness needs positive weights: a zero weight makes the number of shortest paths ill-defined".into(), + )); + } + } + meter.flush()?; + } + + // The sources, ascending, so an exhaustive sample is the exact run bit for bit. + let mut sources = Buffer::capacity(n, context)?; + sources.values.extend(0..n); + let sampled = options.sampling_size.is_some_and(|size| size < n); + if let (true, Some(size)) = (sampled, options.sampling_size) { + for index in 0..size { + let pick = index + random::below(options.seed, 0, index as u64, n - index); + sources.values.swap(index, pick); + } + sources.values.truncate(size); + sources.values.sort_unstable(); + } + let source_count = sources.values.len(); + + let mut scores = Buffer::filled(n, 0.0f64, context)?; + parallel::ordered_blocks( + source_count.div_ceil(BLOCK), + |block| { + let first = block * BLOCK; + let last = (first + BLOCK).min(source_count); + let mut workspace = Workspace::new(n, weighted, context)?; + for &source in &sources.values[first..last] { + workspace.accumulate(graph, source, context)?; + } + Ok(workspace.local) + }, + |_, local: Buffer| { + context.charge_work(n)?; + for (score, part) in scores.values.iter_mut().zip(&local.values) { + *score += part; + } + Ok(()) + }, + )?; + + let mut factor = 1.0; + if graph.orientation() == Orientation::Undirected { + factor *= 0.5; + } + if sampled { + factor *= n as f64 / source_count as f64; + } + if options.normalized { + let pairs = (n as f64 - 1.0) * (n as f64 - 2.0); + let pairs = if graph.orientation() == Orientation::Undirected { + pairs / 2.0 + } else { + pairs + }; + // Undirected sums were already halved above; the pair count is halved too. + factor = if pairs > 0.0 { factor / pairs } else { 0.0 }; + } + for score in &mut scores.values { + *score *= factor; + } + Ok(Betweenness { + graph: graph.clone(), + scores, + }) +} + +/// Per-block scratch: one single-source pass at a time, summed into `local`. +struct Workspace { + distance: Buffer, + paths: Buffer, + dependency: Buffer, + order: Buffer, + heap: Option, + local: Buffer, +} + +impl Workspace { + fn new(n: usize, weighted: bool, context: &ExecutionContext) -> Result { + Ok(Self { + distance: Buffer::filled(n, f64::INFINITY, context)?, + paths: Buffer::filled(n, 0.0, context)?, + dependency: Buffer::filled(n, 0.0, context)?, + order: Buffer::capacity(n, context)?, + heap: weighted.then(|| MinHeap::new(n, context)).transpose()?, + local: Buffer::filled(n, 0.0, context)?, + }) + } + + fn accumulate( + &mut self, + graph: &GraphProjection, + source: usize, + context: &ExecutionContext, + ) -> Result<()> { + let adjacency = graph.outgoing(); + let mut meter = Meter::new(context); + self.order.values.clear(); + self.distance.values[source] = 0.0; + self.paths.values[source] = 1.0; + + // Forward: settle nodes in nondecreasing distance, counting shortest paths. + if let Some(heap) = &mut self.heap { + heap.improve(source, 0.0, context)?; + while let Some((cost, node)) = heap.pop(context)? { + self.order.values.push(node); + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + for arc in range { + let next = adjacency.targets.values[arc]; + let candidate = cost + adjacency.weight(arc); + if candidate < self.distance.values[next] { + self.distance.values[next] = candidate; + self.paths.values[next] = self.paths.values[node]; + heap.improve(next, candidate, context)?; + } else if candidate == self.distance.values[next] { + self.paths.values[next] += self.paths.values[node]; + } + } + } + } else { + self.order.values.push(source); + let mut head = 0; + while head < self.order.values.len() { + let node = self.order.values[head]; + head += 1; + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + let candidate = self.distance.values[node] + 1.0; + for arc in range { + let next = adjacency.targets.values[arc]; + if self.distance.values[next].is_infinite() { + self.distance.values[next] = candidate; + self.order.values.push(next); + } + if self.distance.values[next] == candidate { + self.paths.values[next] += self.paths.values[node]; + } + } + } + } + + // Backward: a node inherits from each successor on a shortest path the + // share of that successor's paths that came through it. + for &node in self.order.values.iter().rev() { + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + for arc in range { + let next = adjacency.targets.values[arc]; + if next != node + && self.distance.values[next] + == self.distance.values[node] + adjacency.weight(arc) + { + self.dependency.values[node] += self.paths.values[node] + / self.paths.values[next] + * (1.0 + self.dependency.values[next]); + } + } + if node != source { + self.local.values[node] += self.dependency.values[node]; + } + } + + // Reset only what this source touched. + for &node in &self.order.values { + self.distance.values[node] = f64::INFINITY; + self.paths.values[node] = 0.0; + self.dependency.values[node] = 0.0; + } + meter.flush() + } +} diff --git a/crates/grust-algorithms/src/biconnected.rs b/crates/grust-algorithms/src/biconnected.rs new file mode 100644 index 00000000..ee91e5f6 --- /dev/null +++ b/crates/grust-algorithms/src/biconnected.rs @@ -0,0 +1,230 @@ +//! Bridges, articulation points and biconnected components from one +//! depth-first pass. + +use crate::{ + AlgorithmError, GraphProjection, Orientation, Result, + buffer::Buffer, + meter::Meter, + table::{NodeColumn, NodeTable}, +}; + +const UNSEEN: usize = usize::MAX; + +/// What one low-link pass learns about an undirected multigraph. +pub struct Biconnectivity { + graph: GraphProjection, + /// Per edge slot: the smallest edge ordinal in its component; `None` for a + /// self-loop, which lies in no component. + components: Buffer>, + bridges: Buffer, + articulation: Buffer, +} + +impl Biconnectivity { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + /// Bridges, as ascending indices into [`GraphProjection::edges`]. + pub fn bridges(&self) -> &[usize] { + &self.bridges.values + } + /// Articulation points, as ascending projection rows. + pub fn articulation_points(&self) -> &[usize] { + &self.articulation.values + } + /// Per entry of [`GraphProjection::edges`], its component, named by the + /// smallest edge ordinal in it. A self-loop has none. + pub fn components(&self) -> &[Option] { + &self.components.values + } + + /// Rows of `sourceNodeId`, `targetNodeId`, `edgeOrdinal`, one per bridge. + pub fn into_bridge_table(self) -> Result { + let slots = self.bridges; + edge_table(&self.graph, &slots.values, None) + } + + /// Rows of `nodeId`, one per articulation point. + pub fn into_articulation_table(self) -> Result { + NodeTable::keyed(&self.graph, "nodeId", self.articulation) + } + + /// Rows of `sourceNodeId`, `targetNodeId`, `edgeOrdinal`, `componentId`, one + /// per edge that is not a self-loop. + pub fn into_component_table(self) -> Result { + let context = self.graph.execution(); + let mut slots = Buffer::capacity(self.components.values.len(), context)?; + slots.values.extend( + (0..self.components.values.len()) + .filter(|&slot| self.components.values[slot].is_some()), + ); + edge_table(&self.graph, &slots.values, Some(&self.components.values)) + } +} + +fn edge_table( + graph: &GraphProjection, + slots: &[usize], + components: Option<&[Option]>, +) -> Result { + let context = graph.execution(); + context.charge_work(slots.len())?; + let integer = |value: usize| { + i64::try_from(value).map_err(|_| AlgorithmError::Numerical("ordinal exceeds Int64".into())) + }; + let mut sources = Buffer::capacity(slots.len(), context)?; + let mut targets = Buffer::capacity(slots.len(), context)?; + let mut ordinals = Buffer::capacity(slots.len(), context)?; + for &slot in slots { + let edge = &graph.edges()[slot]; + sources.values.push(edge.source); + targets.values.push(edge.target); + ordinals.values.push(integer(edge.ordinal)?); + } + let mut table = NodeTable::keyed(graph, "sourceNodeId", sources)? + .column("targetNodeId", NodeColumn::Node(targets))? + .column("edgeOrdinal", NodeColumn::Integer(ordinals))?; + if let Some(components) = components { + let mut ids = Buffer::capacity(slots.len(), context)?; + for &slot in slots { + ids.values + .push(integer(components[slot].unwrap_or_default())?); + } + table = table.column("componentId", NodeColumn::Integer(ids))?; + } + Ok(table) +} + +/// Find every bridge, articulation point and biconnected component. +/// +/// A **bridge** is an edge whose removal disconnects its endpoints. An +/// **articulation point** is a node whose removal disconnects two of its +/// neighbours. A **biconnected component** is a maximal set of edges in which +/// any two lie on a common simple cycle; a bridge is a component of one edge. +/// +/// **Multigraphs.** Parallel edges form a cycle of length two, so neither is a +/// bridge and both share a component. The pass tracks the edge it arrived by, +/// not the node it arrived from, which is what makes that come out right. +/// Self-loops disconnect nothing and lie in no component. +/// +/// The pass is iterative, so a path of a million nodes does not overflow a +/// stack, and sequential: low-link order is the algorithm. Roots are taken in +/// row order and arcs in adjacency order, so the result is a function of the +/// projection alone. Defined on undirected graphs; project with +/// `orientation: "undirected"`. +pub fn biconnectivity(graph: &GraphProjection) -> Result { + let context = graph.execution(); + context.checkpoint()?; + if graph.orientation() != Orientation::Undirected { + return Err(AlgorithmError::InvalidArguments( + "bridges, articulation points and biconnected components are defined on undirected graphs; project with orientation \"undirected\"".into(), + )); + } + let n = graph.node_count(); + let edges = graph.edges(); + let adjacency = graph.outgoing(); + let mut meter = Meter::new(context); + + let mut discovered = Buffer::filled(n, UNSEEN, context)?; + let mut low = Buffer::filled(n, UNSEEN, context)?; + let mut is_articulation = Buffer::filled(n, false, context)?; + let mut is_bridge = Buffer::filled(edges.len(), false, context)?; + let mut components = Buffer::filled(edges.len(), None, context)?; + // (node, next arc to look at, edge slot it was reached by). + let mut path: Buffer<(usize, usize, usize)> = Buffer::capacity(n, context)?; + // Edges seen since the component in progress began. Each enters once. + let mut pending: Buffer = Buffer::capacity(edges.len(), context)?; + let mut clock = 0usize; + + for root in 0..n { + meter.tick(1)?; + if discovered.values[root] != UNSEEN { + continue; + } + discovered.values[root] = clock; + low.values[root] = clock; + clock += 1; + let mut root_children = 0usize; + path.values + .push((root, adjacency.range(root).start, UNSEEN)); + while let Some(&mut (node, ref mut next, arrived_by)) = path.values.last_mut() { + if *next < adjacency.range(node).end { + let arc = *next; + *next += 1; + meter.tick(1)?; + let slot = adjacency.edge_slots.values[arc]; + let other = adjacency.targets.values[arc]; + if slot == arrived_by || other == node { + continue; + } + if discovered.values[other] == UNSEEN { + discovered.values[other] = clock; + low.values[other] = clock; + clock += 1; + pending.values.push(slot); + path.values + .push((other, adjacency.range(other).start, slot)); + } else if discovered.values[other] < discovered.values[node] { + // Back to an ancestor. The other direction of the same edge, + // met later from the ancestor's side, fails this test. + pending.values.push(slot); + low.values[node] = low.values[node].min(discovered.values[other]); + } + continue; + } + path.values.pop(); + let Some(&(parent, _, _)) = path.values.last() else { + break; + }; + low.values[parent] = low.values[parent].min(low.values[node]); + if parent == root { + root_children += 1; + } + if low.values[node] >= discovered.values[parent] { + // Nothing below `node` reaches above `parent`: a component ends. + if parent != root { + is_articulation.values[parent] = true; + } + is_bridge.values[arrived_by] = low.values[node] > discovered.values[parent]; + let start = pending + .values + .iter() + .rposition(|&slot| slot == arrived_by) + .ok_or_else(|| { + AlgorithmError::OutputContract( + "tree edge missing from its component".into(), + ) + })?; + meter.tick(pending.values.len() - start)?; + let name = pending.values[start..] + .iter() + .map(|&slot| edges[slot].ordinal) + .min(); + for &slot in &pending.values[start..] { + components.values[slot] = name; + } + pending.values.truncate(start); + } + } + is_articulation.values[root] = root_children > 1; + } + + meter.tick(n + edges.len())?; + let count = |flags: &[bool]| flags.iter().filter(|&&flag| flag).count(); + let mut bridges = Buffer::capacity(count(&is_bridge.values), context)?; + bridges + .values + .extend((0..edges.len()).filter(|&slot| is_bridge.values[slot])); + let mut articulation = Buffer::capacity(count(&is_articulation.values), context)?; + articulation + .values + .extend((0..n).filter(|&node| is_articulation.values[node])); + meter.flush()?; + Ok(Biconnectivity { + graph: graph.clone(), + components, + bridges, + articulation, + }) +} diff --git a/crates/grust-algorithms/src/buffer.rs b/crates/grust-algorithms/src/buffer.rs index 59015325..a46a719a 100644 --- a/crates/grust-algorithms/src/buffer.rs +++ b/crates/grust-algorithms/src/buffer.rs @@ -55,6 +55,23 @@ impl Buffer { } } +impl Buffer { + /// `filled` for values that are made, not cloned (atomics). + pub(crate) fn filled_with( + count: usize, + make: impl Fn() -> T, + context: &ExecutionContext, + ) -> Result { + let mut buffer = Self::capacity(count, context)?; + for offset in (0..count).step_by(1024) { + let end = offset + (count - offset).min(1024); + context.charge_work(end - offset)?; + buffer.values.resize_with(end, &make); + } + Ok(buffer) + } +} + impl std::ops::Deref for Buffer { type Target = [T]; fn deref(&self) -> &Self::Target { diff --git a/crates/grust-algorithms/src/closeness.rs b/crates/grust-algorithms/src/closeness.rs new file mode 100644 index 00000000..f6b659ff --- /dev/null +++ b/crates/grust-algorithms/src/closeness.rs @@ -0,0 +1,228 @@ +//! Closeness and harmonic centrality: one shortest-path sweep per node. + +use crate::{ + AlgorithmError, GraphProjection, Result, + buffer::Buffer, + meter::Meter, + parallel, + shortest::MinHeap, + table::{NodeColumn, NodeTable}, +}; +use grust_procedures::ExecutionContext; + +/// Closeness controls. +#[derive(Clone, Copy, Debug, Default)] +pub struct ClosenessOptions { + /// Scale each score by the share of other nodes reached, `r / (n-1)`, as + /// Wasserman and Faust do, so a node central only to a small component does + /// not rank with one central to a large component. + pub wasserman_faust: bool, +} + +/// Harmonic controls. +#[derive(Clone, Copy, Debug)] +pub struct HarmonicOptions { + /// Divide by `n-1`, the most any node can score on an unweighted graph. + pub normalized: bool, +} + +impl Default for HarmonicOptions { + fn default() -> Self { + Self { normalized: true } + } +} + +/// A distance-based centrality score per node. +pub struct DistanceCentrality { + graph: GraphProjection, + scores: Buffer, +} + +impl DistanceCentrality { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + /// Score per node, in projection row order. + pub fn values(&self) -> &[f64] { + &self.scores.values + } + /// The result as a node table with a `score` column. + pub fn into_table(self) -> Result { + NodeTable::new(&self.graph).column("score", NodeColumn::Number(self.scores)) + } +} + +/// Closeness of `v`: `r / Σ d(v, u)` over the `r` other nodes `v` reaches, and +/// zero when it reaches none. +/// +/// **Disconnected graphs** are the whole question. This is the per-component +/// form: the mean distance to what is reachable, inverted. It never divides by +/// an infinite distance, but it lets a node in a two-node component score 1. +/// `wasserman_faust` multiplies by `r / (n-1)` to correct that. +/// +/// **Direction.** Distances run *from* the node along the projection's arcs. +/// Project with the opposite orientation to measure distances *to* the node. +/// +/// **Weights** are distances when the projection has them; a zero weight is +/// rejected, because a zero distance between distinct nodes has no reciprocal. +/// Parallel edges and self-loops change no distance. +/// +/// Scores and charged work are identical at any pool width; memory held at once +/// is one workspace per running block of 64 sources. +pub fn closeness(graph: &GraphProjection, options: ClosenessOptions) -> Result { + let others = graph.node_count().saturating_sub(1) as f64; + sweep(graph, |reached, distance_sum, _| { + if reached == 0 { + return 0.0; + } + let reached = reached as f64; + let score = reached / distance_sum; + if options.wasserman_faust { + score * reached / others + } else { + score + } + }) +} + +/// Harmonic centrality of `v`: `Σ 1 / d(v, u)` over the other nodes `v` reaches. +/// An unreachable node contributes zero, so no convention for disconnected +/// graphs is needed. Direction, weights and determinism are as for +/// [`closeness`]. +pub fn harmonic(graph: &GraphProjection, options: HarmonicOptions) -> Result { + let others = graph.node_count().saturating_sub(1) as f64; + sweep(graph, |_, _, reciprocal_sum| { + if options.normalized && others > 0.0 { + reciprocal_sum / others + } else { + reciprocal_sum + } + }) +} + +/// Sources per block; fixed, so nothing depends on the pool width. +const BLOCK: usize = 64; + +/// Run one sweep per node and score it from (nodes reached, Σ d, Σ 1/d). +fn sweep( + graph: &GraphProjection, + score: impl Fn(usize, f64, f64) -> f64 + Sync, +) -> Result { + let context = graph.execution(); + context.checkpoint()?; + let n = graph.node_count(); + let adjacency = graph.outgoing(); + let weighted = adjacency.weights.is_some(); + if let Some(weights) = &adjacency.weights { + let mut meter = Meter::new(context); + for &weight in &weights.values { + meter.tick(1)?; + if weight == 0.0 { + return Err(AlgorithmError::InvalidArguments( + "distance centralities need positive weights: a zero distance between distinct nodes has no reciprocal".into(), + )); + } + } + meter.flush()?; + } + let mut scores = Buffer::filled(n, 0.0f64, context)?; + parallel::ordered_blocks( + n.div_ceil(BLOCK), + |block| { + let first = block * BLOCK; + let last = (first + BLOCK).min(n); + let mut workspace = Workspace::new(n, weighted, context)?; + let mut part = Buffer::capacity(last - first, context)?; + for source in first..last { + let (reached, distance_sum, reciprocal_sum) = + workspace.run(graph, source, context)?; + part.values + .push(score(reached, distance_sum, reciprocal_sum)); + } + Ok(part) + }, + |block, part: Buffer| { + let first = block * BLOCK; + scores.values[first..first + part.values.len()].copy_from_slice(&part.values); + Ok(()) + }, + )?; + Ok(DistanceCentrality { + graph: graph.clone(), + scores, + }) +} + +struct Workspace { + distance: Buffer, + order: Buffer, + heap: Option, +} + +impl Workspace { + fn new(n: usize, weighted: bool, context: &ExecutionContext) -> Result { + Ok(Self { + distance: Buffer::filled(n, f64::INFINITY, context)?, + order: Buffer::capacity(n, context)?, + heap: weighted.then(|| MinHeap::new(n, context)).transpose()?, + }) + } + + /// Settle everything reachable from `source`, summing in settle order, which + /// the queue and the heap's tie-break make a function of the graph alone. + fn run( + &mut self, + graph: &GraphProjection, + source: usize, + context: &ExecutionContext, + ) -> Result<(usize, f64, f64)> { + let adjacency = graph.outgoing(); + let mut meter = Meter::new(context); + self.order.values.clear(); + self.distance.values[source] = 0.0; + if let Some(heap) = &mut self.heap { + heap.improve(source, 0.0, context)?; + while let Some((cost, node)) = heap.pop(context)? { + self.order.values.push(node); + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + for arc in range { + let next = adjacency.targets.values[arc]; + let candidate = cost + adjacency.weight(arc); + if candidate < self.distance.values[next] { + self.distance.values[next] = candidate; + heap.improve(next, candidate, context)?; + } + } + } + } else { + self.order.values.push(source); + let mut head = 0; + while head < self.order.values.len() { + let node = self.order.values[head]; + head += 1; + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + for arc in range { + let next = adjacency.targets.values[arc]; + if self.distance.values[next].is_infinite() { + self.distance.values[next] = self.distance.values[node] + 1.0; + self.order.values.push(next); + } + } + } + } + let (mut distance_sum, mut reciprocal_sum) = (0.0, 0.0); + for &node in &self.order.values[1..] { + distance_sum += self.distance.values[node]; + reciprocal_sum += 1.0 / self.distance.values[node]; + } + let reached = self.order.values.len() - 1; + for &node in &self.order.values { + self.distance.values[node] = f64::INFINITY; + } + meter.flush()?; + Ok((reached, distance_sum, reciprocal_sum)) + } +} diff --git a/crates/grust-algorithms/src/fastrp.rs b/crates/grust-algorithms/src/fastrp.rs new file mode 100644 index 00000000..61c9fdb9 --- /dev/null +++ b/crates/grust-algorithms/src/fastrp.rs @@ -0,0 +1,234 @@ +//! FastRP node embeddings: very sparse random projection, averaged over +//! neighbourhoods of growing radius (Chen, Sultan, Tian, Chen and Skiena, 2019). + +use crate::{ + AlgorithmError, GraphProjection, Result, + buffer::Buffer, + meter::Meter, + parallel, random, + table::{NodeColumn, NodeTable}, +}; + +/// FastRP controls. +#[derive(Clone, Copy, Debug)] +pub struct FastRpOptions<'a> { + /// Floats per node; positive. + pub dimension: usize, + /// Weight of each averaging round in the result: entry `k` weighs the + /// embedding after `k + 1` rounds. Finite; the number of rounds is its length. + pub iteration_weights: &'a [f64], + /// Weight of the node's own random vector, round zero. + pub self_influence: f64, + /// Scale a node's random vector by `degree ^ strength`. Negative values + /// damp hubs, positive ones favour them. + pub normalization_strength: f64, + /// The same seed gives the same embedding. + pub seed: u64, +} + +impl Default for FastRpOptions<'_> { + fn default() -> Self { + Self { + dimension: 128, + iteration_weights: &[0.0, 1.0, 1.0], + self_influence: 0.0, + normalization_strength: 0.0, + seed: 0, + } + } +} + +/// An embedding per node. +pub struct FastRp { + graph: GraphProjection, + values: Buffer, + dimension: usize, +} + +impl FastRp { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + /// Floats per node. + pub fn dimension(&self) -> usize { + self.dimension + } + /// Embeddings row by row, in projection row order. + pub fn values(&self) -> &[f32] { + &self.values.values + } + /// The embedding of one projection row. + pub fn embedding(&self, row: usize) -> &[f32] { + &self.values.values[row * self.dimension..(row + 1) * self.dimension] + } + /// `nodeId`, `embedding`. + pub fn into_table(self) -> Result { + NodeTable::new(&self.graph).column( + "embedding", + NodeColumn::Vector { + values: self.values, + dimension: self.dimension, + }, + ) + } +} + +/// Nodes per chunk; fixed, so nothing depends on the pool width. +const CHUNK: usize = 1024; + +/// Embed every node. +/// +/// **Round zero.** Each node draws a vector whose entries are `+√3`, `0`, `-√3` +/// with probabilities 1/6, 2/3, 1/6, scaled by `degree ^ normalization_strength` +/// (a node of degree zero scales by zero unless the strength is zero). Entry +/// `j` of node `v` is a pure function of `(seed, v, j)`, so neither the order +/// of work nor the pool can change it. +/// +/// **Each round** replaces a node's vector with the weighted mean of the +/// vectors of the nodes its arcs reach, so after `k` rounds a node reflects its +/// `k`-hop out-neighbourhood; a node with no arcs goes to zero. Parallel edges +/// and self-loops count as arcs. Each round's vectors are scaled to unit length +/// and added to the result with that round's weight. `degree` is the weighted +/// out-degree. +/// +/// Arithmetic is single precision, as the output is. Identical at any pool +/// width. The embedding is not a function of the graph alone — it depends on +/// the seed and on node rows — so compare embeddings only within one run. +pub fn fast_rp(graph: &GraphProjection, options: FastRpOptions<'_>) -> Result { + let context = graph.execution(); + context.checkpoint()?; + let d = options.dimension; + if d == 0 + || !options.self_influence.is_finite() + || !options.normalization_strength.is_finite() + || options.iteration_weights.iter().any(|w| !w.is_finite()) + { + return Err(AlgorithmError::InvalidArguments( + "FastRP needs a positive embeddingDimension and finite weights".into(), + )); + } + let n = graph.node_count(); + let adjacency = graph.outgoing(); + let length = n.checked_mul(d).ok_or_else(|| { + AlgorithmError::InvalidArguments("embedding size overflows the address space".into()) + })?; + let mut current = Buffer::filled(length, 0.0f32, context)?; + let mut next = Buffer::filled(length, 0.0f32, context)?; + let mut result = Buffer::filled(length, 0.0f32, context)?; + if n == 0 { + return Ok(FastRp { + graph: graph.clone(), + values: result, + dimension: d, + }); + } + let degree = + |node: usize| -> f64 { adjacency.range(node).map(|arc| adjacency.weight(arc)).sum() }; + + // Round zero. + let root3 = 3.0f32.sqrt(); + parallel::for_chunks(&mut current.values, CHUNK * d, |first, chunk| { + let mut meter = Meter::new(context); + for (offset, row) in chunk.chunks_mut(d).enumerate() { + let node = first / d + offset; + meter.tick(d + adjacency.range(node).len())?; + let scale = if options.normalization_strength == 0.0 { + 1.0 + } else { + degree(node).powf(options.normalization_strength) as f32 + }; + let scale = if scale.is_finite() { scale } else { 0.0 }; + for (index, value) in row.iter_mut().enumerate() { + *value = match random::below(options.seed, node as u64, index as u64, 6) { + 0 => root3 * scale, + 1 => -root3 * scale, + _ => 0.0, + }; + } + } + meter.flush() + })?; + accumulate( + &mut result.values, + ¤t.values, + d, + options.self_influence as f32, + context, + )?; + + for &weight in options.iteration_weights { + let previous = ¤t.values; + parallel::for_chunks(&mut next.values, CHUNK * d, |first, chunk| { + let mut meter = Meter::new(context); + for (offset, row) in chunk.chunks_mut(d).enumerate() { + let node = first / d + offset; + let range = adjacency.range(node); + meter.tick(d * (1 + range.len()))?; + row.fill(0.0); + let mut total = 0.0f32; + for arc in range { + let arc_weight = adjacency.weight(arc) as f32; + total += arc_weight; + let other = adjacency.targets.values[arc]; + for (value, &theirs) in + row.iter_mut().zip(&previous[other * d..(other + 1) * d]) + { + *value += arc_weight * theirs; + } + } + if total > 0.0 { + for value in row.iter_mut() { + *value /= total; + } + } + } + meter.flush() + })?; + std::mem::swap(&mut current, &mut next); + accumulate( + &mut result.values, + ¤t.values, + d, + weight as f32, + context, + )?; + } + if result.values.iter().any(|value| !value.is_finite()) { + return Err(AlgorithmError::Numerical( + "embedding left the finite range".into(), + )); + } + Ok(FastRp { + graph: graph.clone(), + values: result, + dimension: d, + }) +} + +/// `into[v] += weight * from[v] / |from[v]|`; a zero vector adds nothing. +fn accumulate( + into: &mut [f32], + from: &[f32], + d: usize, + weight: f32, + context: &grust_procedures::ExecutionContext, +) -> Result<()> { + if weight == 0.0 { + return Ok(()); + } + parallel::for_chunks(into, CHUNK * d, |first, chunk| { + context.charge_work(chunk.len())?; + for (offset, row) in chunk.chunks_mut(d).enumerate() { + let source = &from[first + offset * d..first + (offset + 1) * d]; + let norm = source.iter().map(|v| v * v).sum::().sqrt(); + if norm > 0.0 { + for (value, &theirs) in row.iter_mut().zip(source) { + *value += weight * theirs / norm; + } + } + } + Ok(()) + })?; + Ok(()) +} diff --git a/crates/grust-algorithms/src/flow.rs b/crates/grust-algorithms/src/flow.rs new file mode 100644 index 00000000..1e4e741b --- /dev/null +++ b/crates/grust-algorithms/src/flow.rs @@ -0,0 +1,283 @@ +//! Maximum flow and minimum cut by Dinic's algorithm. + +use crate::{ + AlgorithmError, GraphProjection, Result, + buffer::Buffer, + meter::Meter, + table::{NodeColumn, NodeTable, TableScalar}, +}; + +const UNSEEN: usize = usize::MAX; + +/// A maximum flow and the minimum cut that proves it. +pub struct MaxFlow { + graph: GraphProjection, + value: f64, + from: Buffer, + to: Buffer, + edges: Buffer, + flows: Buffer, + source_side: Buffer, +} + +impl MaxFlow { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + /// The flow's value, which is also the cut's capacity. + pub fn value(&self) -> f64 { + self.value + } + /// One entry per edge carrying flow: (from row, to row, index into + /// [`GraphProjection::edges`], flow), in edge order. `from -> to` is the + /// direction the flow travels, whichever way the edge was stored. + pub fn flows(&self) -> impl Iterator + '_ { + (0..self.flows.values.len()).map(|row| { + ( + self.from.values[row], + self.to.values[row], + self.edges.values[row], + self.flows.values[row], + ) + }) + } + /// Per node, whether it lies on the source's side of the minimum cut. + pub fn source_side(&self) -> &[bool] { + &self.source_side.values + } + + /// Rows of `sourceNodeId`, `targetNodeId`, `edgeOrdinal`, `flow`, one per + /// edge carrying flow, then the `maxFlow` scalar. + pub fn into_flow_table(self) -> Result { + let context = self.graph.execution(); + let count = self.edges.values.len(); + context.charge_work(count)?; + let mut ordinals = Buffer::capacity(count, context)?; + for &slot in &self.edges.values { + ordinals.values.push( + i64::try_from(self.graph.edges()[slot].ordinal) + .map_err(|_| AlgorithmError::Numerical("edge ordinal exceeds Int64".into()))?, + ); + } + Ok(NodeTable::keyed(&self.graph, "sourceNodeId", self.from)? + .column("targetNodeId", NodeColumn::Node(self.to))? + .column("edgeOrdinal", NodeColumn::Integer(ordinals))? + .column("flow", NodeColumn::Number(self.flows))? + .scalar("maxFlow", TableScalar::Number(self.value))) + } + + /// Rows of `nodeId`, `sourceSide`, one per node, then the `maxFlow` scalar. + pub fn into_cut_table(self) -> Result { + Ok(NodeTable::new(&self.graph) + .column("sourceSide", NodeColumn::Boolean(self.source_side))? + .scalar("maxFlow", TableScalar::Number(self.value))) + } +} + +/// The most that can be sent from `source` to `target` when each arc carries at +/// most its weight, and a minimum cut: the nodes still reachable from `source` +/// once the flow is in place. Arcs leaving that set are full, and their +/// capacities sum to the flow's value. +/// +/// **Capacities** are the projection's weights, or one per arc without them. +/// Parallel edges are separate capacities. Self-loops and zero capacities carry +/// nothing. On an undirected projection an edge carries up to its capacity in +/// either direction, and its row reports the net flow in the direction it runs. +/// +/// **Exactness.** With integral capacities every quantity is exact. With +/// fractional ones the flow is a sum of floating-point augmentations: it is +/// feasible, and maximal up to rounding. +/// +/// Dinic's algorithm, iteratively, so a long path does not overflow a stack. +/// Sequential: a blocking flow is built one augmenting path at a time. +pub fn max_flow(graph: &GraphProjection, source: &str, target: &str) -> Result { + let context = graph.execution(); + context.checkpoint()?; + let source = graph.source(source)?; + let target = graph.source(target)?; + if source == target { + return Err(AlgorithmError::InvalidArguments( + "max flow needs a target other than its source".into(), + )); + } + let n = graph.node_count(); + let adjacency = graph.outgoing(); + let mut meter = Meter::new(context); + let carries = |node: usize, arc: usize| { + adjacency.targets.values[arc] != node && adjacency.weight(arc) > 0.0 + }; + + // Residual arcs grouped by tail: each projection arc, and its reverse. + let mut offsets = Buffer::filled(n + 1, 0usize, context)?; + for node in 0..n { + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + for arc in range { + if carries(node, arc) { + offsets.values[node + 1] += 1; + offsets.values[adjacency.targets.values[arc] + 1] += 1; + } + } + } + for node in 0..n { + offsets.values[node + 1] += offsets.values[node]; + } + let count = offsets.values[n]; + let mut head = Buffer::filled(count, 0usize, context)?; + let mut capacity = Buffer::filled(count, 0.0f64, context)?; + let mut pair = Buffer::filled(count, 0usize, context)?; + // The edge a forward arc stands for; `UNSEEN` marks a reverse arc. + let mut edge = Buffer::filled(count, UNSEEN, context)?; + let mut cursor = Buffer::capacity(n, context)?; + cursor.values.extend_from_slice(&offsets.values[..n]); + for node in 0..n { + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + for arc in range { + if !carries(node, arc) { + continue; + } + let other = adjacency.targets.values[arc]; + let forward = cursor.values[node]; + cursor.values[node] += 1; + let backward = cursor.values[other]; + cursor.values[other] += 1; + head.values[forward] = other; + capacity.values[forward] = adjacency.weight(arc); + pair.values[forward] = backward; + edge.values[forward] = adjacency.edge_slots.values[arc]; + head.values[backward] = node; + pair.values[backward] = forward; + } + } + + let mut level = Buffer::filled(n, UNSEEN, context)?; + let mut queue = Buffer::capacity(n, context)?; + let mut path: Buffer = Buffer::capacity(n, context)?; + let mut value = 0.0f64; + loop { + // Layer the residual graph by distance from the source. + level.values.fill(UNSEEN); + queue.values.clear(); + level.values[source] = 0; + queue.values.push(source); + let mut front = 0; + meter.tick(n)?; + while front < queue.values.len() { + let node = queue.values[front]; + front += 1; + let range = offsets.values[node]..offsets.values[node + 1]; + meter.tick(1 + range.len())?; + for arc in range { + let next = head.values[arc]; + if capacity.values[arc] > 0.0 && level.values[next] == UNSEEN { + level.values[next] = level.values[node] + 1; + queue.values.push(next); + } + } + } + if level.values[target] == UNSEEN { + break; + } + // Saturate it: augment along layered paths until none is left. An arc + // that led nowhere is never tried again within this layering. + cursor.values.copy_from_slice(&offsets.values[..n]); + loop { + path.values.clear(); + let mut node = source; + while node != target { + meter.tick(1)?; + let arc = cursor.values[node]; + if arc < offsets.values[node + 1] { + let next = head.values[arc]; + if capacity.values[arc] > 0.0 && level.values[next] == level.values[node] + 1 { + path.values.push(arc); + node = next; + } else { + cursor.values[node] += 1; + } + } else if let Some(arc) = path.values.pop() { + // Dead end: step back and give up on the arc that led here. + node = head.values[pair.values[arc]]; + cursor.values[node] += 1; + } else { + break; + } + } + if node != target { + break; + } + let bottleneck = path + .values + .iter() + .map(|&arc| capacity.values[arc]) + .fold(f64::INFINITY, f64::min); + meter.tick(path.values.len())?; + for &arc in &path.values { + capacity.values[arc] -= bottleneck; + capacity.values[pair.values[arc]] += bottleneck; + } + value += bottleneck; + } + } + if !value.is_finite() { + return Err(AlgorithmError::Numerical("flow value is not finite".into())); + } + + // Net flow per edge. A reverse arc's capacity is exactly what its forward + // arc carries; an undirected edge has two forward arcs, one each way. + let edges = graph.edges(); + let mut net = Buffer::filled(edges.len(), 0.0f64, context)?; + // Which way the edge's first-seen forward arc runs: (tail, head). + let mut along = Buffer::filled(edges.len(), (UNSEEN, UNSEEN), context)?; + for node in 0..n { + let range = offsets.values[node]..offsets.values[node + 1]; + meter.tick(1 + range.len())?; + for arc in range { + let slot = edge.values[arc]; + if slot == UNSEEN { + continue; + } + let carried = capacity.values[pair.values[arc]]; + if along.values[slot].0 == UNSEEN { + along.values[slot] = (node, head.values[arc]); + net.values[slot] += carried; + } else { + net.values[slot] -= carried; + } + } + } + let used = net.values.iter().filter(|&&flow| flow != 0.0).count(); + meter.tick(edges.len() + n)?; + let mut from = Buffer::capacity(used, context)?; + let mut to = Buffer::capacity(used, context)?; + let mut slots = Buffer::capacity(used, context)?; + let mut flows = Buffer::capacity(used, context)?; + for slot in 0..edges.len() { + let flow = net.values[slot]; + if flow == 0.0 { + continue; + } + let (tail, tip) = along.values[slot]; + let (tail, tip) = if flow > 0.0 { (tail, tip) } else { (tip, tail) }; + from.values.push(tail); + to.values.push(tip); + slots.values.push(slot); + flows.values.push(flow.abs()); + } + let mut source_side = Buffer::capacity(n, context)?; + source_side + .values + .extend(level.values.iter().map(|&depth| depth != UNSEEN)); + meter.flush()?; + Ok(MaxFlow { + graph: graph.clone(), + value, + from, + to, + edges: slots, + flows, + source_side, + }) +} diff --git a/crates/grust-algorithms/src/kcore.rs b/crates/grust-algorithms/src/kcore.rs new file mode 100644 index 00000000..74ca74e5 --- /dev/null +++ b/crates/grust-algorithms/src/kcore.rs @@ -0,0 +1,138 @@ +//! k-core decomposition by bucket peeling (Batagelj and Zaversnik), O(V + A). + +use crate::{ + AlgorithmError, GraphProjection, Orientation, Result, + buffer::Buffer, + meter::Meter, + table::{NodeColumn, NodeTable, TableScalar}, +}; + +/// Core number per node. The k-core is the maximal subgraph in which every node +/// has degree at least k; a node's core number is the largest k whose core +/// contains it. +pub struct KCore { + table: NodeTable, +} + +impl KCore { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + self.table.projection() + } + /// Core number per node, in projection row order; isolates have core 0. + pub fn core_values(&self) -> &[i64] { + self.table + .integers("coreValue") + .expect("k-core declares coreValue") + } + /// The largest core number: the graph's degeneracy. Zero for an empty graph. + pub fn degeneracy(&self) -> i64 { + match self.table.scalar_value("degeneracy") { + Some(TableScalar::Integer(value)) => value, + _ => 0, + } + } + /// The result as a node table: `coreValue`, then the `degeneracy` scalar. + pub fn into_table(self) -> NodeTable { + self.table + } +} + +/// Decompose an undirected projection into cores. +/// +/// Degree counts incident arcs with multiplicity, so **parallel edges each +/// count**; two nodes joined by three edges form a 3-core. **Self-loops are +/// ignored**: a loop never helps a node stay in a core. Any other orientation +/// is rejected rather than silently symmetrized — project with +/// `orientation: "undirected"`. +pub fn k_core(graph: &GraphProjection) -> Result { + let context = graph.execution(); + context.checkpoint()?; + if graph.orientation() != Orientation::Undirected { + return Err(AlgorithmError::InvalidArguments( + "kCore is defined on undirected graphs; project with orientation \"undirected\"".into(), + )); + } + let n = graph.node_count(); + let adjacency = graph.outgoing(); + let mut meter = Meter::new(context); + + // Degree without self-loops. Under the undirected arc contract a loop + // appears once in its node's row, every other edge once in each endpoint's. + let mut degree = Buffer::filled(n, 0usize, context)?; + let mut maximum = 0usize; + for node in 0..n { + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + let loops = adjacency.targets.values[range.clone()] + .iter() + .filter(|&&target| target == node) + .count(); + degree.values[node] = range.len() - loops; + maximum = maximum.max(degree.values[node]); + } + + // Counting sort of nodes by degree: `bin[d]` is where degree d starts in + // `order`, and `position[v]` is where v sits. + let mut bin = Buffer::filled(maximum + 2, 0usize, context)?; + for node in 0..n { + bin.values[degree.values[node] + 1] += 1; + } + for d in 0..=maximum { + bin.values[d + 1] += bin.values[d]; + } + let mut order = Buffer::filled(n, 0usize, context)?; + let mut position = Buffer::filled(n, 0usize, context)?; + { + let mut next = Buffer::filled(maximum + 1, 0usize, context)?; + next.values.copy_from_slice(&bin.values[..=maximum]); + for node in 0..n { + let slot = next.values[degree.values[node]]; + next.values[degree.values[node]] += 1; + order.values[slot] = node; + position.values[node] = slot; + } + } + meter.tick(n)?; + + // Peel in nondecreasing degree. Removing `node` lowers each heavier + // neighbour by one per arc, moving it one bin down by swapping it with the + // first node of its bin. A neighbour never drops below the current degree. + for index in 0..n { + let node = order.values[index]; + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + for arc in range { + let other = adjacency.targets.values[arc]; + if other == node || degree.values[other] <= degree.values[node] { + continue; + } + let d = degree.values[other]; + let first = bin.values[d]; + let swapped = order.values[first]; + if swapped != other { + let at = position.values[other]; + order.values[first] = other; + order.values[at] = swapped; + position.values[other] = first; + position.values[swapped] = at; + } + bin.values[d] += 1; + degree.values[other] -= 1; + } + } + meter.flush()?; + + let mut cores = Buffer::capacity(n, context)?; + let mut degeneracy = 0i64; + for &value in °ree.values { + let value = i64::try_from(value) + .map_err(|_| AlgorithmError::Numerical("core number exceeds Int64".into()))?; + degeneracy = degeneracy.max(value); + cores.values.push(value); + } + let table = NodeTable::new(graph) + .column("coreValue", NodeColumn::Integer(cores))? + .scalar("degeneracy", TableScalar::Integer(degeneracy)); + Ok(KCore { table }) +} diff --git a/crates/grust-algorithms/src/label_propagation.rs b/crates/grust-algorithms/src/label_propagation.rs new file mode 100644 index 00000000..290c20ff --- /dev/null +++ b/crates/grust-algorithms/src/label_propagation.rs @@ -0,0 +1,169 @@ +//! Label propagation with a deterministic, sequential update schedule. + +use crate::{ + AlgorithmError, GraphProjection, Result, + buffer::Buffer, + meter::Meter, + random, + table::{NodeColumn, NodeTable, TableScalar}, +}; + +/// Label propagation controls. +#[derive(Clone, Copy, Debug)] +pub struct LabelPropagationOptions { + /// Stop after this many passes over the nodes, converged or not. + pub max_iterations: usize, + /// `None` visits nodes in row order on every pass. A seed shuffles the order + /// afresh for each pass; the same seed gives the same result. + pub seed: Option, +} + +impl Default for LabelPropagationOptions { + fn default() -> Self { + Self { + max_iterations: 10, + seed: None, + } + } +} + +/// A community per node. +pub struct LabelPropagation { + graph: GraphProjection, + communities: Buffer, + iterations: usize, + converged: bool, +} + +impl LabelPropagation { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + /// For each node, the smallest row in its community. + pub fn communities(&self) -> &[usize] { + &self.communities.values + } + /// Passes run, including the one that changed nothing. + pub fn iterations(&self) -> usize { + self.iterations + } + /// Whether a pass changed nothing, rather than `max_iterations` stopping it. + pub fn converged(&self) -> bool { + self.converged + } + /// `communityId`, then the `iterations` and `converged` scalars. + pub fn into_table(self) -> Result { + let iterations = i64::try_from(self.iterations) + .map_err(|_| AlgorithmError::Numerical("iteration count exceeds Int64".into()))?; + Ok(NodeTable::new(&self.graph) + .column("communityId", NodeColumn::Node(self.communities))? + .scalar("iterations", TableScalar::Integer(iterations)) + .scalar("converged", TableScalar::Boolean(self.converged))) + } +} + +/// Every node starts in its own community and repeatedly adopts the label that +/// carries the most weight among the nodes with an arc **into** it, so labels +/// flow along the projection's arcs. A self-loop supports the node's own label. +/// +/// **Schedule.** Updates are asynchronous — a node sees labels already changed +/// in the same pass — and sequential, in row order or a seeded order per pass. +/// An asynchronous parallel schedule would not be reproducible, and a +/// synchronous one oscillates on bipartite graphs. +/// +/// **Ties.** A node keeps its label when that label is among the heaviest; +/// otherwise it takes the smallest of the heaviest. On an undirected projection +/// every change then strictly raises the weight of same-label edges, so the run +/// converges given enough passes. On a directed one it may not, and `converged` +/// says so. +/// +/// Communities are named by their smallest member, as `wcc` names components. +pub fn label_propagation( + graph: &GraphProjection, + options: LabelPropagationOptions, +) -> Result { + let context = graph.execution(); + context.checkpoint()?; + if options.max_iterations == 0 { + return Err(AlgorithmError::InvalidArguments( + "maxIterations must be positive".into(), + )); + } + let n = graph.node_count(); + let incoming = graph.incoming()?; + let mut labels = Buffer::capacity(n, context)?; + labels.values.extend(0..n); + let mut order = Buffer::capacity(n, context)?; + order.values.extend(0..n); + // Weight per label for the node in hand, and which labels it touched. + let mut support = Buffer::filled(n, 0.0f64, context)?; + let mut touched = Buffer::::capacity(n, context)?; + let mut meter = Meter::new(context); + + let mut iterations = 0; + let mut converged = false; + while iterations < options.max_iterations && !converged { + if let Some(seed) = options.seed { + meter.tick(n)?; + random::shuffle(&mut order.values, seed, iterations as u64); + } + iterations += 1; + let mut changed = false; + for &node in &order.values { + let range = incoming.range(node); + meter.tick(1 + range.len())?; + for arc in range { + let weight = incoming.weight(arc); + if weight == 0.0 { + // No support, and it must not register its label twice. + continue; + } + let label = labels.values[incoming.targets.values[arc]]; + if support.values[label] == 0.0 { + touched.values.push(label); + } + support.values[label] += weight; + } + let current = labels.values[node]; + let mut best = current; + let mut best_weight = support.values[current]; + for &label in &touched.values { + let weight = support.values[label]; + if weight > best_weight + || (weight == best_weight && best != current && label < best) + { + best = label; + best_weight = weight; + } + } + for &label in &touched.values { + support.values[label] = 0.0; + } + touched.values.clear(); + if best != current { + labels.values[node] = best; + changed = true; + } + } + converged = !changed; + } + + // Name each community by its smallest member: a label's first owner may + // have left it. + meter.tick(2 * n)?; + let mut smallest = Buffer::filled(n, usize::MAX, context)?; + for (node, &label) in labels.values.iter().enumerate() { + smallest.values[label] = smallest.values[label].min(node); + } + for label in &mut labels.values { + *label = smallest.values[*label]; + } + meter.flush()?; + Ok(LabelPropagation { + graph: graph.clone(), + communities: labels, + iterations, + converged, + }) +} diff --git a/crates/grust-algorithms/src/leiden.rs b/crates/grust-algorithms/src/leiden.rs new file mode 100644 index 00000000..4fca8e3e --- /dev/null +++ b/crates/grust-algorithms/src/leiden.rs @@ -0,0 +1,273 @@ +//! Leiden community detection: Louvain with a refinement phase that keeps +//! every community connected. + +use crate::{ + GraphProjection, Orientation, Result, + buffer::Buffer, + louvain::{self, Level, Louvain, LouvainOptions}, + meter::Meter, + random, +}; +use grust_procedures::ExecutionContext; + +/// Leiden takes Louvain's controls. +pub type LeidenOptions = LouvainOptions; +/// Leiden returns Louvain's result shape. +pub type Leiden = Louvain; + +/// Detect communities by modularity, guaranteeing each one is **connected**. +/// +/// Louvain can leave a community in two pieces: a node that held it together +/// moves away and nothing looks back. Leiden (Traag, Waltman and van Eck, 2019) +/// adds a refinement between moving and coarsening. Inside each community found +/// by the move phase, nodes start alone again and merge only with a neighbour's +/// group, only while still alone, and only when the merge does not lower +/// modularity. A group is therefore grown one attached node at a time and is +/// connected; the next level coarsens these groups, not the move phase's +/// communities, and starts its move phase from those communities. +/// +/// **The result is always the refined partition**, so it is connected however +/// the run ends, including at `max_levels`. Communities follow arcs in either +/// direction: connected means weakly connected on a directed projection. An arc +/// of zero weight attaches nothing. +/// +/// **Refinement is greedy here.** The paper picks among the admissible merges at +/// random, with a temperature `theta`; this picks the best one, ties to the +/// smallest group, which is that rule's limit as `theta` goes to zero. It keeps +/// the connectivity guarantee and makes the run reproducible, and it gives up +/// the paper's guarantee of eventually reaching every optimal partition. +/// +/// Orientation, multigraph and self-loop semantics, options and determinism are +/// Louvain's. Its modularity is often, but not always, higher than Louvain's. +pub fn leiden(graph: &GraphProjection, options: LeidenOptions) -> Result { + let context = graph.execution(); + context.checkpoint()?; + louvain::validate(&options)?; + let n = graph.node_count(); + let directed = graph.orientation() != Orientation::Undirected; + let mut level = louvain::first_level(graph, directed, context)?; + + // `assignment[v]` is v's node in the current level: the refined groups. + let mut assignment = Buffer::capacity(n, context)?; + assignment.values.extend(0..n); + let mut initial: Option> = None; + let mut levels = 0usize; + let mut converged = false; + + for depth in 0..options.max_levels { + let outcome = louvain::move_nodes( + &level, + directed, + &options, + depth as u64, + initial.as_ref().map(|labels| labels.values.as_slice()), + context, + )?; + let refined = refine( + &level, + &outcome.community, + directed, + &options, + depth as u64, + context, + )?; + let (dense, count) = louvain::renumber(&refined, context)?; + if count == level.nodes() { + // Nothing merged: the groups carried so far are the answer. + converged = outcome.settled; + break; + } + levels += 1; + let mut meter = Meter::new(context); + for slot in &mut assignment.values { + meter.tick(1)?; + *slot = dense.values[*slot]; + } + // The next level's nodes are these groups; each starts in the community + // the move phase gave its members, named by that community's first group. + let mut first_group = Buffer::filled(level.nodes(), usize::MAX, context)?; + let mut next_initial = Buffer::filled(count, 0usize, context)?; + for node in 0..level.nodes() { + meter.tick(1)?; + let community = outcome.community.values[node]; + let group = dense.values[node]; + if first_group.values[community] == usize::MAX { + first_group.values[community] = group; + } + next_initial.values[group] = first_group.values[community]; + } + meter.flush()?; + level = louvain::coarsen(&level, &dense, count, directed, context)?; + initial = Some(next_initial); + } + louvain::finish( + graph, + &assignment.values, + levels, + converged, + options.resolution, + ) +} + +/// Split each community of `community` into connected groups. Returns a label +/// per node; a group is labelled by the node that founded it. +fn refine( + level: &Level, + community: &Buffer, + directed: bool, + options: &LouvainOptions, + depth: u64, + context: &ExecutionContext, +) -> Result> { + let n = level.nodes(); + let mut meter = Meter::new(context); + let gamma = options.resolution; + // Both matrix entries of an undirected arc, as in the move phase. + let scale = if directed { 1.0 } else { 2.0 }; + let in_rows = |node: usize| match &level.incoming { + Some((offsets, _, _)) => offsets.values[node]..offsets.values[node + 1], + None => 0..0, + }; + + // Strengths, per node and per community, and each node's weight to the + // rest of its own community. + let mut k_out = Buffer::capacity(n, context)?; + let mut k_in = Buffer::capacity(n, context)?; + let mut external = Buffer::filled(n, 0.0f64, context)?; + let mut community_out = Buffer::filled(n, 0.0f64, context)?; + let mut community_in = Buffer::filled(n, 0.0f64, context)?; + for node in 0..n { + let row = level.row(node); + meter.tick(1 + row.len() + in_rows(node).len())?; + let own = level.inside.values[node]; + let mut out = own; + for arc in row { + let weight = level.weights.values[arc]; + out += weight; + if community.values[level.targets.values[arc]] == community.values[node] { + external.values[node] += scale * weight; + } + } + let mut into = own; + match &level.incoming { + Some((_, sources, weights)) => { + for arc in in_rows(node) { + into += weights.values[arc]; + if community.values[sources.values[arc]] == community.values[node] { + external.values[node] += weights.values[arc]; + } + } + } + None => into = out, + } + k_out.values.push(out); + k_in.values.push(into); + community_out.values[community.values[node]] += out; + community_in.values[community.values[node]] += into; + } + let total: f64 = k_out.values.iter().sum(); + + let mut group = Buffer::capacity(n, context)?; + group.values.extend(0..n); + if total <= 0.0 || !total.is_finite() { + return Ok(group); + } + let mut group_out = Buffer::capacity(n, context)?; + group_out.values.extend_from_slice(&k_out.values); + let mut group_in = Buffer::capacity(n, context)?; + group_in.values.extend_from_slice(&k_in.values); + let mut alone = Buffer::filled(n, true, context)?; + + // A set S inside community C is well connected when its weight to the rest + // of C is at least what the null model expects. + let well_connected = |weight: f64, out: f64, into: f64, c: usize| { + weight + >= gamma + * (out * (community_in.values[c] - into) + into * (community_out.values[c] - out)) + / total + }; + + let mut order = Buffer::capacity(n, context)?; + order.values.extend(0..n); + if let Some(seed) = options.seed { + // A stream of its own, so refinement does not replay the move order. + random::shuffle(&mut order.values, seed, depth ^ (1 << 63)); + } + let mut affinity = Buffer::filled(n, 0.0f64, context)?; + let mut touched = Buffer::::capacity(n, context)?; + + for &node in &order.values { + let home = community.values[node]; + meter.tick(1 + level.row(node).len() + in_rows(node).len())?; + if !alone.values[node] + || !well_connected( + external.values[node], + k_out.values[node], + k_in.values[node], + home, + ) + { + continue; + } + let mut reach = |other: usize, weight: f64| { + if community.values[other] == home && weight > 0.0 { + let label = group.values[other]; + if affinity.values[label] == 0.0 { + touched.values.push(label); + } + affinity.values[label] += weight; + } + }; + for arc in level.row(node) { + reach(level.targets.values[arc], scale * level.weights.values[arc]); + } + if let Some((_, sources, weights)) = &level.incoming { + for arc in in_rows(node) { + reach(sources.values[arc], weights.values[arc]); + } + } + + // The node is alone, so lifting it out leaves an empty group scoring 0. + let mut best = None; + let mut best_score = 0.0; + for &label in &touched.values { + if !well_connected( + external.values[label], + group_out.values[label], + group_in.values[label], + home, + ) { + continue; + } + let score = affinity.values[label] + - gamma + * (k_out.values[node] * group_in.values[label] + + k_in.values[node] * group_out.values[label]) + / total; + let better = match best { + None => score >= best_score, + Some(current) => score > best_score || (score == best_score && label < current), + }; + if better { + best = Some(label); + best_score = score; + } + } + if let Some(label) = best { + // `external` is kept per group under its label. + external.values[label] = + external.values[label] + external.values[node] - 2.0 * affinity.values[label]; + group_out.values[label] += k_out.values[node]; + group_in.values[label] += k_in.values[node]; + group.values[node] = label; + alone.values[node] = false; + alone.values[label] = false; + } + for &label in &touched.values { + affinity.values[label] = 0.0; + } + touched.values.clear(); + } + meter.flush()?; + Ok(group) +} diff --git a/crates/grust-algorithms/src/lib.rs b/crates/grust-algorithms/src/lib.rs index f9ad8073..ef18c5d4 100644 --- a/crates/grust-algorithms/src/lib.rs +++ b/crates/grust-algorithms/src/lib.rs @@ -10,17 +10,48 @@ mod arrow_input; mod arrow_output; #[cfg(feature = "arrow")] pub use arrow_output::{ArrowResultBatch, ArrowResultCursor}; +mod betweenness; +pub use betweenness::{Betweenness, BetweennessOptions, betweenness}; +mod biconnected; +pub use biconnected::{Biconnectivity, biconnectivity}; mod buffer; +mod closeness; +pub use closeness::{ClosenessOptions, DistanceCentrality, HarmonicOptions, closeness, harmonic}; mod degree; pub use degree::{Degrees, degree}; +mod fastrp; +pub use fastrp::{FastRp, FastRpOptions, fast_rp}; +mod flow; mod graph_input; +pub use flow::{MaxFlow, max_flow}; +mod kcore; +mod label_propagation; +pub use label_propagation::{LabelPropagation, LabelPropagationOptions, label_propagation}; +mod leiden; +pub use leiden::{Leiden, LeidenOptions, leiden}; +mod louvain; +pub use kcore::{KCore, k_core}; +pub use louvain::{Louvain, LouvainOptions, louvain}; +mod meter; mod ordering; +mod parallel; pub use ordering::{NodeOrder, TopologicalOrder, depth_first, topological_sort}; mod pagerank; mod projection; +mod random; mod shortest; +mod similarity; mod statistics; +pub use similarity::{NodeSimilarity, NodeSimilarityOptions, SimilarityMetric, node_similarity}; +mod spanning; +pub use spanning::{SpanningForest, SpanningObjective, SpanningTreeOptions, spanning_tree}; +mod spectral; +pub use spectral::{IteratedScores, IterationOptions, KatzOptions, eigenvector, hits, katz}; +mod table; +mod triangles; pub use statistics::{CsrEstimate, ProjectionStatistics}; +pub use table::{NodeTable, TableScalar, TableType, TableValue}; +pub use triangles::{TriangleOptions, Triangles, triangles}; mod traversal; pub use pagerank::{PageRank, PageRankOptions, pagerank}; diff --git a/crates/grust-algorithms/src/louvain.rs b/crates/grust-algorithms/src/louvain.rs new file mode 100644 index 00000000..4ac6652a --- /dev/null +++ b/crates/grust-algorithms/src/louvain.rs @@ -0,0 +1,627 @@ +//! Louvain community detection: greedy modularity optimization by local moves +//! and coarsening (Blondel et al. 2008), for undirected and directed graphs. +//! +//! One formula serves both. With `A` the (weighted) adjacency matrix, `M` the +//! sum of all its entries, and per community `in_c = Σ A_ij` over members, +//! `out_c = Σ k_out`, `into_c = Σ k_in`: +//! +//! ```text +//! Q = Σ_c [ in_c / M − γ · out_c · into_c / M² ] +//! ``` +//! +//! On an `Undirected` projection `A` is symmetric (each edge is two arcs, a +//! loop contributes `2w` to `A_uu`), `M = 2m`, and this is Newman's modularity. +//! On `Outgoing` and `Incoming` projections `A_ij` is the arc weight, `M = m`, +//! and this is Leicht and Newman's directed modularity. + +use crate::{ + AlgorithmError, GraphProjection, Orientation, Result, + buffer::Buffer, + meter::Meter, + random, + table::{NodeColumn, NodeTable, TableScalar}, +}; +use grust_procedures::ExecutionContext; + +/// Louvain controls. +#[derive(Clone, Copy, Debug)] +pub struct LouvainOptions { + /// Resolution γ: finite and nonnegative. 1 is standard modularity; larger + /// values favour smaller communities, and 0 merges each connected component. + pub resolution: f64, + /// Positive maximum number of coarsening levels. + pub max_levels: usize, + /// Positive maximum number of sweeps over the nodes within one level. + pub max_iterations: usize, + /// A sweep, or a level, that improves modularity by less than this ends its + /// loop. Finite and nonnegative. + pub tolerance: f64, + /// Visit nodes in a seeded random order. `None` visits them in row order. + /// Either way the result is a function of the projection and the options. + pub seed: Option, +} + +impl Default for LouvainOptions { + fn default() -> Self { + Self { + resolution: 1.0, + max_levels: 10, + max_iterations: 10, + tolerance: 1e-4, + seed: None, + } + } +} + +/// A community per node, with the evidence of how it was reached. +pub struct Louvain { + graph: GraphProjection, + communities: Buffer, + modularity: f64, + levels: usize, + converged: bool, +} + +impl Louvain { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + /// Per node, the row of its community's smallest member: canonical ids, as + /// `wcc` reports components. + pub fn communities(&self) -> &[usize] { + &self.communities.values + } + /// Modularity of the returned partition, recomputed from the projection + /// rather than accumulated from the moves. + pub fn modularity(&self) -> f64 { + self.modularity + } + /// Coarsening levels that moved at least one node. + pub fn levels(&self) -> usize { + self.levels + } + /// Whether the search ended because no move improved modularity, rather + /// than because `max_levels` or `max_iterations` stopped it. + pub fn converged(&self) -> bool { + self.converged + } + /// `communityId`, then the `modularity`, `levels` and `converged` scalars. + pub fn into_table(self) -> Result { + let levels = i64::try_from(self.levels) + .map_err(|_| AlgorithmError::Numerical("level count exceeds Int64".into()))?; + Ok(NodeTable::new(&self.graph) + .column("communityId", NodeColumn::Node(self.communities))? + .scalar("modularity", TableScalar::Number(self.modularity)) + .scalar("levels", TableScalar::Integer(levels)) + .scalar("converged", TableScalar::Boolean(self.converged))) + } +} + +/// The graph one level works on. Rows hold arcs to **other** nodes only; what a +/// node carries to itself — loops at the first level, a whole community's +/// internal weight after coarsening — is `inside`, already in `A_uu` units. +pub(super) struct Level { + pub(super) offsets: Buffer, + pub(super) targets: Buffer, + pub(super) weights: Buffer, + pub(super) inside: Buffer, + /// In-arcs, for directed graphs only; an undirected row is its own mirror. + pub(super) incoming: Option<(Buffer, Buffer, Buffer)>, +} + +impl Level { + pub(super) fn nodes(&self) -> usize { + self.inside.values.len() + } + pub(super) fn row(&self, node: usize) -> std::ops::Range { + self.offsets.values[node]..self.offsets.values[node + 1] + } +} + +/// Detect communities by modularity. +/// +/// **Orientation.** An `Undirected` projection is optimized for Newman's +/// modularity; `Outgoing` and `Incoming` projections for Leicht and Newman's +/// directed modularity, which rewards arcs that are surprising given the +/// source's out-strength and the target's in-strength. +/// +/// **Multigraphs.** Parallel edges add their weights. A self-loop of weight `w` +/// contributes `2w` to its node's strength and to its community's internal +/// weight on an undirected projection, and `w` on a directed one. Isolates, and +/// every node of a graph with no weight at all, stay singleton communities. +/// +/// **Determinism.** Nodes are visited in row order, or in the order `seed` +/// fixes; ties go to staying put, then to the community whose current id is +/// smallest. Moves are applied one at a time, so the result does not depend on +/// the rayon pool: asynchronous parallel moves, which NetworKit's PLM uses, are +/// not reproducible, and reproducibility is the contract here. +pub fn louvain(graph: &GraphProjection, options: LouvainOptions) -> Result { + let context = graph.execution(); + context.checkpoint()?; + validate(&options)?; + let n = graph.node_count(); + let directed = graph.orientation() != Orientation::Undirected; + let mut level = first_level(graph, directed, context)?; + + // `assignment[v]` is v's node in the current level. + let mut assignment = Buffer::capacity(n, context)?; + assignment.values.extend(0..n); + let mut levels = 0usize; + let mut converged = false; + + for depth in 0..options.max_levels { + let outcome = move_nodes(&level, directed, &options, depth as u64, None, context)?; + if !outcome.moved { + converged = outcome.settled; + break; + } + levels += 1; + let (dense, count) = renumber(&outcome.community, context)?; + let mut meter = Meter::new(context); + for slot in &mut assignment.values { + meter.tick(1)?; + *slot = dense.values[*slot]; + } + meter.flush()?; + if count == level.nodes() { + // Every node moved into a community of one: nothing left to merge. + converged = true; + break; + } + if outcome.gain < options.tolerance { + converged = true; + break; + } + level = coarsen(&level, &dense, count, directed, context)?; + } + + finish( + graph, + &assignment.values, + levels, + converged, + options.resolution, + ) +} + +pub(super) fn validate(options: &LouvainOptions) -> Result<()> { + if !options.resolution.is_finite() + || options.resolution < 0.0 + || !options.tolerance.is_finite() + || options.tolerance < 0.0 + || options.max_levels == 0 + || options.max_iterations == 0 + { + return Err(AlgorithmError::InvalidArguments( + "community detection requires finite nonnegative resolution and tolerance, and positive maxLevels and maxIterations".into(), + )); + } + Ok(()) +} + +/// Name communities by their smallest member and measure the partition. +pub(super) fn finish( + graph: &GraphProjection, + assignment: &[usize], + levels: usize, + converged: bool, + resolution: f64, +) -> Result { + let context = graph.execution(); + let n = graph.node_count(); + context.charge_work(2 * n)?; + // Canonical ids: the smallest member's row. + let mut smallest = Buffer::filled(n, usize::MAX, context)?; + for (node, &community) in assignment.iter().enumerate() { + if smallest.values[community] == usize::MAX { + smallest.values[community] = node; + } + } + let mut communities = Buffer::capacity(n, context)?; + communities + .values + .extend(assignment.iter().map(|&c| smallest.values[c])); + let modularity = modularity_of(graph, &communities.values, resolution, context)?; + Ok(Louvain { + graph: graph.clone(), + communities, + modularity, + levels, + converged, + }) +} + +/// Modularity of `communities` (any labels below the node count) on `graph`, +/// by the formula at the top of this file. Independent of the search. +pub(crate) fn modularity_of( + graph: &GraphProjection, + communities: &[usize], + resolution: f64, + context: &ExecutionContext, +) -> Result { + let n = graph.node_count(); + let directed = graph.orientation() != Orientation::Undirected; + let adjacency = graph.outgoing(); + let mut inside = Buffer::filled(n, 0.0f64, context)?; + let mut out = Buffer::filled(n, 0.0f64, context)?; + let mut into = Buffer::filled(n, 0.0f64, context)?; + let mut total = 0.0f64; + let mut meter = Meter::new(context); + for source in 0..n { + let range = adjacency.range(source); + meter.tick(1 + range.len())?; + for arc in range { + let target = adjacency.targets.values[arc]; + // An undirected loop occupies one arc but is two matrix entries' worth. + let weight = if !directed && target == source { + 2.0 * adjacency.weight(arc) + } else { + adjacency.weight(arc) + }; + total += weight; + out.values[communities[source]] += weight; + into.values[communities[target]] += weight; + if communities[source] == communities[target] { + inside.values[communities[source]] += weight; + } + } + } + meter.flush()?; + if total <= 0.0 { + return Ok(0.0); + } + let mut modularity = 0.0; + for community in 0..n { + modularity += inside.values[community] / total + - resolution * (out.values[community] / total) * (into.values[community] / total); + } + if !modularity.is_finite() { + return Err(AlgorithmError::Numerical("modularity is not finite".into())); + } + Ok(modularity) +} + +pub(super) fn first_level( + graph: &GraphProjection, + directed: bool, + context: &ExecutionContext, +) -> Result { + let n = graph.node_count(); + let adjacency = graph.outgoing(); + let arcs = adjacency.targets.values.len(); + let mut offsets = Buffer::filled(n + 1, 0usize, context)?; + let mut targets = Buffer::capacity(arcs, context)?; + let mut weights = Buffer::capacity(arcs, context)?; + let mut inside = Buffer::filled(n, 0.0f64, context)?; + let mut meter = Meter::new(context); + for node in 0..n { + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + for arc in range { + let target = adjacency.targets.values[arc]; + let weight = adjacency.weight(arc); + if target == node { + inside.values[node] += if directed { weight } else { 2.0 * weight }; + } else { + targets.values.push(target); + weights.values.push(weight); + } + } + offsets.values[node + 1] = targets.values.len(); + } + meter.flush()?; + let incoming = directed + .then(|| transpose(&offsets, &targets, &weights, context)) + .transpose()?; + Ok(Level { + offsets, + targets, + weights, + inside, + incoming, + }) +} + +type Csr = (Buffer, Buffer, Buffer); + +fn transpose( + offsets: &Buffer, + targets: &Buffer, + weights: &Buffer, + context: &ExecutionContext, +) -> Result { + let n = offsets.values.len() - 1; + let arcs = targets.values.len(); + let mut meter = Meter::new(context); + let mut reversed = Buffer::filled(n + 1, 0usize, context)?; + for &target in &targets.values { + reversed.values[target + 1] += 1; + } + for node in 0..n { + reversed.values[node + 1] += reversed.values[node]; + } + let mut next = Buffer::capacity(n, context)?; + next.values.extend_from_slice(&reversed.values[..n]); + let mut sources = Buffer::filled(arcs, 0usize, context)?; + let mut carried = Buffer::filled(arcs, 0.0f64, context)?; + for source in 0..n { + let range = offsets.values[source]..offsets.values[source + 1]; + meter.tick(1 + range.len())?; + for arc in range { + let slot = next.values[targets.values[arc]]; + next.values[targets.values[arc]] += 1; + sources.values[slot] = source; + carried.values[slot] = weights.values[arc]; + } + } + meter.flush()?; + Ok((reversed, sources, carried)) +} + +pub(super) struct Moves { + pub(super) community: Buffer, + /// Any node changed community in this level. + pub(super) moved: bool, + /// The last sweep moved nothing: a local optimum, not a limit. + pub(super) settled: bool, + /// Modularity gained in this level. + pub(super) gain: f64, +} + +/// Local moving from `initial` (singletons when `None`). Leiden restarts each +/// level from the partition the level above found, not from singletons. +pub(super) fn move_nodes( + level: &Level, + directed: bool, + options: &LouvainOptions, + depth: u64, + initial: Option<&[usize]>, + context: &ExecutionContext, +) -> Result { + let n = level.nodes(); + let mut meter = Meter::new(context); + + // Strengths, and the total M they sum to. + let mut k_out = Buffer::capacity(n, context)?; + let mut k_in = Buffer::capacity(n, context)?; + for node in 0..n { + let row = level.row(node); + meter.tick(1 + row.len())?; + let own = level.inside.values[node]; + k_out + .values + .push(own + level.weights.values[row].iter().sum::()); + k_in.values.push(match &level.incoming { + Some((offsets, _, weights)) => { + own + weights.values[offsets.values[node]..offsets.values[node + 1]] + .iter() + .sum::() + } + None => k_out.values[node], + }); + } + let total: f64 = k_out.values.iter().sum(); + let mut community = Buffer::capacity(n, context)?; + match initial { + Some(labels) => community.values.extend_from_slice(labels), + None => community.values.extend(0..n), + } + if !total.is_finite() { + return Err(AlgorithmError::Numerical( + "total edge weight is not finite".into(), + )); + } + if total <= 0.0 { + return Ok(Moves { + community, + moved: false, + settled: true, + gain: 0.0, + }); + } + + let mut out_of = Buffer::filled(n, 0.0f64, context)?; + let mut into = Buffer::filled(n, 0.0f64, context)?; + for node in 0..n { + out_of.values[community.values[node]] += k_out.values[node]; + into.values[community.values[node]] += k_in.values[node]; + } + let mut order = Buffer::capacity(n, context)?; + order.values.extend(0..n); + if let Some(seed) = options.seed { + random::shuffle(&mut order.values, seed, depth); + } + + // Weight between the node under consideration and each nearby community; + // `touched` lists the communities to reset, so a sweep never clears O(n). + let mut affinity = Buffer::filled(n, 0.0f64, context)?; + let mut seen = Buffer::filled(n, false, context)?; + let mut touched = Buffer::capacity(n, context)?; + let gamma = options.resolution; + let mut moved = false; + let mut settled = false; + let mut level_gain = 0.0f64; + + for _ in 0..options.max_iterations { + let mut sweep_moves = 0usize; + let mut sweep_gain = 0.0f64; + for &node in &order.values { + let home = community.values[node]; + touched.values.clear(); + seen.values[home] = true; + touched.values.push(home); + let row = level.row(node); + meter.tick(1 + row.len())?; + for arc in row { + let other = community.values[level.targets.values[arc]]; + if !seen.values[other] { + seen.values[other] = true; + touched.values.push(other); + } + // An undirected arc stands for both matrix entries. + affinity.values[other] += + level.weights.values[arc] * if directed { 1.0 } else { 2.0 }; + } + if let Some((offsets, sources, weights)) = &level.incoming { + let row = offsets.values[node]..offsets.values[node + 1]; + meter.tick(row.len())?; + for arc in row { + let other = community.values[sources.values[arc]]; + if !seen.values[other] { + seen.values[other] = true; + touched.values.push(other); + } + affinity.values[other] += weights.values[arc]; + } + } + + // Lift the node out, then score every candidate the same way. + out_of.values[home] -= k_out.values[node]; + into.values[home] -= k_in.values[node]; + let score = |candidate: usize| { + affinity.values[candidate] + - gamma + * (k_out.values[node] * into.values[candidate] + + k_in.values[node] * out_of.values[candidate]) + / total + }; + let stay = score(home); + let mut best = home; + let mut best_score = stay; + for &candidate in &touched.values { + let value = score(candidate); + if value > best_score || (value == best_score && best != home && candidate < best) { + best = candidate; + best_score = value; + } + } + out_of.values[best] += k_out.values[node]; + into.values[best] += k_in.values[node]; + if best != home { + community.values[node] = best; + sweep_moves += 1; + sweep_gain += (best_score - stay) / total; + } + for &candidate in &touched.values { + affinity.values[candidate] = 0.0; + seen.values[candidate] = false; + } + } + level_gain += sweep_gain; + if sweep_moves == 0 { + settled = true; + break; + } + moved = true; + if sweep_gain < options.tolerance { + settled = true; + break; + } + } + meter.flush()?; + Ok(Moves { + community, + moved, + settled, + gain: level_gain, + }) +} + +/// Dense ids in order of each community's smallest member. +pub(super) fn renumber( + community: &Buffer, + context: &ExecutionContext, +) -> Result<(Buffer, usize)> { + let n = community.values.len(); + let mut id = Buffer::filled(n, usize::MAX, context)?; + let mut dense = Buffer::capacity(n, context)?; + let mut count = 0; + for &label in &community.values { + if id.values[label] == usize::MAX { + id.values[label] = count; + count += 1; + } + dense.values.push(id.values[label]); + } + Ok((dense, count)) +} + +pub(super) fn coarsen( + level: &Level, + dense: &Buffer, + count: usize, + directed: bool, + context: &ExecutionContext, +) -> Result { + let n = level.nodes(); + let mut meter = Meter::new(context); + + // Members grouped by community, by counting sort. + let mut start = Buffer::filled(count + 1, 0usize, context)?; + for &community in &dense.values { + start.values[community + 1] += 1; + } + for community in 0..count { + start.values[community + 1] += start.values[community]; + } + let mut next = Buffer::capacity(count, context)?; + next.values.extend_from_slice(&start.values[..count]); + let mut members = Buffer::filled(n, 0usize, context)?; + for node in 0..n { + let community = dense.values[node]; + members.values[next.values[community]] = node; + next.values[community] += 1; + } + meter.tick(n)?; + + let mut offsets = Buffer::filled(count + 1, 0usize, context)?; + let mut targets = Buffer::capacity(level.targets.values.len(), context)?; + let mut weights = Buffer::capacity(level.targets.values.len(), context)?; + let mut inside = Buffer::filled(count, 0.0f64, context)?; + let mut sum = Buffer::filled(count, 0.0f64, context)?; + let mut seen = Buffer::filled(count, false, context)?; + let mut touched = Buffer::capacity(count, context)?; + for community in 0..count { + touched.values.clear(); + for &node in &members.values[start.values[community]..start.values[community + 1]] { + inside.values[community] += level.inside.values[node]; + let row = level.row(node); + meter.tick(1 + row.len())?; + for arc in row { + let other = dense.values[level.targets.values[arc]]; + let weight = level.weights.values[arc]; + if other == community { + // Both arcs of an undirected internal edge pass through here. + inside.values[community] += weight; + continue; + } + if !seen.values[other] { + seen.values[other] = true; + touched.values.push(other); + } + sum.values[other] += weight; + } + } + // Sorted so the coarse rows do not depend on member order. + touched.values.sort_unstable(); + for &other in &touched.values { + targets.values.push(other); + weights.values.push(sum.values[other]); + sum.values[other] = 0.0; + seen.values[other] = false; + } + offsets.values[community + 1] = targets.values.len(); + } + meter.flush()?; + let incoming = directed + .then(|| transpose(&offsets, &targets, &weights, context)) + .transpose()?; + Ok(Level { + offsets, + targets, + weights, + inside, + incoming, + }) +} diff --git a/crates/grust-algorithms/src/meter.rs b/crates/grust-algorithms/src/meter.rs new file mode 100644 index 00000000..6449337d --- /dev/null +++ b/crates/grust-algorithms/src/meter.rs @@ -0,0 +1,42 @@ +//! Work accounting in blocks. A kernel's inner loop does a few nanoseconds of +//! work per arc; admitting each arc separately costs more than the arc. A meter +//! counts locally and admits at most every 1024 units, so budgets, cancellation +//! and deadlines are still observed promptly and totals stay exact. + +use grust_procedures::{ExecutionContext, Result}; + +const BLOCK: usize = 1024; + +pub(crate) struct Meter<'a> { + context: &'a ExecutionContext, + pending: usize, +} + +impl<'a> Meter<'a> { + pub(crate) fn new(context: &'a ExecutionContext) -> Self { + Self { + context, + pending: 0, + } + } + + /// Count `units` of work, admitting a block once enough has accumulated. + #[inline] + pub(crate) fn tick(&mut self, units: usize) -> Result<()> { + self.pending += units; + if self.pending >= BLOCK { + self.flush()?; + } + Ok(()) + } + + /// Admit whatever is still counted. Call before returning a result: work + /// left unadmitted would make a budget look larger than it is. + pub(crate) fn flush(&mut self) -> Result<()> { + if self.pending > 0 { + self.context + .charge_work(std::mem::take(&mut self.pending))?; + } + Ok(()) + } +} diff --git a/crates/grust-algorithms/src/parallel.rs b/crates/grust-algorithms/src/parallel.rs new file mode 100644 index 00000000..ee61f18c --- /dev/null +++ b/crates/grust-algorithms/src/parallel.rs @@ -0,0 +1,166 @@ +//! Deterministic fan-out. A kernel splits its nodes into contiguous blocks of +//! roughly equal work, runs one task per block on the caller's rayon pool, and +//! combines the block results **in block order**. The partition depends only on +//! the graph and the pool width, and every combination used with it is exact +//! (integer sums, per-node writes to disjoint slots), so the answer is the same +//! at any width. Without the `parallel` feature the blocks run in a plain loop. + +use grust_procedures::Result; + +/// Tasks the caller's pool can run at once; 1 without the `parallel` feature. +pub(crate) fn width() -> usize { + #[cfg(feature = "parallel")] + { + rayon::current_num_threads().max(1) + } + #[cfg(not(feature = "parallel"))] + { + 1 + } +} + +/// Split `0..weights.len()` into at most `parts` contiguous ranges of roughly +/// equal total weight. Every node lands in exactly one range, in order. +pub(crate) fn balanced_ranges(weights: &[usize], parts: usize) -> Vec> { + let n = weights.len(); + if n == 0 { + return Vec::new(); + } + let parts = parts.clamp(1, n); + let total: u128 = weights.iter().map(|&w| w as u128 + 1).sum(); + let mut ranges = Vec::with_capacity(parts); + let mut start = 0; + let mut seen: u128 = 0; + for (node, &weight) in weights.iter().enumerate() { + seen += weight as u128 + 1; + let filled = ranges.len() as u128 + 1; + if seen * parts as u128 >= total * filled && ranges.len() + 1 < parts { + ranges.push(start..node + 1); + start = node + 1; + } + } + ranges.push(start..n); + ranges.retain(|range| !range.is_empty()); + ranges +} + +/// Run `task` on every range and return the results in range order. +pub(crate) fn map_ranges( + ranges: &[std::ops::Range], + task: impl Fn(std::ops::Range) -> Result + Sync, +) -> Result> { + #[cfg(feature = "parallel")] + { + use rayon::prelude::*; + ranges + .par_iter() + .map(|range| task(range.clone())) + .collect::>>() + } + #[cfg(not(feature = "parallel"))] + { + ranges.iter().map(|range| task(range.clone())).collect() + } +} + +/// Run `task(block)` for `0..blocks` and hand each result to `merge` **in block +/// order**. Blocks run `width()` at a time, so at most that many results are +/// alive at once; which blocks exist, what each computes and the order they are +/// merged in do not depend on the pool, so neither does a floating-point sum +/// built by `merge`. +pub(crate) fn ordered_blocks( + blocks: usize, + task: impl Fn(usize) -> Result + Sync, + mut merge: impl FnMut(usize, T) -> Result<()>, +) -> Result<()> { + let wave = width(); + let mut first = 0; + while first < blocks { + let last = (first + wave).min(blocks); + let indices: Vec = (first..last).collect(); + #[cfg(feature = "parallel")] + let results = { + use rayon::prelude::*; + indices + .par_iter() + .map(|&block| task(block)) + .collect::>>()? + }; + #[cfg(not(feature = "parallel"))] + let results = indices + .iter() + .map(|&block| task(block)) + .collect::>>()?; + for (block, result) in indices.into_iter().zip(results) { + merge(block, result)?; + } + first = last; + } + Ok(()) +} + +/// Run `task(first_index, chunk)` over fixed-size chunks of `values` and return +/// the results in chunk order. Chunks are cut by `size` alone, so a reduction +/// the caller folds over the returned vector groups its terms the same way at +/// any pool width. +pub(crate) fn for_chunks( + values: &mut [T], + size: usize, + task: impl Fn(usize, &mut [T]) -> Result + Sync, +) -> Result> { + #[cfg(feature = "parallel")] + { + use rayon::prelude::*; + values + .par_chunks_mut(size) + .enumerate() + .map(|(index, chunk)| task(index * size, chunk)) + .collect() + } + #[cfg(not(feature = "parallel"))] + values + .chunks_mut(size) + .enumerate() + .map(|(index, chunk)| task(index * size, chunk)) + .collect() +} + +/// Sort by a **total** order. The result is then the same whatever the pool +/// does, so the parallel sort is safe to use where a result must not vary. +pub(crate) fn sort_total( + values: &mut [T], + order: impl Fn(&T, &T) -> std::cmp::Ordering + Sync, +) { + #[cfg(feature = "parallel")] + { + use rayon::prelude::*; + values.par_sort_unstable_by(order); + } + #[cfg(not(feature = "parallel"))] + values.sort_unstable_by(order); +} + +#[cfg(test)] +mod tests { + use super::balanced_ranges; + + #[test] + fn ranges_cover_every_node_once_in_order_and_balance_weight() { + for parts in 1..9 { + for weights in [ + vec![], + vec![0], + vec![5, 0, 0, 0, 5], + vec![1; 17], + vec![100, 1, 1, 1], + ] { + let ranges = balanced_ranges(&weights, parts); + let covered: Vec = ranges.iter().flat_map(|r| r.clone()).collect(); + assert_eq!(covered, (0..weights.len()).collect::>()); + assert!(ranges.len() <= parts.max(1)); + } + } + // A heavy head does not drag the light tail into its block. + assert_eq!(balanced_ranges(&[100, 1, 1, 1], 2), vec![0..1, 1..4]); + } +} diff --git a/crates/grust-algorithms/src/projection.rs b/crates/grust-algorithms/src/projection.rs index 78a4ee76..5a3398d0 100644 --- a/crates/grust-algorithms/src/projection.rs +++ b/crates/grust-algorithms/src/projection.rs @@ -60,6 +60,7 @@ struct ProjectionData { edges: Buffer, outgoing: Adjacency, reverse: Mutex>>, + incoming: Mutex>>, context: ExecutionContext, _retained: MemoryReservation, } @@ -189,6 +190,7 @@ impl GraphProjection { edges, outgoing, reverse: Mutex::new(None), + incoming: Mutex::new(None), context: context.clone(), _retained: retained, }), @@ -244,6 +246,27 @@ impl GraphProjection { pub(crate) fn outgoing(&self) -> &Adjacency { &self.inner.outgoing } + /// Arcs by target, with weights and edge slots: what `reverse` omits. On an + /// undirected projection every row already mirrors itself, so this is the + /// outgoing adjacency and nothing is built. Otherwise it is built once, + /// admitted and charged, and shared by every kernel on this projection. + pub(crate) fn incoming(&self) -> Result> { + self.inner.context.checkpoint()?; + if self.inner.orientation == Orientation::Undirected { + return Ok(InArcs::Mirror(&self.inner.outgoing)); + } + let mut cached = self + .inner + .incoming + .lock() + .map_err(|_| ProcedureError::ResourceStatePoisoned)?; + if let Some(incoming) = &*cached { + return Ok(InArcs::Built(Arc::clone(incoming))); + } + let incoming = Arc::new(self.inner.outgoing.transposed(&self.inner.context)?); + *cached = Some(Arc::clone(&incoming)); + Ok(InArcs::Built(incoming)) + } pub(crate) fn reverse(&self) -> Result> { self.inner.context.checkpoint()?; let mut cached = self @@ -259,3 +282,87 @@ impl GraphProjection { Ok(reverse) } } + +/// In-arcs of a projection: its own rows when undirected, else a built transpose. +pub(crate) enum InArcs<'a> { + Mirror(&'a Adjacency), + Built(Arc), +} + +impl std::ops::Deref for InArcs<'_> { + type Target = Adjacency; + fn deref(&self) -> &Adjacency { + match self { + Self::Mirror(adjacency) => adjacency, + Self::Built(adjacency) => adjacency, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use grust_procedures::ExecutionLimits; + + #[test] + fn in_arcs_carry_every_arc_with_its_weight_and_are_built_once() { + let context = ExecutionContext::new(ExecutionLimits { + memory_bytes: 1 << 20, + work_units: 1 << 20, + batch_rows: 1024, + deadline: None, + }) + .unwrap(); + let edges = [(0, 1), (2, 1), (0, 1), (1, 1), (3, 0)]; + let weights = vec![1.0, 2.0, 3.0, 4.0, 5.0]; + let build = |orientation| { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..4).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + Some(weights.clone()), + orientation, + &context, + ) + .unwrap() + }; + let arcs = |adjacency: &Adjacency, flip: bool| { + let mut arcs = Vec::new(); + for node in 0..4 { + for arc in adjacency.range(node) { + let other = adjacency.targets.values[arc]; + let (from, to) = if flip { (other, node) } else { (node, other) }; + arcs.push((from, to, adjacency.weight(arc).to_bits())); + } + } + arcs.sort_unstable(); + arcs + }; + for orientation in [Orientation::Outgoing, Orientation::Incoming] { + let graph = build(orientation); + let incoming = graph.incoming().unwrap(); + assert_eq!(arcs(&incoming, true), arcs(graph.outgoing(), false)); + // Rows list their sources in ascending order. + for node in 0..4 { + let row = &incoming.targets.values[incoming.range(node)]; + assert!(row.windows(2).all(|pair| pair[0] <= pair[1])); + } + let held = context.usage().unwrap().live_bytes; + drop(graph.incoming().unwrap()); + assert_eq!(context.usage().unwrap().live_bytes, held, "built once"); + } + let graph = build(Orientation::Undirected); + let held = context.usage().unwrap().live_bytes; + assert!(matches!(graph.incoming().unwrap(), InArcs::Mirror(_))); + assert_eq!(context.usage().unwrap().live_bytes, held, "nothing built"); + } +} diff --git a/crates/grust-algorithms/src/projection/adjacency.rs b/crates/grust-algorithms/src/projection/adjacency.rs index 96a55a92..3ba7aa02 100644 --- a/crates/grust-algorithms/src/projection/adjacency.rs +++ b/crates/grust-algorithms/src/projection/adjacency.rs @@ -135,6 +135,52 @@ impl Adjacency { } } +impl Adjacency { + /// The same arcs grouped by target: row `v` lists the sources of arcs into + /// `v`, each with its weight and original edge slot. Within a row, arcs keep + /// the order of their sources, so the result is a function of this CSR alone. + pub(crate) fn transposed(&self, context: &ExecutionContext) -> Result { + let n = self.offsets.values.len() - 1; + let arcs = self.targets.values.len(); + let mut offsets = Buffer::filled(n + 1, 0usize, context)?; + for chunk in self.targets.values.chunks(1024) { + context.charge_work(chunk.len())?; + for &target in chunk { + offsets.values[target + 1] += 1; + } + } + prefix(&mut offsets.values, context)?; + let mut positions = Buffer::capacity(n, context)?; + positions.values.extend_from_slice(&offsets.values[..n]); + let mut targets = Buffer::filled(arcs, 0usize, context)?; + let mut edge_slots = Buffer::filled(arcs, 0usize, context)?; + let mut weights = self + .weights + .as_ref() + .map(|_| Buffer::filled(arcs, 0.0f64, context)) + .transpose()?; + for source in 0..n { + let range = self.range(source); + context.charge_work(1 + range.len())?; + for arc in range { + let slot = positions.values[self.targets.values[arc]]; + positions.values[self.targets.values[arc]] += 1; + targets.values[slot] = source; + edge_slots.values[slot] = self.edge_slots.values[arc]; + if let (Some(into), Some(from)) = (&mut weights, &self.weights) { + into.values[slot] = from.values[arc]; + } + } + } + Ok(Adjacency { + offsets, + targets, + edge_slots, + weights, + }) + } +} + fn prefix(offsets: &mut [usize], context: &ExecutionContext) -> Result<()> { for index in 1..offsets.len() { context.charge_work(1)?; diff --git a/crates/grust-algorithms/src/random.rs b/crates/grust-algorithms/src/random.rs new file mode 100644 index 00000000..7835b9ee --- /dev/null +++ b/crates/grust-algorithms/src/random.rs @@ -0,0 +1,71 @@ +//! A counter-based generator: every draw is a pure function of +//! `(seed, stream, counter)`, so a parallel schedule cannot change a result and +//! no generator state is shared. SplitMix64's finalizer over a mixed key. + +#[inline] +fn mix(mut z: u64) -> u64 { + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) +} + +/// The `counter`-th 64-bit draw of `stream` under `seed`. +#[inline] +pub(crate) fn draw(seed: u64, stream: u64, counter: u64) -> u64 { + mix(mix(seed ^ 0x9E37_79B9_7F4A_7C15).wrapping_add(mix(stream)) ^ mix(counter.wrapping_add(1))) +} + +/// A uniform index below `bound` (which must be positive), by Lemire's +/// multiply-shift; the bias is below 2^-64 * bound and irrelevant here. +#[inline] +pub(crate) fn below(seed: u64, stream: u64, counter: u64, bound: usize) -> usize { + ((u128::from(draw(seed, stream, counter)) * bound as u128) >> 64) as usize +} + +/// Fisher–Yates over `values`, reproducible from `(seed, stream)`. +pub(crate) fn shuffle(values: &mut [usize], seed: u64, stream: u64) { + for index in (1..values.len()).rev() { + values.swap(index, below(seed, stream, index as u64, index + 1)); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn draws_are_pure_well_spread_and_in_range() { + assert_eq!(draw(7, 1, 42), draw(7, 1, 42)); + assert_ne!(draw(7, 1, 42), draw(7, 1, 43)); + assert_ne!(draw(7, 1, 42), draw(7, 2, 42)); + assert_ne!(draw(7, 1, 42), draw(8, 1, 42)); + let mut buckets = [0usize; 8]; + for counter in 0..80_000 { + let value = below(3, 0, counter, 8); + assert!(value < 8); + buckets[value] += 1; + } + assert!( + buckets + .iter() + .all(|&count| (9_500..10_500).contains(&count)), + "{buckets:?}" + ); + } + + #[test] + fn shuffles_are_permutations_and_reproducible() { + let mut a: Vec = (0..100).collect(); + let mut b = a.clone(); + shuffle(&mut a, 11, 0); + shuffle(&mut b, 11, 0); + assert_eq!(a, b); + assert_ne!(a, (0..100).collect::>()); + let mut sorted = a.clone(); + sorted.sort_unstable(); + assert_eq!(sorted, (0..100).collect::>()); + let mut c: Vec = (0..100).collect(); + shuffle(&mut c, 12, 0); + assert_ne!(a, c); + } +} diff --git a/crates/grust-algorithms/src/shortest.rs b/crates/grust-algorithms/src/shortest.rs index 61fce410..c4a7cf1d 100644 --- a/crates/grust-algorithms/src/shortest.rs +++ b/crates/grust-algorithms/src/shortest.rs @@ -235,13 +235,13 @@ fn run(graph: &GraphProjection, source: &str, paths: bool) -> Result<(Distances, // Each vertex occupies at most one heap entry. Dense inverse positions avoid // unbounded stale entries on graphs with many parallel edges or relaxations. -struct MinHeap { +pub(crate) struct MinHeap { entries: Buffer<(f64, usize)>, positions: Buffer, } impl MinHeap { - fn new(n: usize, context: &ExecutionContext) -> Result { + pub(crate) fn new(n: usize, context: &ExecutionContext) -> Result { Ok(Self { entries: Buffer::capacity(n, context)?, positions: Buffer::filled(n, usize::MAX, context)?, @@ -260,7 +260,12 @@ impl MinHeap { self.positions.values[self.entries.values[right].1] = right; } - fn improve(&mut self, node: usize, cost: f64, context: &ExecutionContext) -> Result<()> { + pub(crate) fn improve( + &mut self, + node: usize, + cost: f64, + context: &ExecutionContext, + ) -> Result<()> { let mut index = self.positions.values[node]; if index == usize::MAX { index = self.entries.values.len(); @@ -281,7 +286,7 @@ impl MinHeap { Ok(()) } - fn pop(&mut self, context: &ExecutionContext) -> Result> { + pub(crate) fn pop(&mut self, context: &ExecutionContext) -> Result> { if self.entries.values.is_empty() { return Ok(None); } diff --git a/crates/grust-algorithms/src/similarity.rs b/crates/grust-algorithms/src/similarity.rs new file mode 100644 index 00000000..5c43aaa3 --- /dev/null +++ b/crates/grust-algorithms/src/similarity.rs @@ -0,0 +1,424 @@ +//! Node similarity: nodes are alike when they point at the same things. + +use crate::{ + AlgorithmError, GraphProjection, Result, + buffer::Buffer, + meter::Meter, + parallel, + table::{NodeColumn, NodeTable}, +}; +use grust_procedures::ExecutionContext; + +/// How two neighbour sets are compared. `S` sums, over shared neighbours, the +/// smaller of the two weights (Jaccard, overlap) or their product (cosine); +/// without weights every weight is one and `S` is the size of the intersection. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum SimilarityMetric { + /// `S / (W(a) + W(b) - S)`: intersection over union. + #[default] + Jaccard, + /// `S / min(W(a), W(b))`: one when one set contains the other. + Overlap, + /// `S / (|a| |b|)`, the cosine of the two weight vectors. + Cosine, +} + +/// Node similarity controls. +#[derive(Clone, Copy, Debug)] +pub struct NodeSimilarityOptions { + /// The comparison. + pub metric: SimilarityMetric, + /// Keep, for each node, its this-many most similar nodes. + pub top_k: usize, + /// Then keep the this-many most similar rows overall; zero keeps all. + pub top_n: usize, + /// Drop pairs below this. Pairs with no shared neighbour are never emitted. + pub similarity_cutoff: f64, + /// Leave out nodes with fewer distinct neighbours than this. + pub degree_cutoff: usize, + /// Leave out nodes with more distinct neighbours than this. + pub upper_degree_cutoff: Option, +} + +impl Default for NodeSimilarityOptions { + fn default() -> Self { + Self { + metric: SimilarityMetric::Jaccard, + top_k: 10, + top_n: 0, + similarity_cutoff: 0.0, + degree_cutoff: 1, + upper_degree_cutoff: None, + } + } +} + +/// Similar pairs, one row per kept `(node1, node2)`. +pub struct NodeSimilarity { + graph: GraphProjection, + first: Buffer, + second: Buffer, + similarity: Buffer, +} + +impl NodeSimilarity { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + /// `node1` per row, as a projection row. + pub fn first(&self) -> &[usize] { + &self.first.values + } + /// `node2` per row, as a projection row. + pub fn second(&self) -> &[usize] { + &self.second.values + } + /// Similarity per row, in `(0, 1]`. + pub fn similarity(&self) -> &[f64] { + &self.similarity.values + } + /// Rows of `node1`, `node2`, `similarity`. + pub fn into_table(self) -> Result { + NodeTable::keyed(&self.graph, "node1", self.first)? + .column("node2", NodeColumn::Node(self.second))? + .column("similarity", NodeColumn::Number(self.similarity)) + } +} + +/// Compare every two nodes by what their arcs point at. +/// +/// **Sets, not multisets.** A node's neighbours are the *distinct* nodes its +/// arcs reach, itself excluded. Parallel edges collapse into one neighbour: +/// without weights it simply is in the set, and with weights it carries their +/// sum. The degree-style kernels here count each parallel edge instead. +/// +/// **Rows.** A pair sharing no neighbour, or scoring zero, is never emitted. +/// Each kept pair appears from both sides, `(a, b)` and `(b, a)`, because +/// `top_k` is per `node1`: `b` may be among `a`'s closest without the reverse. +/// Rows come ordered by `node1`, then by similarity descending, then `node2`; +/// with `top_n`, by similarity descending, then `node1`, then `node2`. +/// +/// **Cost** is the number of `(a, shared neighbour, b)` triples, which a +/// popular neighbour makes quadratic in its popularity. Every triple is charged, +/// so the work budget is the guard; `upper_degree_cutoff` does not help there, +/// as it bounds a node's own neighbours, not how many point at it. +/// +/// Rows and charged work are identical at any pool width. +pub fn node_similarity( + graph: &GraphProjection, + options: NodeSimilarityOptions, +) -> Result { + let context = graph.execution(); + context.checkpoint()?; + let invalid = |message: &str| Err(AlgorithmError::InvalidArguments(message.into())); + if options.top_k == 0 { + return invalid("topK must be positive"); + } + if !(0.0..=1.0).contains(&options.similarity_cutoff) { + return invalid("similarityCutoff must be between 0 and 1"); + } + if options.degree_cutoff == 0 { + return invalid("degreeCutoff must be positive"); + } + if options + .upper_degree_cutoff + .is_some_and(|upper| upper < options.degree_cutoff) + { + return invalid("upperDegreeCutoff must not be below degreeCutoff"); + } + + let sets = Sets::build(graph, options, context)?; + let n = sets.nodes(); + let blocks = n.div_ceil(BLOCK); + let mut chunks: Buffer> = Buffer::capacity(blocks, context)?; + let mut total = 0usize; + parallel::ordered_blocks( + blocks, + |block| { + let first = block * BLOCK; + let last = (first + BLOCK).min(n); + let mut workspace = Workspace::new(n, options.top_k, context)?; + for node in first..last { + workspace.rank(&sets, node, options, context)?; + } + // Keep exactly what was found; the scratch goes back. + let mut kept = Buffer::capacity(workspace.rows.values.len(), context)?; + kept.values.extend_from_slice(&workspace.rows.values); + Ok(kept) + }, + |_, kept: Buffer| { + total += kept.values.len(); + chunks.values.push(kept); + Ok(()) + }, + )?; + + let mut meter = Meter::new(context); + let mut rows = Buffer::capacity(total, context)?; + for chunk in chunks.values.drain(..) { + meter.tick(chunk.values.len())?; + rows.values.extend_from_slice(&chunk.values); + } + drop(chunks); + if options.top_n > 0 { + meter.tick(total.saturating_mul(total.max(2).ilog2() as usize))?; + rows.values.sort_unstable_by(|a, b| { + b.similarity + .total_cmp(&a.similarity) + .then(a.first.cmp(&b.first)) + .then(a.second.cmp(&b.second)) + }); + rows.values.truncate(options.top_n); + } + let count = rows.values.len(); + meter.tick(count)?; + let mut first = Buffer::capacity(count, context)?; + let mut second = Buffer::capacity(count, context)?; + let mut similarity = Buffer::capacity(count, context)?; + for row in &rows.values { + first.values.push(row.first); + second.values.push(row.second); + similarity.values.push(row.similarity); + } + meter.flush()?; + Ok(NodeSimilarity { + graph: graph.clone(), + first, + second, + similarity, + }) +} + +/// Nodes per block; fixed, so nothing depends on the pool width. +const BLOCK: usize = 64; + +#[derive(Clone, Copy)] +struct Row { + first: usize, + second: usize, + similarity: f64, +} + +/// Distinct neighbours per node, and per neighbour the nodes that have it. +struct Sets { + offsets: Buffer, + /// (neighbour, weight), ascending by neighbour within a row. + entries: Buffer<(usize, f64)>, + member_offsets: Buffer, + /// (node, weight), ascending by node within a row. + members: Buffer<(usize, f64)>, + /// `W`: the weights' sum, or for cosine their Euclidean norm. + measure: Buffer, + /// Whether the degree cutoffs admit the node. + admitted: Buffer, +} + +impl Sets { + fn nodes(&self) -> usize { + self.measure.values.len() + } + + fn build( + graph: &GraphProjection, + options: NodeSimilarityOptions, + context: &ExecutionContext, + ) -> Result { + let adjacency = graph.outgoing(); + let weighted = adjacency.weights.is_some(); + let n = graph.node_count(); + let mut meter = Meter::new(context); + let mut offsets = Buffer::capacity(n + 1, context)?; + let mut entries: Buffer<(usize, f64)> = + Buffer::capacity(adjacency.targets.values.len(), context)?; + let mut measure = Buffer::capacity(n, context)?; + let mut admitted = Buffer::capacity(n, context)?; + let mut member_offsets = Buffer::filled(n + 1, 0usize, context)?; + offsets.values.push(0); + for node in 0..n { + let range = adjacency.range(node); + let degree = range.len(); + meter.tick(1 + degree.saturating_mul(degree.max(2).ilog2() as usize))?; + let start = entries.values.len(); + for arc in range { + let target = adjacency.targets.values[arc]; + if target != node { + entries.values.push((target, adjacency.weight(arc))); + } + } + // Ordering equal neighbours by weight fixes the order their sum is taken in. + entries.values[start..] + .sort_unstable_by(|a, b| a.0.cmp(&b.0).then(a.1.total_cmp(&b.1))); + let mut write = start; + for read in start..entries.values.len() { + let entry = entries.values[read]; + if write > start && entries.values[write - 1].0 == entry.0 { + // Without weights a neighbour is in the set or not. + if weighted { + entries.values[write - 1].1 += entry.1; + } + } else { + entries.values[write] = entry; + write += 1; + } + } + entries.values.truncate(write); + let distinct = write - start; + admitted.values.push( + distinct >= options.degree_cutoff + && options + .upper_degree_cutoff + .is_none_or(|upper| distinct <= upper), + ); + let row = &entries.values[start..]; + measure.values.push(match options.metric { + SimilarityMetric::Cosine => row.iter().map(|e| e.1 * e.1).sum::().sqrt(), + _ => row.iter().map(|e| e.1).sum(), + }); + if admitted.values[node] { + for &(target, _) in row { + member_offsets.values[target + 1] += 1; + } + } + offsets.values.push(write); + } + // Only admitted nodes are anyone's candidate, so only they are members. + for index in 1..=n { + member_offsets.values[index] += member_offsets.values[index - 1]; + } + meter.tick(n + entries.values.len())?; + let mut members = Buffer::filled(member_offsets.values[n], (0usize, 0.0f64), context)?; + let mut position = Buffer::capacity(n, context)?; + position + .values + .extend_from_slice(&member_offsets.values[..n]); + for node in 0..n { + if !admitted.values[node] { + continue; + } + for &(target, weight) in &entries.values[offsets.values[node]..offsets.values[node + 1]] + { + members.values[position.values[target]] = (node, weight); + position.values[target] += 1; + } + } + meter.flush()?; + Ok(Self { + offsets, + entries, + member_offsets, + members, + measure, + admitted, + }) + } +} + +struct Workspace { + /// `S` per candidate for the node in hand. + shared: Buffer, + seen: Buffer, + candidates: Buffer, + ranked: Buffer, + rows: Buffer, +} + +impl Workspace { + fn new(n: usize, top_k: usize, context: &ExecutionContext) -> Result { + Ok(Self { + shared: Buffer::filled(n, 0.0, context)?, + seen: Buffer::filled(n, false, context)?, + candidates: Buffer::capacity(n, context)?, + ranked: Buffer::capacity(n, context)?, + // Enough for every node of a block to keep `top_k` partners, up to + // 1024 each; `rank` admits more before it is needed. + rows: Buffer::capacity(BLOCK * top_k.min(n.saturating_sub(1)).min(1024), context)?, + }) + } + + fn rank( + &mut self, + sets: &Sets, + node: usize, + options: NodeSimilarityOptions, + context: &ExecutionContext, + ) -> Result<()> { + if !sets.admitted.values[node] { + return Ok(()); + } + let mut meter = Meter::new(context); + let cosine = options.metric == SimilarityMetric::Cosine; + for &(target, weight) in + &sets.entries.values[sets.offsets.values[node]..sets.offsets.values[node + 1]] + { + let members = &sets.members.values + [sets.member_offsets.values[target]..sets.member_offsets.values[target + 1]]; + meter.tick(1 + members.len())?; + for &(other, other_weight) in members { + if other == node { + continue; + } + if !self.seen.values[other] { + self.seen.values[other] = true; + self.candidates.values.push(other); + } + self.shared.values[other] += if cosine { + weight * other_weight + } else { + weight.min(other_weight) + }; + } + } + let own = sets.measure.values[node]; + for &other in &self.candidates.values { + let shared = self.shared.values[other]; + let theirs = sets.measure.values[other]; + self.shared.values[other] = 0.0; + self.seen.values[other] = false; + let denominator = match options.metric { + SimilarityMetric::Jaccard => own + theirs - shared, + SimilarityMetric::Overlap => own.min(theirs), + SimilarityMetric::Cosine => own * theirs, + }; + let similarity = shared / denominator; + // Zero weights can leave nothing shared, or nothing to divide by. + if similarity > 0.0 && similarity.is_finite() && similarity >= options.similarity_cutoff + { + self.ranked.values.push(Row { + first: node, + second: other, + similarity, + }); + } + } + let found = self.ranked.values.len(); + meter.tick( + self.candidates.values.len() + found.saturating_mul(found.max(2).ilog2() as usize), + )?; + self.candidates.values.clear(); + let order = |a: &Row, b: &Row| { + b.similarity + .total_cmp(&a.similarity) + .then(a.second.cmp(&b.second)) + }; + if found > options.top_k { + self.ranked + .values + .select_nth_unstable_by(options.top_k, order); + self.ranked.values.truncate(options.top_k); + } + self.ranked.values.sort_unstable_by(order); + if self.rows.values.len() + self.ranked.values.len() > self.rows.values.capacity() { + // Only when `top_k` exceeds 1024: admit the larger scratch first. + let mut larger = Buffer::capacity( + (self.rows.values.len() + self.ranked.values.len()).saturating_mul(2), + context, + )?; + larger.values.extend_from_slice(&self.rows.values); + self.rows = larger; + } + self.rows.values.extend_from_slice(&self.ranked.values); + self.ranked.values.clear(); + meter.flush() + } +} diff --git a/crates/grust-algorithms/src/spanning.rs b/crates/grust-algorithms/src/spanning.rs new file mode 100644 index 00000000..17834da8 --- /dev/null +++ b/crates/grust-algorithms/src/spanning.rs @@ -0,0 +1,196 @@ +//! Minimum or maximum spanning forest by Kruskal's algorithm. + +use crate::{ + AlgorithmError, GraphProjection, Orientation, Result, + buffer::Buffer, + meter::Meter, + parallel, + table::{NodeColumn, NodeTable, TableScalar}, +}; + +/// Which forest to build. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub enum SpanningObjective { + /// Least total weight. + #[default] + Minimum, + /// Greatest total weight. + Maximum, +} + +/// Spanning forest controls. +#[derive(Clone, Copy, Debug, Default)] +pub struct SpanningTreeOptions<'a> { + /// Minimum or maximum. + pub objective: SpanningObjective, + /// Return only the tree that spans this node's component. + pub source: Option<&'a str>, +} + +/// The chosen edges. +pub struct SpanningForest { + graph: GraphProjection, + edges: Buffer, + weights: Buffer, + total: f64, +} + +impl SpanningForest { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + /// Chosen edges, as ascending indices into [`GraphProjection::edges`]. + pub fn edges(&self) -> &[usize] { + &self.edges.values + } + /// Weight of each chosen edge; one when the projection has no weights. + pub fn weights(&self) -> &[f64] { + &self.weights.values + } + /// Sum of the chosen weights, added in the order of [`Self::edges`]. + pub fn total_weight(&self) -> f64 { + self.total + } + /// Rows of `sourceNodeId`, `targetNodeId`, `edgeOrdinal`, `weight`, then the + /// `totalWeight` scalar. + pub fn into_table(self) -> Result { + let context = self.graph.execution(); + let count = self.edges.values.len(); + context.charge_work(count)?; + let mut sources = Buffer::capacity(count, context)?; + let mut targets = Buffer::capacity(count, context)?; + let mut ordinals = Buffer::capacity(count, context)?; + for &slot in &self.edges.values { + let edge = &self.graph.edges()[slot]; + sources.values.push(edge.source); + targets.values.push(edge.target); + ordinals.values.push( + i64::try_from(edge.ordinal) + .map_err(|_| AlgorithmError::Numerical("edge ordinal exceeds Int64".into()))?, + ); + } + Ok(NodeTable::keyed(&self.graph, "sourceNodeId", sources)? + .column("targetNodeId", NodeColumn::Node(targets))? + .column("edgeOrdinal", NodeColumn::Integer(ordinals))? + .column("weight", NodeColumn::Number(self.weights))? + .scalar("totalWeight", TableScalar::Number(self.total))) + } +} + +/// Choose, from an undirected projection, a forest that spans every component +/// with the least (or greatest) total weight. +/// +/// **Ties** go to the smaller edge ordinal. Edges are therefore in a total +/// order, and the forest is *the* greedy one for that order rather than one of +/// several equally light ones: the same projection always gives the same edges. +/// Of parallel edges the best is taken; a self-loop never is. Without weights +/// every edge weighs one and the result is the spanning forest of smallest +/// ordinals. +/// +/// **`source`** keeps only the tree spanning that node's component, which is +/// what a Prim run from that node would return. +/// +/// Kruskal with union by size and path halving. Only the sort runs on the +/// caller's rayon pool; its order is total, so the result does not depend on it. +/// Defined on undirected graphs; project with `orientation: "undirected"`. +pub fn spanning_tree( + graph: &GraphProjection, + options: SpanningTreeOptions<'_>, +) -> Result { + let context = graph.execution(); + context.checkpoint()?; + if graph.orientation() != Orientation::Undirected { + return Err(AlgorithmError::InvalidArguments( + "a spanning tree is defined on undirected graphs; project with orientation \"undirected\"".into(), + )); + } + let source = options.source.map(|id| graph.source(id)).transpose()?; + let n = graph.node_count(); + let edges = graph.edges(); + let adjacency = graph.outgoing(); + let mut meter = Meter::new(context); + + // Weight per edge, read off either of its arcs. + let mut weight = Buffer::filled(edges.len(), 1.0f64, context)?; + if adjacency.weights.is_some() { + for node in 0..n { + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + for arc in range { + weight.values[adjacency.edge_slots.values[arc]] = adjacency.weight(arc); + } + } + } + + let mut order = Buffer::capacity(edges.len(), context)?; + order.values.extend(0..edges.len()); + let count = edges.len(); + meter.tick(count.saturating_mul(count.max(2).ilog2() as usize))?; + meter.flush()?; + let maximum = options.objective == SpanningObjective::Maximum; + parallel::sort_total(&mut order.values, |&a: &usize, &b: &usize| { + let by_weight = weight.values[a].total_cmp(&weight.values[b]); + let by_weight = if maximum { + by_weight.reverse() + } else { + by_weight + }; + by_weight.then(edges[a].ordinal.cmp(&edges[b].ordinal)) + }); + + let mut parent = Buffer::capacity(n, context)?; + parent.values.extend(0..n); + let mut size = Buffer::filled(n, 1usize, context)?; + let find = |parent: &mut [usize], mut node: usize| { + while parent[node] != node { + parent[node] = parent[parent[node]]; + node = parent[node]; + } + node + }; + let mut chosen = Buffer::filled(edges.len(), false, context)?; + let mut remaining = n.saturating_sub(1); + for &slot in &order.values { + if remaining == 0 { + break; + } + meter.tick(1)?; + let a = find(&mut parent.values, edges[slot].source); + let b = find(&mut parent.values, edges[slot].target); + if a == b { + continue; + } + let (small, large) = if size.values[a] < size.values[b] { + (a, b) + } else { + (b, a) + }; + parent.values[small] = large; + size.values[large] += size.values[small]; + chosen.values[slot] = true; + remaining -= 1; + } + + let wanted = source.map(|node| find(&mut parent.values, node)); + meter.tick(edges.len())?; + let mut kept = Buffer::capacity(n.saturating_sub(1).min(edges.len()), context)?; + let mut weights = Buffer::capacity(n.saturating_sub(1).min(edges.len()), context)?; + let mut total = 0.0; + for (slot, edge) in edges.iter().enumerate() { + if chosen.values[slot] + && wanted.is_none_or(|root| find(&mut parent.values, edge.source) == root) + { + kept.values.push(slot); + weights.values.push(weight.values[slot]); + total += weight.values[slot]; + } + } + meter.flush()?; + Ok(SpanningForest { + graph: graph.clone(), + edges: kept, + weights, + total, + }) +} diff --git a/crates/grust-algorithms/src/spectral.rs b/crates/grust-algorithms/src/spectral.rs new file mode 100644 index 00000000..4f57c2ea --- /dev/null +++ b/crates/grust-algorithms/src/spectral.rs @@ -0,0 +1,331 @@ +//! Eigenvector centrality, Katz centrality and HITS: power iterations that pull +//! each node's next value from its neighbours' current ones. + +use crate::{ + AlgorithmError, GraphProjection, Result, + buffer::Buffer, + meter::Meter, + parallel, + projection::Adjacency, + table::{NodeColumn, NodeTable, TableScalar}, +}; +use grust_procedures::ExecutionContext; + +/// Stopping rule shared by the three iterations. +#[derive(Clone, Copy, Debug)] +pub struct IterationOptions { + /// Stop when the L1 change between iterations is at most this. + pub tolerance: f64, + /// Positive maximum number of iterations. + pub max_iterations: usize, +} + +impl Default for IterationOptions { + fn default() -> Self { + Self { + tolerance: 1e-8, + max_iterations: 1000, + } + } +} + +/// Katz controls. +#[derive(Clone, Copy, Debug)] +pub struct KatzOptions { + /// Attenuation per hop; positive. The series converges only below + /// `1 / λmax`. `1 / (largest in-strength)` is below that, so it is a safe + /// choice, not the largest one. + pub alpha: f64, + /// What every node starts from; positive. + pub beta: f64, + /// Scale the result to unit Euclidean length. + pub normalized: bool, + /// When to stop. + pub iteration: IterationOptions, +} + +impl Default for KatzOptions { + fn default() -> Self { + Self { + alpha: 0.1, + beta: 1.0, + normalized: false, + iteration: IterationOptions::default(), + } + } +} + +/// Scores with convergence evidence. Reaching the iteration limit returns +/// `converged() == false`; convergence is never asserted silently. +pub struct IteratedScores { + graph: GraphProjection, + columns: Vec<(&'static str, Buffer)>, + iterations: usize, + residual: f64, + converged: bool, +} + +impl IteratedScores { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + /// A score column by name: `score`, or `hub` and `authority` for HITS. + pub fn values(&self, name: &str) -> Option<&[f64]> { + self.columns + .iter() + .find_map(|(column, values)| (*column == name).then_some(values.values.as_slice())) + } + /// Completed iterations. + pub fn iterations(&self) -> usize { + self.iterations + } + /// Last L1 change. + pub fn residual(&self) -> f64 { + self.residual + } + /// Whether the tolerance was reached. + pub fn converged(&self) -> bool { + self.converged + } + /// The score columns, then `iterations`, `converged` and `residual`. + pub fn into_table(self) -> Result { + let iterations = i64::try_from(self.iterations) + .map_err(|_| AlgorithmError::Numerical("iteration count exceeds Int64".into()))?; + let mut table = NodeTable::new(&self.graph); + for (name, values) in self.columns { + table = table.column(name, NodeColumn::Number(values))?; + } + Ok(table + .scalar("iterations", TableScalar::Integer(iterations)) + .scalar("converged", TableScalar::Boolean(self.converged)) + .scalar("residual", TableScalar::Number(self.residual))) + } +} + +/// Nodes per chunk. Fixed, so every sum groups its terms the same way whatever +/// the pool, and results are identical at any width. +const CHUNK: usize = 4096; + +fn validate(options: IterationOptions) -> Result<()> { + if !options.tolerance.is_finite() || options.tolerance < 0.0 || options.max_iterations == 0 { + return Err(AlgorithmError::InvalidArguments( + "tolerance must be finite and nonnegative, and maxIterations positive".into(), + )); + } + Ok(()) +} + +/// `into[v] = keep * from[v] + beta + alpha * Σ weight * from[u]` over the arcs +/// `arcs` lists for `v`. Returns the sum of squares of `into`. +fn pull( + arcs: &Adjacency, + from: &[f64], + into: &mut [f64], + (keep, beta, alpha): (f64, f64, f64), + context: &ExecutionContext, +) -> Result { + let sums = parallel::for_chunks(into, CHUNK, |first, chunk| { + let mut meter = Meter::new(context); + let mut squares = 0.0; + for (offset, value) in chunk.iter_mut().enumerate() { + let node = first + offset; + let range = arcs.range(node); + meter.tick(1 + range.len())?; + let mut sum = 0.0; + for arc in range { + sum += arcs.weight(arc) * from[arcs.targets.values[arc]]; + } + *value = keep * from[node] + beta + alpha * sum; + squares += *value * *value; + } + meter.flush()?; + Ok(squares) + })?; + Ok(sums.into_iter().sum()) +} + +/// Divide `values` by `divisor` and return their L1 distance from `previous`. +fn settle( + values: &mut [f64], + previous: &[f64], + divisor: f64, + context: &ExecutionContext, +) -> Result { + let sums = parallel::for_chunks(values, CHUNK, |first, chunk| { + context.charge_work(chunk.len())?; + let mut change = 0.0; + for (offset, value) in chunk.iter_mut().enumerate() { + *value /= divisor; + change += (*value - previous[first + offset]).abs(); + } + Ok(change) + })?; + Ok(sums.into_iter().sum()) +} + +fn nonfinite(what: &str) -> AlgorithmError { + AlgorithmError::Numerical(format!("{what} left the finite range")) +} + +/// Eigenvector centrality: a node is important when important nodes point at it. +/// +/// The score is the principal eigenvector of the adjacency matrix, of unit +/// Euclidean length and nonnegative, with influence flowing **along** the +/// projection's arcs. It iterates `x ← (A + I)x`, not `x ← Ax`: the shift +/// leaves the eigenvectors alone and removes the oscillation that stops plain +/// power iteration from converging on a bipartite graph. It starts uniform. +/// +/// Where the principal eigenvector is not unique (a disconnected graph) the +/// result is *an* eigenvector, the one reached from the uniform start. On a +/// graph with no cycle there is nothing to be principal; the iteration drifts +/// toward the sinks and usually stops at the limit with `converged == false`. +/// +/// Identical at any pool width. +pub fn eigenvector(graph: &GraphProjection, options: IterationOptions) -> Result { + let context = graph.execution(); + context.checkpoint()?; + validate(options)?; + let n = graph.node_count(); + let incoming = graph.incoming()?; + let mut current = Buffer::filled(n, 1.0 / (n.max(1) as f64).sqrt(), context)?; + let mut next = Buffer::filled(n, 0.0, context)?; + let (mut iterations, mut residual, mut converged) = (0, 0.0, n == 0); + while !converged && iterations < options.max_iterations { + let squares = pull( + &incoming, + ¤t.values, + &mut next.values, + (1.0, 0.0, 1.0), + context, + )?; + if !squares.is_finite() { + return Err(nonfinite("eigenvector scores")); + } + residual = settle(&mut next.values, ¤t.values, squares.sqrt(), context)?; + std::mem::swap(&mut current, &mut next); + iterations += 1; + converged = residual <= options.tolerance; + } + Ok(IteratedScores { + graph: graph.clone(), + columns: vec![("score", current)], + iterations, + residual, + converged, + }) +} + +/// Katz centrality: `x = beta + alpha * Σ x[u]` over the nodes `u` pointing at +/// each node, so every walk into a node counts, attenuated by `alpha` per hop. +/// +/// The series converges only for `alpha < 1 / λmax`. That is not checked up +/// front, because the cheap bound `1 / (largest in-strength)` would refuse many +/// valid values. Instead a run that does not settle returns +/// `converged == false`, and one whose scores overflow fails and says why. +/// +/// Identical at any pool width. +pub fn katz(graph: &GraphProjection, options: KatzOptions) -> Result { + let context = graph.execution(); + context.checkpoint()?; + validate(options.iteration)?; + let positive = |value: f64| value.is_finite() && value > 0.0; + if !positive(options.alpha) || !positive(options.beta) { + return Err(AlgorithmError::InvalidArguments( + "alpha and beta must be finite and positive".into(), + )); + } + let n = graph.node_count(); + let incoming = graph.incoming()?; + let mut current = Buffer::filled(n, 0.0, context)?; + let mut next = Buffer::filled(n, 0.0, context)?; + let (mut iterations, mut residual, mut converged) = (0, 0.0, n == 0); + let mut squares = 0.0; + while !converged && iterations < options.iteration.max_iterations { + squares = pull( + &incoming, + ¤t.values, + &mut next.values, + (0.0, options.beta, options.alpha), + context, + )?; + if !squares.is_finite() { + return Err(AlgorithmError::Numerical( + "Katz scores left the finite range: alpha must be below 1/λmax, and any alpha below 1/(largest in-strength) is".into(), + )); + } + residual = settle(&mut next.values, ¤t.values, 1.0, context)?; + std::mem::swap(&mut current, &mut next); + iterations += 1; + converged = residual <= options.iteration.tolerance; + } + if options.normalized && squares > 0.0 { + settle(&mut current.values, &next.values, squares.sqrt(), context)?; + } + Ok(IteratedScores { + graph: graph.clone(), + columns: vec![("score", current)], + iterations, + residual, + converged, + }) +} + +/// HITS: a good **hub** points at good authorities, a good **authority** is +/// pointed at by good hubs. Both have unit Euclidean length; a graph with no +/// arcs scores zero everywhere. The residual is the L1 change of both together. +/// +/// Identical at any pool width. +pub fn hits(graph: &GraphProjection, options: IterationOptions) -> Result { + let context = graph.execution(); + context.checkpoint()?; + validate(options)?; + let n = graph.node_count(); + let incoming = graph.incoming()?; + let outgoing = graph.outgoing(); + let mut hubs = Buffer::filled(n, 1.0 / (n.max(1) as f64).sqrt(), context)?; + let mut authorities = Buffer::filled(n, 0.0, context)?; + let mut scratch = Buffer::filled(n, 0.0, context)?; + let (mut iterations, mut residual, mut converged) = (0, 0.0, n == 0); + while !converged && iterations < options.max_iterations { + // Authorities from hubs, over in-arcs; then hubs from the new + // authorities, over out-arcs. A zero vector stays zero, undivided. + let squares = pull( + &incoming, + &hubs.values, + &mut scratch.values, + (0.0, 0.0, 1.0), + context, + )?; + if !squares.is_finite() { + return Err(nonfinite("authority scores")); + } + let divisor = if squares > 0.0 { squares.sqrt() } else { 1.0 }; + residual = settle(&mut scratch.values, &authorities.values, divisor, context)?; + std::mem::swap(&mut authorities, &mut scratch); + + let squares = pull( + outgoing, + &authorities.values, + &mut scratch.values, + (0.0, 0.0, 1.0), + context, + )?; + if !squares.is_finite() { + return Err(nonfinite("hub scores")); + } + let divisor = if squares > 0.0 { squares.sqrt() } else { 1.0 }; + residual += settle(&mut scratch.values, &hubs.values, divisor, context)?; + std::mem::swap(&mut hubs, &mut scratch); + + iterations += 1; + converged = residual <= options.tolerance; + } + Ok(IteratedScores { + graph: graph.clone(), + columns: vec![("hub", hubs), ("authority", authorities)], + iterations, + residual, + converged, + }) +} diff --git a/crates/grust-algorithms/src/table.rs b/crates/grust-algorithms/src/table.rs new file mode 100644 index 00000000..6190feff --- /dev/null +++ b/crates/grust-algorithms/src/table.rs @@ -0,0 +1,282 @@ +//! Node-aligned result tables: one row per projected node, typed named columns. +//! +//! Most graph algorithms answer "a value per node" — a core number, a triangle +//! count, a centrality, a community. Each used to need its own result type and +//! its own arm in the Arrow and row adapters. A kernel now hands its buffers to +//! a `NodeTable`, and both adapters are written once. Buffers move in; nothing +//! is copied, and their memory admission travels with them. + +use crate::{AlgorithmError, GraphProjection, Result, buffer::Buffer}; +use grust_core::NodeId; + +/// One value per projected node, in projection row order. +// The catalog's kernels arrive one at a time; some column kinds precede their first user. +#[allow(dead_code)] +pub(crate) enum NodeColumn { + /// Signed integers; never null. + Integer(Buffer), + /// Finite doubles; never null. + Number(Buffer), + /// Doubles where NaN means "no value" and is emitted as null. + OptionalNumber(Buffer), + /// A projection row, emitted as that node's external id. + Node(Buffer), + /// Booleans; never null. + Boolean(Buffer), + /// A fixed number of single-precision floats per row, stored row by row. + Vector { + /// `rows * dimension` values. + values: Buffer, + /// Floats per row; positive. + dimension: usize, + }, +} + +impl NodeColumn { + fn len(&self) -> usize { + match self { + Self::Integer(values) => values.values.len(), + Self::Number(values) | Self::OptionalNumber(values) => values.values.len(), + Self::Node(values) => values.values.len(), + Self::Boolean(values) => values.values.len(), + Self::Vector { values, dimension } => values.values.len() / (*dimension).max(1), + } + } +} + +/// A whole-result value, repeated on every row the way `pagerank` repeats its +/// iteration count: tabular consumers have nowhere else to read it. +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum TableScalar { + /// Signed integer. + Integer(i64), + /// Double. + Number(f64), + /// Boolean. + Boolean(bool), +} + +/// One cell, borrowed from the table. +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum TableValue<'a> { + /// Signed integer. + Integer(i64), + /// Finite double. + Number(f64), + /// No value for this node. + Null, + /// Another projected node, by external id. + Node(&'a NodeId), + /// Boolean. + Boolean(bool), + /// A fixed-length vector. + Vector(&'a [f32]), +} + +/// The declared type of a column, for adapters that build typed output. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum TableType { + /// Signed 64-bit integer. + Integer, + /// Double, possibly null. + Number, + /// External node id. + Node, + /// Boolean. + Boolean, + /// Fixed-length list of this many single-precision floats. + Vector(usize), +} + +/// Typed per-node results retaining projection identity and memory admission. +/// Columns come first, in the order the kernel declared them, then scalars. +/// +/// A table is usually one row per node, keyed by that node as `nodeId`. A +/// kernel that answers with a list of nodes or of node pairs keys its rows +/// itself: any number of rows, each led by the node the kernel names. +pub struct NodeTable { + graph: GraphProjection, + key: Option<(&'static str, Buffer)>, + columns: Vec<(&'static str, NodeColumn)>, + scalars: Vec<(&'static str, TableScalar)>, +} + +impl NodeTable { + pub(crate) fn new(graph: &GraphProjection) -> Self { + Self { + graph: graph.clone(), + key: None, + columns: Vec::new(), + scalars: Vec::new(), + } + } + + /// A table with one row per entry of `rows`, led by that node under `name`. + pub(crate) fn keyed( + graph: &GraphProjection, + name: &'static str, + rows: Buffer, + ) -> Result { + if rows.values.iter().any(|&row| row >= graph.node_count()) { + return Err(AlgorithmError::OutputContract(format!( + "key `{name}` refers to a node outside the projection" + ))); + } + let mut table = Self::new(graph); + table.key = Some((name, rows)); + Ok(table) + } + + pub(crate) fn column(mut self, name: &'static str, column: NodeColumn) -> Result { + let n = self.graph.node_count(); + let rows = self.rows(); + if column.len() != rows { + return Err(AlgorithmError::OutputContract(format!( + "column `{name}` has {} rows in a table of {rows}", + column.len() + ))); + } + if let NodeColumn::Vector { values, dimension } = &column + && (*dimension == 0 || values.values.len() != rows * dimension) + { + return Err(AlgorithmError::OutputContract(format!( + "column `{name}` is not {rows} rows of {dimension} values" + ))); + } + if let NodeColumn::Node(rows) = &column + && rows.values.iter().any(|&row| row >= n) + { + return Err(AlgorithmError::OutputContract(format!( + "column `{name}` refers to a node outside the projection" + ))); + } + self.columns.push((name, column)); + Ok(self) + } + + pub(crate) fn scalar(mut self, name: &'static str, value: TableScalar) -> Self { + self.scalars.push((name, value)); + self + } + + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + + /// Rows: one per projected node, unless the kernel keyed the table itself. + pub fn rows(&self) -> usize { + self.key + .as_ref() + .map_or(self.graph.node_count(), |(_, rows)| rows.values.len()) + } + + /// Name of the leading node column: `nodeId` unless the kernel chose one. + pub fn key_name(&self) -> &'static str { + self.key.as_ref().map_or("nodeId", |(name, _)| name) + } + + /// The projection row of the node that leads `row`. + pub fn key(&self, row: usize) -> usize { + self.key.as_ref().map_or(row, |(_, rows)| rows.values[row]) + } + + /// Columns after the key, per-node columns first and then scalars. + pub fn width(&self) -> usize { + self.columns.len() + self.scalars.len() + } + + /// Name and declared type of column `index`. + pub fn field(&self, index: usize) -> (&'static str, TableType) { + if let Some((name, column)) = self.columns.get(index) { + let kind = match column { + NodeColumn::Integer(_) => TableType::Integer, + NodeColumn::Number(_) | NodeColumn::OptionalNumber(_) => TableType::Number, + NodeColumn::Node(_) => TableType::Node, + NodeColumn::Boolean(_) => TableType::Boolean, + NodeColumn::Vector { dimension, .. } => TableType::Vector(*dimension), + }; + return (name, kind); + } + let (name, scalar) = self.scalars[index - self.columns.len()]; + let kind = match scalar { + TableScalar::Integer(_) => TableType::Integer, + TableScalar::Number(_) => TableType::Number, + TableScalar::Boolean(_) => TableType::Boolean, + }; + (name, kind) + } + + /// The cell at `column`, `row`. + pub fn value(&self, column: usize, row: usize) -> TableValue<'_> { + if let Some((_, values)) = self.columns.get(column) { + return match values { + NodeColumn::Integer(values) => TableValue::Integer(values.values[row]), + NodeColumn::Number(values) => TableValue::Number(values.values[row]), + NodeColumn::OptionalNumber(values) => { + let value = values.values[row]; + if value.is_nan() { + TableValue::Null + } else { + TableValue::Number(value) + } + } + NodeColumn::Node(values) => { + TableValue::Node(&self.graph.node_ids()[values.values[row]]) + } + NodeColumn::Boolean(values) => TableValue::Boolean(values.values[row]), + NodeColumn::Vector { values, dimension } => { + TableValue::Vector(&values.values[row * dimension..(row + 1) * dimension]) + } + }; + } + match self.scalars[column - self.columns.len()].1 { + TableScalar::Integer(value) => TableValue::Integer(value), + TableScalar::Number(value) => TableValue::Number(value), + TableScalar::Boolean(value) => TableValue::Boolean(value), + } + } + + /// An integer column by name. + pub fn integers(&self, name: &str) -> Option<&[i64]> { + self.columns.iter().find_map(|(n, c)| match c { + NodeColumn::Integer(values) if *n == name => Some(values.values.as_slice()), + _ => None, + }) + } + + /// A double column by name; NaN marks a node without a value. + pub fn numbers(&self, name: &str) -> Option<&[f64]> { + self.columns.iter().find_map(|(n, c)| match c { + NodeColumn::Number(values) | NodeColumn::OptionalNumber(values) if *n == name => { + Some(values.values.as_slice()) + } + _ => None, + }) + } + + /// A node-reference column by name, as projection rows. + pub fn nodes(&self, name: &str) -> Option<&[usize]> { + self.columns.iter().find_map(|(n, c)| match c { + NodeColumn::Node(values) if *n == name => Some(values.values.as_slice()), + _ => None, + }) + } + + /// A vector column by name: its values row by row, and its dimension. + pub fn vectors(&self, name: &str) -> Option<(&[f32], usize)> { + self.columns.iter().find_map(|(n, c)| match c { + NodeColumn::Vector { values, dimension } if *n == name => { + Some((values.values.as_slice(), *dimension)) + } + _ => None, + }) + } + + /// A whole-result scalar by name. + pub fn scalar_value(&self, name: &str) -> Option { + self.scalars + .iter() + .find_map(|(n, value)| (*n == name).then_some(*value)) + } +} diff --git a/crates/grust-algorithms/src/triangles.rs b/crates/grust-algorithms/src/triangles.rs new file mode 100644 index 00000000..980d5fe0 --- /dev/null +++ b/crates/grust-algorithms/src/triangles.rs @@ -0,0 +1,223 @@ +//! Triangle counts and local clustering coefficients on the simple undirected +//! graph underlying a projection, by the degree-ordered forward algorithm. + +use std::sync::atomic::{AtomicU64, Ordering}; + +use crate::{ + AlgorithmError, GraphProjection, Orientation, Result, + buffer::Buffer, + meter::Meter, + parallel, + table::{NodeColumn, NodeTable, TableScalar}, +}; + +/// Triangle controls. +#[derive(Clone, Copy, Debug, Default)] +pub struct TriangleOptions { + /// Leave out nodes whose simple degree exceeds this. Such a node reports + /// `-1` triangles and no coefficient, and triangles through it are not + /// counted for anyone: hubs dominate the cost and rarely the question. + pub max_degree: Option, +} + +/// Per-node triangle counts and clustering coefficients. +pub struct Triangles { + graph: GraphProjection, + triangles: Buffer, + coefficients: Buffer, + total: i64, + average: f64, +} + +impl Triangles { + /// Projection supplying external ids and snapshot identity. + pub fn projection(&self) -> &GraphProjection { + &self.graph + } + /// Triangles through each node; `-1` for a node left out by `max_degree`. + pub fn triangles(&self) -> &[i64] { + &self.triangles.values + } + /// `2t / (d(d-1))` per node, NaN where it is undefined: simple degree below + /// two, or a node left out by `max_degree`. + pub fn coefficients(&self) -> &[f64] { + &self.coefficients.values + } + /// Distinct triangles in the graph. + pub fn triangle_count(&self) -> i64 { + self.total + } + /// Mean coefficient over the nodes that have one; zero if none do. + pub fn average_coefficient(&self) -> f64 { + self.average + } + /// `triangles`, then the `triangleCount` scalar. + pub fn into_triangle_table(self) -> Result { + Ok(NodeTable::new(&self.graph) + .column("triangles", NodeColumn::Integer(self.triangles))? + .scalar("triangleCount", TableScalar::Integer(self.total))) + } + /// `coefficient`, `triangles`, then the `averageCoefficient` scalar. + pub fn into_coefficient_table(self) -> Result { + Ok(NodeTable::new(&self.graph) + .column("coefficient", NodeColumn::OptionalNumber(self.coefficients))? + .column("triangles", NodeColumn::Integer(self.triangles))? + .scalar("averageCoefficient", TableScalar::Number(self.average))) + } +} + +/// Count triangles on an undirected projection. +/// +/// Triangles are a property of the **simple** graph: **parallel edges count +/// once and self-loops are ignored**, unlike `degree` and `kCore`, which count +/// multiplicity. A node's degree here is its number of distinct neighbours. Any +/// other orientation is rejected rather than symmetrized — project with +/// `orientation: "undirected"`. +/// +/// Each triangle is found once, at its lowest-ranked corner, where rank orders +/// nodes by (degree, row): O(A^1.5) intersections. Nodes are split into blocks +/// of equal estimated work and counted on the caller's rayon pool into atomic +/// per-node counters; the counts, and the work charged, are the same at any +/// pool width. +pub fn triangles(graph: &GraphProjection, options: TriangleOptions) -> Result { + let context = graph.execution(); + context.checkpoint()?; + if graph.orientation() != Orientation::Undirected { + return Err(AlgorithmError::InvalidArguments( + "triangles are defined on undirected graphs; project with orientation \"undirected\"" + .into(), + )); + } + let n = graph.node_count(); + let adjacency = graph.outgoing(); + let mut meter = Meter::new(context); + + // Distinct neighbours per node, sorted, loops removed. + let mut simple_offsets = Buffer::filled(n + 1, 0usize, context)?; + let mut simple = Buffer::capacity(adjacency.targets.values.len(), context)?; + for node in 0..n { + let range = adjacency.range(node); + meter.tick(1 + range.len())?; + let first = simple.values.len(); + simple.values.extend( + adjacency.targets.values[range] + .iter() + .copied() + .filter(|&target| target != node), + ); + simple.values[first..].sort_unstable(); + let mut kept = first; + for index in first..simple.values.len() { + if kept == first || simple.values[kept - 1] != simple.values[index] { + simple.values[kept] = simple.values[index]; + kept += 1; + } + } + simple.values.truncate(kept); + simple_offsets.values[node + 1] = kept; + } + let degree = |node: usize| simple_offsets.values[node + 1] - simple_offsets.values[node]; + let included = |node: usize| options.max_degree.is_none_or(|limit| degree(node) <= limit); + let rank = |node: usize| (degree(node), node); + + // Forward lists: included neighbours of higher rank, still sorted by row. + let mut forward_offsets = Buffer::filled(n + 1, 0usize, context)?; + let mut forward = Buffer::capacity(simple.values.len() / 2 + 1, context)?; + for node in 0..n { + meter.tick(1 + degree(node))?; + if included(node) { + let row = &simple.values[simple_offsets.values[node]..simple_offsets.values[node + 1]]; + forward.values.extend( + row.iter() + .copied() + .filter(|&other| included(other) && rank(other) > rank(node)), + ); + } + forward_offsets.values[node + 1] = forward.values.len(); + } + meter.flush()?; + let forward_row = |node: usize| { + &forward.values[forward_offsets.values[node]..forward_offsets.values[node + 1]] + }; + + // Work per node is about |fwd(u)| merges of length up to |fwd(u)| + |fwd(v)|. + let mut weights = Buffer::capacity(n, context)?; + for node in 0..n { + let len = forward_row(node).len(); + weights.values.push(len.saturating_mul(len)); + } + let ranges = parallel::balanced_ranges(&weights.values, parallel::width()); + let counts = Buffer::filled_with(n, || AtomicU64::new(0), context)?; + + parallel::map_ranges(&ranges, |range| { + let mut meter = Meter::new(context); + for u in range { + let row_u = forward_row(u); + for &v in row_u { + let row_v = forward_row(v); + meter.tick(1 + row_u.len() + row_v.len())?; + // Both rows are sorted by node row: a plain merge finds every + // common forward neighbour w, closing the triangle u-v-w once. + let (mut i, mut j) = (0, 0); + while i < row_u.len() && j < row_v.len() { + match row_u[i].cmp(&row_v[j]) { + std::cmp::Ordering::Less => i += 1, + std::cmp::Ordering::Greater => j += 1, + std::cmp::Ordering::Equal => { + for corner in [u, v, row_u[i]] { + counts.values[corner].fetch_add(1, Ordering::Relaxed); + } + i += 1; + j += 1; + } + } + } + } + } + meter.flush() + })?; + + let mut triangles = Buffer::capacity(n, context)?; + let mut coefficients = Buffer::capacity(n, context)?; + let mut corners: u128 = 0; + let mut coefficient_sum = 0.0f64; + let mut coefficient_count = 0usize; + let mut meter = Meter::new(context); + for node in 0..n { + meter.tick(1)?; + if !included(node) { + triangles.values.push(-1); + coefficients.values.push(f64::NAN); + continue; + } + let count = counts.values[node].load(Ordering::Relaxed); + corners += u128::from(count); + triangles.values.push( + i64::try_from(count) + .map_err(|_| AlgorithmError::Numerical("triangle count exceeds Int64".into()))?, + ); + let d = degree(node); + if d < 2 { + coefficients.values.push(f64::NAN); + } else { + let coefficient = 2.0 * count as f64 / (d as f64 * (d as f64 - 1.0)); + coefficient_sum += coefficient; + coefficient_count += 1; + coefficients.values.push(coefficient); + } + } + meter.flush()?; + let total = i64::try_from(corners / 3) + .map_err(|_| AlgorithmError::Numerical("triangle count exceeds Int64".into()))?; + Ok(Triangles { + graph: graph.clone(), + triangles, + coefficients, + total, + average: if coefficient_count == 0 { + 0.0 + } else { + coefficient_sum / coefficient_count as f64 + }, + }) +} diff --git a/crates/grust-algorithms/tests/betweenness.rs b/crates/grust-algorithms/tests/betweenness.rs new file mode 100644 index 00000000..da91f386 --- /dev/null +++ b/crates/grust-algorithms/tests/betweenness.rs @@ -0,0 +1,405 @@ +//! Betweenness against an all-pairs definition that counts shortest paths. + +use grust_algorithms::{ + AlgorithmError, BetweennessOptions, ExecutionContext, ExecutionLimits, GraphProjection, + Orientation, ProjectionEdge, SnapshotIdentity, betweenness, +}; + +const ORIENTATIONS: [Orientation; 3] = [ + Orientation::Outgoing, + Orientation::Incoming, + Orientation::Undirected, +]; + +fn limits(work_units: usize) -> ExecutionLimits { + ExecutionLimits { + memory_bytes: 256 * 1024 * 1024, + work_units, + batch_rows: 1024, + deadline: None, + } +} + +fn context() -> ExecutionContext { + ExecutionContext::new(limits(2_000_000_000)).unwrap() +} + +fn graph( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + weights.map(<[f64]>::to_vec), + orientation, + context, + ) + .unwrap() +} + +/// The definition, with no traversal in it: all-pairs distances by +/// Floyd-Warshall, path counts by summing over last arcs, then the pair sum. +fn by_definition( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, +) -> Vec { + let mut arcs = Vec::new(); + for (index, &(a, b)) in edges.iter().enumerate() { + let w = weights.map_or(1.0, |w| w[index]); + if a == b { + continue; + } + match orientation { + Orientation::Outgoing => arcs.push((a, b, w)), + Orientation::Incoming => arcs.push((b, a, w)), + Orientation::Undirected => { + arcs.push((a, b, w)); + arcs.push((b, a, w)); + } + } + } + let mut dist = vec![vec![f64::INFINITY; n]; n]; + for (v, row) in dist.iter_mut().enumerate() { + row[v] = 0.0; + } + for &(a, b, w) in &arcs { + dist[a][b] = dist[a][b].min(w); + } + for k in 0..n { + for i in 0..n { + for j in 0..n { + let through = dist[i][k] + dist[k][j]; + if through < dist[i][j] { + dist[i][j] = through; + } + } + } + } + let mut sigma = vec![vec![0.0f64; n]; n]; + for s in 0..n { + let mut order: Vec = (0..n).filter(|&t| dist[s][t].is_finite()).collect(); + order.sort_by(|&a, &b| dist[s][a].total_cmp(&dist[s][b])); + sigma[s][s] = 1.0; + for &t in &order { + if t == s { + continue; + } + sigma[s][t] = arcs + .iter() + .filter(|&&(u, v, w)| v == t && dist[s][u] + w == dist[s][t]) + .map(|&(u, _, _)| sigma[s][u]) + .sum(); + } + } + let mut scores = vec![0.0; n]; + for s in 0..n { + for t in 0..n { + if s == t || sigma[s][t] == 0.0 { + continue; + } + for v in 0..n { + if v != s && v != t && dist[s][v] + dist[v][t] == dist[s][t] { + scores[v] += sigma[s][v] * sigma[v][t] / sigma[s][t]; + } + } + } + } + if orientation == Orientation::Undirected { + for score in &mut scores { + *score /= 2.0; + } + } + scores +} + +fn assert_close(actual: &[f64], expected: &[f64], what: &str) { + assert_eq!(actual.len(), expected.len(), "{what}"); + for (row, (a, e)) in actual.iter().zip(expected).enumerate() { + assert!( + (a - e).abs() <= 1e-9 * (1.0 + e.abs()), + "{what}: row {row} is {a}, the definition says {e}" + ); + } +} + +fn check(n: usize, edges: &[(usize, usize)], weights: Option<&[f64]>) { + for orientation in ORIENTATIONS { + let context = context(); + let result = betweenness( + &graph(n, edges, weights, orientation, &context), + BetweennessOptions::default(), + ) + .unwrap(); + assert_close( + result.values(), + &by_definition(n, edges, weights, orientation), + &format!("{orientation:?} {edges:?} {weights:?}"), + ); + } +} + +struct Xorshift(u64); +impl Xorshift { + fn below(&mut self, bound: usize) -> usize { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 % bound as u64) as usize + } +} + +#[test] +fn betweenness_matches_the_definition_on_every_small_graph() { + let mut checked = 0; + for n in 0..=5usize { + let pairs: Vec<(usize, usize)> = (0..n) + .flat_map(|a| (a + 1..n).map(move |b| (a, b))) + .collect(); + for mask in 0u32..1 << pairs.len() { + let edges: Vec<_> = pairs + .iter() + .enumerate() + .filter(|(bit, _)| mask >> bit & 1 == 1) + .map(|(_, &pair)| pair) + .collect(); + check(n, &edges, None); + checked += 3; + } + } + assert_eq!(checked, 3 * (1 + 1 + 2 + 8 + 64 + 1024)); +} + +#[test] +fn betweenness_matches_the_definition_on_random_multigraphs_with_and_without_weights() { + let mut random = Xorshift(0x2545_F491_4F6C_DD1D); + for round in 0..1500 { + let n = 2 + random.below(8); + let count = random.below(3 * n); + // Any pair, in either direction, repeated or looping. + let edges: Vec<_> = (0..count) + .map(|_| (random.below(n), random.below(n))) + .collect(); + if round % 2 == 0 { + check(n, &edges, None); + } else { + // Small integers: sums are exact, so ties are real ties. + let weights: Vec = (0..count).map(|_| (1 + random.below(3)) as f64).collect(); + check(n, &edges, Some(&weights)); + } + } +} + +#[test] +fn betweenness_states_its_textbook_values() { + let context = context(); + // A path 0-1-2-3-4: the middle lies between 2*2 pairs, its neighbours 1*3. + let path = [(0, 1), (1, 2), (2, 3), (3, 4)]; + let undirected = betweenness( + &graph(5, &path, None, Orientation::Undirected, &context), + BetweennessOptions::default(), + ) + .unwrap(); + assert_eq!(undirected.values(), [0.0, 3.0, 4.0, 3.0, 0.0]); + let directed = betweenness( + &graph(5, &path, None, Orientation::Outgoing, &context), + BetweennessOptions::default(), + ) + .unwrap(); + assert_eq!(directed.values(), [0.0, 3.0, 4.0, 3.0, 0.0]); + + // A star's hub lies between every pair of leaves; normalised, that is 1. + let star = [(0, 1), (0, 2), (0, 3), (0, 4)]; + let normalized = betweenness( + &graph(5, &star, None, Orientation::Undirected, &context), + BetweennessOptions { + normalized: true, + ..Default::default() + }, + ) + .unwrap(); + assert_eq!(normalized.values(), [1.0, 0.0, 0.0, 0.0, 0.0]); + + // Parallel edges are distinct shortest paths: two of the three 0-3 routes + // pass through 1. + let doubled = [(0, 1), (0, 1), (0, 2), (1, 3), (2, 3)]; + let result = betweenness( + &graph(4, &doubled, None, Orientation::Outgoing, &context), + BetweennessOptions::default(), + ) + .unwrap(); + assert_close( + result.values(), + &[0.0, 2.0 / 3.0, 1.0 / 3.0, 0.0], + "parallel edges", + ); + + // Fewer than three nodes: nothing is between anything, normalised or not. + let pair = betweenness( + &graph(2, &[(0, 1)], None, Orientation::Undirected, &context), + BetweennessOptions { + normalized: true, + ..Default::default() + }, + ) + .unwrap(); + assert_eq!(pair.values(), [0.0, 0.0]); +} + +#[test] +fn betweenness_rejects_zero_weights_and_an_empty_sample() { + let context = context(); + let projection = graph( + 3, + &[(0, 1), (1, 2)], + Some(&[1.0, 0.0]), + Orientation::Outgoing, + &context, + ); + let held = context.usage().unwrap().live_bytes; + assert!(matches!( + betweenness(&projection, BetweennessOptions::default()), + Err(AlgorithmError::InvalidArguments(message)) if message.contains("zero weight") + )); + assert!(matches!( + betweenness( + &projection, + BetweennessOptions { + sampling_size: Some(0), + ..Default::default() + } + ), + Err(AlgorithmError::InvalidArguments(_)) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); +} + +fn scrambled(n: usize, edges: usize) -> Vec<(usize, usize)> { + let mut random = Xorshift(0x9E37_79B9_7F4A_7C15); + (0..edges) + .map(|_| { + let a = random.below(n); + let b = random.below(n).pow(2) / n; + (a, b) + }) + .collect() +} + +#[test] +fn a_sample_of_every_node_is_the_exact_answer_and_a_smaller_one_is_seeded() { + let edges = scrambled(300, 1500); + let context = context(); + let projection = graph(300, &edges, None, Orientation::Outgoing, &context); + let exact = betweenness(&projection, BetweennessOptions::default()).unwrap(); + for size in [300, 5000] { + let all = betweenness( + &projection, + BetweennessOptions { + sampling_size: Some(size), + seed: 7, + ..Default::default() + }, + ) + .unwrap(); + assert_eq!(all.values(), exact.values()); + } + let sample = |seed| { + betweenness( + &projection, + BetweennessOptions { + sampling_size: Some(60), + seed, + ..Default::default() + }, + ) + .unwrap() + .values() + .to_vec() + }; + assert_eq!(sample(1), sample(1)); + assert_ne!(sample(1), sample(2)); + // The estimate is scaled to the exact total, not a fifth of it. + let total: f64 = exact.values().iter().sum(); + let estimated: f64 = sample(1).iter().sum(); + assert!( + (estimated / total - 1.0).abs() < 0.5, + "{estimated} vs {total}" + ); +} + +#[test] +fn betweenness_is_identical_at_any_pool_width_and_charges_the_same_work() { + let edges = scrambled(500, 3000); + let weights: Vec = (0..edges.len()).map(|i| (1 + i % 4) as f64).collect(); + for weighted in [false, true] { + let run = |threads: usize| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .unwrap(); + pool.install(|| { + let context = context(); + let projection = graph( + 500, + &edges, + weighted.then_some(&weights[..]), + Orientation::Undirected, + &context, + ); + let before = context.usage().unwrap().work_units; + let result = betweenness(&projection, BetweennessOptions::default()).unwrap(); + let work = context.usage().unwrap().work_units - before; + let bits: Vec = result.values().iter().map(|v| v.to_bits()).collect(); + (bits, work) + }) + }; + let (bits, work) = run(1); + for threads in [2, 3, 8] { + let (other, other_work) = run(threads); + assert_eq!(other, bits, "{threads} threads, weighted {weighted}"); + assert_eq!(other_work, work, "{threads} threads charged differently"); + } + } +} + +#[test] +fn betweenness_observes_cancellation_and_budget_and_releases_scratch() { + let edges = scrambled(300, 3000); + let context = context(); + let projection = graph(300, &edges, None, Orientation::Undirected, &context); + let projection_work = context.usage().unwrap().work_units; + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + betweenness(&projection, BetweennessOptions::default()), + Err(AlgorithmError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + let tight = ExecutionContext::new(limits(projection_work + 50_000)).unwrap(); + let projection = graph(300, &edges, None, Orientation::Undirected, &tight); + let held = tight.usage().unwrap().live_bytes; + assert!(matches!( + betweenness(&projection, BetweennessOptions::default()), + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + assert_eq!(tight.usage().unwrap().live_bytes, held); +} diff --git a/crates/grust-algorithms/tests/biconnected.rs b/crates/grust-algorithms/tests/biconnected.rs new file mode 100644 index 00000000..1bca3e7d --- /dev/null +++ b/crates/grust-algorithms/tests/biconnected.rs @@ -0,0 +1,273 @@ +//! Bridges, articulation points and biconnected components, each against its +//! definition: remove the thing and recount, or enumerate the simple cycles. + +use grust_algorithms::{ + AlgorithmError, ExecutionContext, ExecutionLimits, GraphProjection, Orientation, + ProjectionEdge, SnapshotIdentity, biconnectivity, +}; + +fn limits(work_units: usize) -> ExecutionLimits { + ExecutionLimits { + memory_bytes: 512 * 1024 * 1024, + work_units, + batch_rows: 1024, + deadline: None, + } +} + +fn context() -> ExecutionContext { + ExecutionContext::new(limits(500_000_000)).unwrap() +} + +fn graph( + n: usize, + edges: &[(usize, usize)], + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + // Not the slot, so a test can tell the two apart. + ordinal: 100 + ordinal, + id: None, + }) + .collect(), + None, + orientation, + context, + ) + .unwrap() +} + +struct Sets(Vec); +impl Sets { + fn new(n: usize) -> Self { + Self((0..n).collect()) + } + fn find(&mut self, mut x: usize) -> usize { + while self.0[x] != x { + self.0[x] = self.0[self.0[x]]; + x = self.0[x]; + } + x + } + fn union(&mut self, a: usize, b: usize) { + let (a, b) = (self.find(a), self.find(b)); + self.0[a.max(b)] = a.min(b); + } +} + +/// Connected components among the nodes left, using the edges left. +fn pieces( + n: usize, + edges: &[(usize, usize)], + without_edge: Option, + without_node: Option, +) -> usize { + let mut sets = Sets::new(n); + for (index, &(a, b)) in edges.iter().enumerate() { + if Some(index) != without_edge && Some(a) != without_node && Some(b) != without_node { + sets.union(a, b); + } + } + (0..n) + .filter(|&v| Some(v) != without_node && sets.find(v) == v) + .count() +} + +/// Two edges share a component exactly when a simple cycle runs through both. +/// Enumerate every simple cycle, parallel pairs included, and merge along each. +fn components_by_cycles(n: usize, edges: &[(usize, usize)]) -> Vec> { + let mut sets = Sets::new(edges.len()); + fn extend( + start: usize, + at: usize, + edges: &[(usize, usize)], + on_path: &mut Vec, + used: &mut Vec, + sets: &mut Sets, + ) { + for (index, &(a, b)) in edges.iter().enumerate() { + if a == b || used.contains(&index) || (a != at && b != at) { + continue; + } + let next = if a == at { b } else { a }; + if next == start && !used.is_empty() { + for &other in used.iter() { + sets.union(index, other); + } + } else if !on_path[next] && next > start { + on_path[next] = true; + used.push(index); + extend(start, next, edges, on_path, used, sets); + used.pop(); + on_path[next] = false; + } + } + } + for start in 0..n { + let mut on_path = vec![false; n]; + on_path[start] = true; + extend( + start, + start, + edges, + &mut on_path, + &mut Vec::new(), + &mut sets, + ); + } + // Name each class by its smallest ordinal, which is 100 + the smallest slot. + (0..edges.len()) + .map(|slot| (edges[slot].0 != edges[slot].1).then(|| 100 + sets.find(slot))) + .collect() +} + +fn check(n: usize, edges: &[(usize, usize)]) { + let context = context(); + let result = biconnectivity(&graph(n, edges, Orientation::Undirected, &context)).unwrap(); + let whole = pieces(n, edges, None, None); + let bridges: Vec = (0..edges.len()) + .filter(|&edge| pieces(n, edges, Some(edge), None) > whole) + .collect(); + assert_eq!(result.bridges(), bridges, "bridges of {edges:?}"); + let points: Vec = (0..n) + .filter(|&node| { + // Removing a node also removes it from the count; an isolate's + // removal lowers the count by one and splits nothing. + let isolated = !edges + .iter() + .any(|&(a, b)| a != b && (a == node || b == node)); + pieces(n, edges, None, Some(node)) > whole - usize::from(isolated) + }) + .collect(); + assert_eq!( + result.articulation_points(), + points, + "articulation points of {edges:?}" + ); + assert_eq!( + result.components(), + components_by_cycles(n, edges), + "components of {edges:?}" + ); +} + +struct Xorshift(u64); +impl Xorshift { + fn below(&mut self, bound: usize) -> usize { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 % bound as u64) as usize + } +} + +#[test] +fn every_simple_graph_of_up_to_five_nodes_matches_the_definitions() { + for n in 0..=5usize { + let pairs: Vec<(usize, usize)> = (0..n) + .flat_map(|a| (a + 1..n).map(move |b| (a, b))) + .collect(); + for mask in 0u32..1 << pairs.len() { + let edges: Vec<_> = pairs + .iter() + .enumerate() + .filter(|(bit, _)| mask >> bit & 1 == 1) + .map(|(_, &pair)| pair) + .collect(); + check(n, &edges); + } + } +} + +#[test] +fn random_multigraphs_with_loops_match_the_definitions() { + let mut random = Xorshift(0x2545_F491_4F6C_DD1D); + for _ in 0..4000 { + let n = 1 + random.below(8); + let count = random.below(n + 4); + let edges: Vec<_> = (0..count) + .map(|_| (random.below(n), random.below(n))) + .collect(); + check(n, &edges); + } +} + +#[test] +fn the_multigraph_cases_are_as_documented() { + let context = context(); + // 0-1 doubled, 1-2 single, a loop on 2, and 3 alone. + let edges = [(0, 1), (1, 0), (1, 2), (2, 2)]; + let result = biconnectivity(&graph(4, &edges, Orientation::Undirected, &context)).unwrap(); + // The doubled pair is a cycle of two: no bridge there, one component. + assert_eq!(result.bridges(), [2]); + assert_eq!(result.articulation_points(), [1]); + assert_eq!(result.components(), [Some(100), Some(100), Some(102), None]); + + // Two triangles sharing node 2: one articulation point, no bridge. + let bowtie = [(0, 1), (1, 2), (2, 0), (2, 3), (3, 4), (4, 2)]; + let result = biconnectivity(&graph(5, &bowtie, Orientation::Undirected, &context)).unwrap(); + assert!(result.bridges().is_empty()); + assert_eq!(result.articulation_points(), [2]); + assert_eq!( + result.components(), + [100, 100, 100, 103, 103, 103].map(Some) + ); + + assert!(matches!( + biconnectivity(&graph(2, &[(0, 1)], Orientation::Outgoing, &context)), + Err(AlgorithmError::InvalidArguments(message)) if message.contains("undirected") + )); +} + +#[test] +fn a_million_node_path_does_not_overflow_the_stack() { + let n = 1_000_000; + let edges: Vec<_> = (1..n).map(|node| (node - 1, node)).collect(); + let context = context(); + let result = biconnectivity(&graph(n, &edges, Orientation::Undirected, &context)).unwrap(); + assert_eq!(result.bridges().len(), n - 1); + assert_eq!(result.articulation_points().len(), n - 2); + // A long cycle is one component, found only when the pass unwinds fully. + let mut cycle = edges; + cycle.push((n - 1, 0)); + let context = self::context(); + let result = biconnectivity(&graph(n, &cycle, Orientation::Undirected, &context)).unwrap(); + assert!(result.bridges().is_empty() && result.articulation_points().is_empty()); + assert!(result.components().iter().all(|&c| c == Some(100))); +} + +#[test] +fn biconnectivity_observes_cancellation_and_budget_and_releases_scratch() { + let edges: Vec<_> = (1..5000).map(|node| (node / 2, node)).collect(); + let context = context(); + let projection = graph(5000, &edges, Orientation::Undirected, &context); + let projection_work = context.usage().unwrap().work_units; + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + biconnectivity(&projection), + Err(AlgorithmError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + let tight = ExecutionContext::new(limits(projection_work + 3000)).unwrap(); + let projection = graph(5000, &edges, Orientation::Undirected, &tight); + let held = tight.usage().unwrap().live_bytes; + assert!(matches!( + biconnectivity(&projection), + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + assert_eq!(tight.usage().unwrap().live_bytes, held); +} diff --git a/crates/grust-algorithms/tests/closeness.rs b/crates/grust-algorithms/tests/closeness.rs new file mode 100644 index 00000000..4a3f6d37 --- /dev/null +++ b/crates/grust-algorithms/tests/closeness.rs @@ -0,0 +1,350 @@ +//! Closeness and harmonic centrality against Floyd-Warshall distances. + +use grust_algorithms::{ + AlgorithmError, ClosenessOptions, ExecutionContext, ExecutionLimits, GraphProjection, + HarmonicOptions, Orientation, ProjectionEdge, SnapshotIdentity, closeness, harmonic, +}; + +const ORIENTATIONS: [Orientation; 3] = [ + Orientation::Outgoing, + Orientation::Incoming, + Orientation::Undirected, +]; + +fn limits(work_units: usize) -> ExecutionLimits { + ExecutionLimits { + memory_bytes: 256 * 1024 * 1024, + work_units, + batch_rows: 1024, + deadline: None, + } +} + +fn context() -> ExecutionContext { + ExecutionContext::new(limits(2_000_000_000)).unwrap() +} + +fn graph( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + weights.map(<[f64]>::to_vec), + orientation, + context, + ) + .unwrap() +} + +fn distances( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, +) -> Vec> { + let mut dist = vec![vec![f64::INFINITY; n]; n]; + for (v, row) in dist.iter_mut().enumerate() { + row[v] = 0.0; + } + for (index, &(a, b)) in edges.iter().enumerate() { + let w = weights.map_or(1.0, |w| w[index]); + if orientation != Orientation::Incoming { + dist[a][b] = dist[a][b].min(w); + } + if orientation != Orientation::Outgoing { + dist[b][a] = dist[b][a].min(w); + } + } + for k in 0..n { + for i in 0..n { + for j in 0..n { + let through = dist[i][k] + dist[k][j]; + if through < dist[i][j] { + dist[i][j] = through; + } + } + } + } + dist +} + +/// (closeness, Wasserman-Faust closeness, harmonic, normalised harmonic). +fn by_definition(dist: &[Vec]) -> [Vec; 4] { + let n = dist.len(); + let mut out = [vec![], vec![], vec![], vec![]]; + for (v, row) in dist.iter().enumerate() { + let reachable: Vec = row + .iter() + .enumerate() + .filter(|&(u, d)| u != v && d.is_finite()) + .map(|(_, &d)| d) + .collect(); + let r = reachable.len() as f64; + let sum: f64 = reachable.iter().sum(); + let plain = if reachable.is_empty() { 0.0 } else { r / sum }; + let reciprocal: f64 = reachable.iter().map(|d| 1.0 / d).sum(); + out[0].push(plain); + out[1].push(if reachable.is_empty() { + 0.0 + } else { + plain * r / (n as f64 - 1.0) + }); + out[2].push(reciprocal); + out[3].push(if n > 1 { + reciprocal / (n as f64 - 1.0) + } else { + reciprocal + }); + } + out +} + +fn assert_close(actual: &[f64], expected: &[f64], what: &str) { + assert_eq!(actual.len(), expected.len(), "{what}"); + for (row, (a, e)) in actual.iter().zip(expected).enumerate() { + assert!( + (a - e).abs() <= 1e-12 * (1.0 + e.abs()), + "{what}: row {row} is {a}, the definition says {e}" + ); + } +} + +fn check(n: usize, edges: &[(usize, usize)], weights: Option<&[f64]>) { + for orientation in ORIENTATIONS { + let context = context(); + let projection = graph(n, edges, weights, orientation, &context); + let expected = by_definition(&distances(n, edges, weights, orientation)); + let what = format!("{orientation:?} {edges:?} {weights:?}"); + for (index, wasserman_faust) in [false, true].into_iter().enumerate() { + let result = closeness(&projection, ClosenessOptions { wasserman_faust }).unwrap(); + assert_close(result.values(), &expected[index], &what); + } + for (index, normalized) in [false, true].into_iter().enumerate() { + let result = harmonic(&projection, HarmonicOptions { normalized }).unwrap(); + assert_close(result.values(), &expected[2 + index], &what); + } + } +} + +struct Xorshift(u64); +impl Xorshift { + fn below(&mut self, bound: usize) -> usize { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 % bound as u64) as usize + } +} + +#[test] +fn distance_centralities_match_the_definition_on_every_small_graph() { + for n in 0..=5usize { + let pairs: Vec<(usize, usize)> = (0..n) + .flat_map(|a| (a + 1..n).map(move |b| (a, b))) + .collect(); + for mask in 0u32..1 << pairs.len() { + let edges: Vec<_> = pairs + .iter() + .enumerate() + .filter(|(bit, _)| mask >> bit & 1 == 1) + .map(|(_, &pair)| pair) + .collect(); + check(n, &edges, None); + } + } +} + +#[test] +fn distance_centralities_match_on_random_multigraphs_up_to_forty_nodes() { + let mut random = Xorshift(0x2545_F491_4F6C_DD1D); + for round in 0..600 { + let n = 2 + random.below(39); + let count = random.below(3 * n); + let edges: Vec<_> = (0..count) + .map(|_| (random.below(n), random.below(n))) + .collect(); + if round % 2 == 0 { + check(n, &edges, None); + } else { + let weights: Vec = (0..count) + .map(|_| (1 + random.below(8)) as f64 / 4.0) + .collect(); + check(n, &edges, Some(&weights)); + } + } +} + +#[test] +fn distance_centralities_state_their_textbook_values() { + let context = context(); + // A path 0-1-2 and a separate pair 3-4. + let edges = [(0, 1), (1, 2), (3, 4)]; + let projection = graph(6, &edges, None, Orientation::Undirected, &context); + let plain = closeness(&projection, ClosenessOptions::default()).unwrap(); + // Per component: the pair scores 1, as high as the path's centre. + assert_eq!(plain.values(), [2.0 / 3.0, 1.0, 2.0 / 3.0, 1.0, 1.0, 0.0]); + let corrected = closeness( + &projection, + ClosenessOptions { + wasserman_faust: true, + }, + ) + .unwrap(); + // Wasserman-Faust: scaled by reached / (n-1), so the pair drops below it. + assert_eq!( + corrected.values(), + [4.0 / 15.0, 2.0 / 5.0, 4.0 / 15.0, 1.0 / 5.0, 1.0 / 5.0, 0.0] + ); + let raw = harmonic(&projection, HarmonicOptions { normalized: false }).unwrap(); + assert_eq!(raw.values(), [1.5, 2.0, 1.5, 1.0, 1.0, 0.0]); + let normalized = harmonic(&projection, HarmonicOptions::default()).unwrap(); + assert_eq!(normalized.values()[1], 2.0 / 5.0); + + // Distances run from the node: a chain's head reaches everything, its tail nothing. + let chain = graph(3, &[(0, 1), (1, 2)], None, Orientation::Outgoing, &context); + assert_eq!( + closeness(&chain, ClosenessOptions::default()) + .unwrap() + .values(), + [2.0 / 3.0, 1.0, 0.0] + ); + let reversed = graph(3, &[(0, 1), (1, 2)], None, Orientation::Incoming, &context); + assert_eq!( + closeness(&reversed, ClosenessOptions::default()) + .unwrap() + .values(), + [0.0, 1.0, 2.0 / 3.0] + ); + + // One node: nothing to be close to, and no division by n-1 = 0. + let single = graph(1, &[], None, Orientation::Undirected, &context); + assert_eq!( + harmonic(&single, HarmonicOptions::default()) + .unwrap() + .values(), + [0.0] + ); + assert_eq!( + closeness( + &single, + ClosenessOptions { + wasserman_faust: true + } + ) + .unwrap() + .values(), + [0.0] + ); +} + +#[test] +fn distance_centralities_reject_zero_weights() { + let context = context(); + let projection = graph( + 3, + &[(0, 1), (1, 2)], + Some(&[1.0, 0.0]), + Orientation::Outgoing, + &context, + ); + let held = context.usage().unwrap().live_bytes; + assert!(matches!( + closeness(&projection, ClosenessOptions::default()), + Err(AlgorithmError::InvalidArguments(message)) if message.contains("zero distance") + )); + assert!(matches!( + harmonic(&projection, HarmonicOptions::default()), + Err(AlgorithmError::InvalidArguments(_)) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); +} + +fn scrambled(n: usize, edges: usize) -> Vec<(usize, usize)> { + let mut random = Xorshift(0x9E37_79B9_7F4A_7C15); + (0..edges) + .map(|_| (random.below(n), random.below(n).pow(2) / n)) + .collect() +} + +#[test] +fn distance_centralities_are_identical_at_any_pool_width_and_charge_the_same_work() { + let edges = scrambled(500, 3000); + let weights: Vec = (0..edges.len()).map(|i| (1 + i % 7) as f64 / 3.0).collect(); + for weighted in [false, true] { + let run = |threads: usize| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .unwrap(); + pool.install(|| { + let context = context(); + let projection = graph( + 500, + &edges, + weighted.then_some(&weights[..]), + Orientation::Outgoing, + &context, + ); + let before = context.usage().unwrap().work_units; + let close = closeness(&projection, ClosenessOptions::default()).unwrap(); + let harm = harmonic(&projection, HarmonicOptions::default()).unwrap(); + let work = context.usage().unwrap().work_units - before; + let bits: Vec = close + .values() + .iter() + .chain(harm.values()) + .map(|v| v.to_bits()) + .collect(); + (bits, work) + }) + }; + let (bits, work) = run(1); + for threads in [2, 3, 8] { + let (other, other_work) = run(threads); + assert_eq!(other, bits, "{threads} threads, weighted {weighted}"); + assert_eq!(other_work, work, "{threads} threads charged differently"); + } + } +} + +#[test] +fn distance_centralities_observe_cancellation_and_budget_and_release_scratch() { + let edges = scrambled(300, 3000); + let context = context(); + let projection = graph(300, &edges, None, Orientation::Undirected, &context); + let projection_work = context.usage().unwrap().work_units; + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + closeness(&projection, ClosenessOptions::default()), + Err(AlgorithmError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + let tight = ExecutionContext::new(limits(projection_work + 50_000)).unwrap(); + let projection = graph(300, &edges, None, Orientation::Undirected, &tight); + let held = tight.usage().unwrap().live_bytes; + assert!(matches!( + harmonic(&projection, HarmonicOptions::default()), + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + assert_eq!(tight.usage().unwrap().live_bytes, held); +} diff --git a/crates/grust-algorithms/tests/fastrp.rs b/crates/grust-algorithms/tests/fastrp.rs new file mode 100644 index 00000000..375512ed --- /dev/null +++ b/crates/grust-algorithms/tests/fastrp.rs @@ -0,0 +1,392 @@ +//! FastRP against the same computation done densely, in double precision. + +use grust_algorithms::{ + AlgorithmError, ExecutionContext, ExecutionLimits, FastRpOptions, GraphProjection, Orientation, + ProjectionEdge, SnapshotIdentity, fast_rp, +}; + +const ORIENTATIONS: [Orientation; 3] = [ + Orientation::Outgoing, + Orientation::Incoming, + Orientation::Undirected, +]; + +fn limits(work_units: usize) -> ExecutionLimits { + ExecutionLimits { + memory_bytes: 512 * 1024 * 1024, + work_units, + batch_rows: 1024, + deadline: None, + } +} + +fn context() -> ExecutionContext { + ExecutionContext::new(limits(4_000_000_000)).unwrap() +} + +fn graph( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + weights.map(<[f64]>::to_vec), + orientation, + context, + ) + .unwrap() +} + +/// `w[v][u]`: weight of arcs v -> u as the orientation reads them; an +/// undirected loop is one arc. +fn arcs( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, +) -> Vec> { + let mut w = vec![vec![0.0; n]; n]; + for (index, &(a, b)) in edges.iter().enumerate() { + let weight = weights.map_or(1.0, |w| w[index]); + match orientation { + Orientation::Outgoing => w[a][b] += weight, + Orientation::Incoming => w[b][a] += weight, + Orientation::Undirected => { + w[a][b] += weight; + if a != b { + w[b][a] += weight; + } + } + } + } + w +} + +fn unit(row: &[f64]) -> Vec { + let norm = row.iter().map(|v| v * v).sum::().sqrt(); + row.iter() + .map(|v| if norm > 0.0 { v / norm } else { 0.0 }) + .collect() +} + +struct Xorshift(u64); +impl Xorshift { + fn below(&mut self, bound: usize) -> usize { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 % bound as u64) as usize + } +} + +#[test] +fn fast_rp_matches_a_dense_recomputation_from_its_own_random_vectors() { + let mut random = Xorshift(0x2545_F491_4F6C_DD1D); + for round in 0..300 { + let n = 1 + random.below(12); + let count = random.below(3 * n); + let edges: Vec<_> = (0..count) + .map(|_| (random.below(n), random.below(n))) + .collect(); + let weights: Vec = (0..count).map(|_| (1 + random.below(4)) as f64).collect(); + let weights = (round % 2 == 1).then_some(&weights[..]); + let d = 8 + random.below(24); + let strength = [0.0, -0.5, 0.5][round % 3]; + let iteration_weights = [0.5, 1.0, -0.25, 2.0]; + let iteration_weights = &iteration_weights[..random.below(5)]; + let self_influence = [0.0, 1.0, 0.3][random.below(3)]; + for orientation in ORIENTATIONS { + let what = format!("{orientation:?} d={d} β={strength} {edges:?} {weights:?}"); + let context = context(); + let projection = graph(n, &edges, weights, orientation, &context); + let options = |iteration_weights, self_influence| FastRpOptions { + dimension: d, + iteration_weights, + self_influence, + normalization_strength: strength, + seed: round as u64, + }; + + // Round zero alone returns each random vector at unit length. Its + // entries are 0 or ±√3·scale, so the count of nonzeros gives the + // length back, and the degree gives the scale. + let zero = fast_rp(&projection, options(&[], 1.0)).unwrap(); + let w = arcs(n, &edges, weights, orientation); + let mut current: Vec> = (0..n) + .map(|v| { + let row = zero.embedding(v); + let nonzero = row.iter().filter(|&&x| x != 0.0).count() as f64; + let degree: f64 = w[v].iter().sum(); + // A node of degree zero scales by zero unless the strength is zero. + let scale = if strength == 0.0 { + 1.0 + } else if degree == 0.0 { + 0.0 + } else { + degree.powf(strength) + }; + row.iter() + .map(|&x| f64::from(x) * (3.0 * nonzero).sqrt() * scale) + .collect() + }) + .collect(); + let mut expected: Vec> = current + .iter() + .map(|row| unit(row).iter().map(|v| v * self_influence).collect()) + .collect(); + for &weight in iteration_weights { + current = (0..n) + .map(|v| { + let total: f64 = w[v].iter().sum(); + (0..d) + .map(|j| { + if total > 0.0 { + (0..n).map(|u| w[v][u] * current[u][j]).sum::() / total + } else { + 0.0 + } + }) + .collect() + }) + .collect(); + for (into, row) in expected.iter_mut().zip(¤t) { + for (value, add) in into.iter_mut().zip(unit(row)) { + *value += weight * add; + } + } + } + + let result = fast_rp(&projection, options(iteration_weights, self_influence)).unwrap(); + assert_eq!(result.dimension(), d); + for (v, wanted_row) in expected.iter().enumerate() { + for (j, (&actual, &wanted)) in + result.embedding(v).iter().zip(wanted_row).enumerate() + { + assert!( + (f64::from(actual) - wanted).abs() < 1e-4, + "{what}: node {v} entry {j}: {actual} vs {wanted}" + ); + } + } + } + } +} + +#[test] +fn the_random_vectors_are_sparse_seeded_and_independent_of_the_graph() { + let context = context(); + let n = 2000; + let edges: Vec<_> = (1..n).map(|node| (node - 1, node)).collect(); + let run = |seed, edges: &[(usize, usize)]| { + fast_rp( + &graph(n, edges, None, Orientation::Outgoing, &context), + FastRpOptions { + dimension: 64, + iteration_weights: &[], + self_influence: 1.0, + seed, + ..Default::default() + }, + ) + .unwrap() + .values() + .to_vec() + }; + let first = run(1, &edges); + assert_eq!(first, run(1, &edges)); + assert_eq!(first, run(1, &[]), "round zero does not look at arcs"); + assert_ne!(first, run(2, &edges)); + // A third of the entries are nonzero, half of those positive. + let nonzero = first.iter().filter(|&&v| v != 0.0).count() as f64; + let positive = first.iter().filter(|&&v| v > 0.0).count() as f64; + let total = first.len() as f64; + assert!( + (nonzero / total - 1.0 / 3.0).abs() < 0.01, + "{}", + nonzero / total + ); + assert!( + (positive / nonzero - 0.5).abs() < 0.01, + "{}", + positive / nonzero + ); +} + +#[test] +fn neighbours_in_a_clique_embed_closer_than_strangers() { + // Not a claim about quality: only that averaging pulls a clique together. + let mut edges = Vec::new(); + for base in [0, 10] { + for a in base..base + 10 { + for b in a + 1..base + 10 { + edges.push((a, b)); + } + } + } + let context = context(); + let result = fast_rp( + &graph(20, &edges, None, Orientation::Undirected, &context), + FastRpOptions { + seed: 9, + ..Default::default() + }, + ) + .unwrap(); + let cosine = |a: usize, b: usize| { + let (x, y) = (result.embedding(a), result.embedding(b)); + let dot: f32 = x.iter().zip(y).map(|(p, q)| p * q).sum(); + let norm = |v: &[f32]| v.iter().map(|p| p * p).sum::().sqrt(); + dot / (norm(x) * norm(y)) + }; + assert!(cosine(0, 5) > 0.9, "{}", cosine(0, 5)); + assert!(cosine(0, 15).abs() < 0.5, "{}", cosine(0, 15)); +} + +#[test] +fn fast_rp_rejects_bad_options_and_handles_the_empty_graph() { + let context = context(); + let projection = graph(3, &[(0, 1)], None, Orientation::Outgoing, &context); + let held = context.usage().unwrap().live_bytes; + for options in [ + FastRpOptions { + dimension: 0, + ..Default::default() + }, + FastRpOptions { + iteration_weights: &[f64::NAN], + ..Default::default() + }, + FastRpOptions { + self_influence: f64::INFINITY, + ..Default::default() + }, + ] { + assert!(matches!( + fast_rp(&projection, options), + Err(AlgorithmError::InvalidArguments(_)) + )); + } + assert_eq!(context.usage().unwrap().live_bytes, held); + let empty = fast_rp( + &graph(0, &[], None, Orientation::Outgoing, &context), + FastRpOptions::default(), + ) + .unwrap(); + assert!(empty.values().is_empty()); +} + +#[test] +fn fast_rp_is_identical_at_any_pool_width_and_charges_the_same_work() { + let mut random = Xorshift(0x9E37_79B9_7F4A_7C15); + let n = 5000; + let edges: Vec<_> = (0..30_000) + .map(|_| (random.below(n), random.below(n))) + .collect(); + let run = |threads: usize| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .unwrap(); + pool.install(|| { + let context = context(); + let projection = graph(n, &edges, None, Orientation::Undirected, &context); + let before = context.usage().unwrap().work_units; + let result = fast_rp( + &projection, + FastRpOptions { + dimension: 32, + normalization_strength: -0.5, + self_influence: 0.2, + ..Default::default() + }, + ) + .unwrap(); + let bits: Vec = result.values().iter().map(|v| v.to_bits()).collect(); + (bits, context.usage().unwrap().work_units - before) + }) + }; + let first = run(1); + for threads in [2, 3, 8] { + assert_eq!(run(threads), first, "{threads} threads"); + } +} + +#[test] +fn fast_rp_observes_cancellation_and_budget_and_releases_scratch() { + let edges: Vec<_> = (1..4000).map(|node| (node / 2, node)).collect(); + let context = context(); + let projection = graph(4000, &edges, None, Orientation::Undirected, &context); + let projection_work = context.usage().unwrap().work_units; + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + fast_rp(&projection, FastRpOptions::default()), + Err(AlgorithmError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + let tight = ExecutionContext::new(limits(projection_work + 100_000)).unwrap(); + let projection = graph(4000, &edges, None, Orientation::Undirected, &tight); + let held = tight.usage().unwrap().live_bytes; + assert!(matches!( + fast_rp(&projection, FastRpOptions::default()), + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + assert_eq!(tight.usage().unwrap().live_bytes, held); +} + +#[cfg(feature = "arrow")] +#[test] +fn fast_rp_arrow_batches_are_fixed_size_lists_of_float32() { + use arrow_array::{Array, FixedSizeListArray, Float32Array}; + let context = context(); + let projection = graph( + 5, + &[(0, 1), (1, 2)], + None, + Orientation::Undirected, + &context, + ); + let result = fast_rp( + &projection, + FastRpOptions { + dimension: 6, + ..Default::default() + }, + ) + .unwrap(); + let expected = result.values().to_vec(); + let mut cursor = result.into_table().unwrap().into_arrow_results(); + let batch = cursor.next_batch().unwrap().unwrap(); + let batch = batch.record_batch(); + assert_eq!(batch.schema().field(1).name(), "embedding"); + let lists = batch + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + assert_eq!( + (lists.len(), lists.value_length(), lists.null_count()), + (5, 6, 0) + ); + let values = lists.values(); + let values = values.as_any().downcast_ref::().unwrap(); + assert_eq!(values.values().to_vec(), expected); +} diff --git a/crates/grust-algorithms/tests/flow.rs b/crates/grust-algorithms/tests/flow.rs new file mode 100644 index 00000000..6bb617d7 --- /dev/null +++ b/crates/grust-algorithms/tests/flow.rs @@ -0,0 +1,264 @@ +//! Max flow against brute-force minimum cuts and the flow's own constraints. + +use grust_algorithms::{ + AlgorithmError, ExecutionContext, ExecutionLimits, GraphProjection, Orientation, + ProjectionEdge, SnapshotIdentity, max_flow, +}; + +const ORIENTATIONS: [Orientation; 3] = [ + Orientation::Outgoing, + Orientation::Incoming, + Orientation::Undirected, +]; + +fn limits(work_units: usize) -> ExecutionLimits { + ExecutionLimits { + memory_bytes: 512 * 1024 * 1024, + work_units, + batch_rows: 1024, + deadline: None, + } +} + +fn context() -> ExecutionContext { + ExecutionContext::new(limits(2_000_000_000)).unwrap() +} + +fn graph( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + weights.map(<[f64]>::to_vec), + orientation, + context, + ) + .unwrap() +} + +/// Capacity from a to b, summed over parallel edges, as the orientation reads them. +fn capacities( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, +) -> Vec> { + let mut c = vec![vec![0.0; n]; n]; + for (index, &(a, b)) in edges.iter().enumerate() { + let w = weights.map_or(1.0, |w| w[index]); + if a == b { + continue; + } + if orientation != Orientation::Incoming { + c[a][b] += w; + } + if orientation != Orientation::Outgoing { + c[b][a] += w; + } + } + c +} + +/// Capacity leaving the set `inside` describes. +fn leaving(c: &[Vec], inside: impl Fn(usize) -> bool) -> f64 { + c.iter() + .enumerate() + .filter(|&(a, _)| inside(a)) + .flat_map(|(_, row)| row.iter().enumerate()) + .filter(|&(b, _)| !inside(b)) + .map(|(_, capacity)| capacity) + .sum() +} + +/// The cheapest cut, over every set holding the source and not the target. +fn min_cut(c: &[Vec], source: usize, target: usize) -> f64 { + (0u32..1 << c.len()) + .filter(|set| set >> source & 1 == 1 && set >> target & 1 == 0) + .map(|set| leaving(c, |node| set >> node & 1 == 1)) + .fold(f64::INFINITY, f64::min) +} + +struct Xorshift(u64); +impl Xorshift { + fn below(&mut self, bound: usize) -> usize { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 % bound as u64) as usize + } +} + +fn check( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, + source: usize, + target: usize, +) { + let what = format!("{orientation:?} {source}->{target} {edges:?} {weights:?}"); + let context = context(); + let projection = graph(n, edges, weights, orientation, &context); + let result = max_flow(&projection, &format!("n{source}"), &format!("n{target}")).unwrap(); + let c = capacities(n, edges, weights, orientation); + assert_eq!(result.value(), min_cut(&c, source, target), "{what}"); + + // The reported cut separates the two and has the flow's value. + let side = result.source_side(); + assert!(side[source] && !side[target], "{what}"); + let cut = leaving(&c, |node| side[node]); + assert_eq!(cut, result.value(), "{what}"); + + // Every edge carries at most its capacity, along a direction it allows; + // what enters a node leaves it, except at the two ends. + let mut balance = vec![0.0; n]; + let mut previous = None; + for (from, to, slot, flow) in result.flows() { + assert!(previous < Some(slot), "{what}: rows out of edge order"); + previous = Some(slot); + let (a, b) = edges[slot]; + let allowed = match orientation { + Orientation::Outgoing => (from, to) == (a, b), + Orientation::Incoming => (from, to) == (b, a), + Orientation::Undirected => (from, to) == (a, b) || (from, to) == (b, a), + }; + assert!(allowed, "{what}: edge {slot} carries {from}->{to}"); + assert!( + flow > 0.0 && flow <= weights.map_or(1.0, |w| w[slot]), + "{what}" + ); + balance[from] -= flow; + balance[to] += flow; + } + for (node, &net) in balance.iter().enumerate() { + let expected = if node == source { + -result.value() + } else if node == target { + result.value() + } else { + 0.0 + }; + assert_eq!(net, expected, "{what}: node {node}"); + } +} + +#[test] +fn max_flow_equals_the_brute_force_minimum_cut_and_is_a_flow() { + let mut random = Xorshift(0x2545_F491_4F6C_DD1D); + for round in 0..2500 { + let n = 2 + random.below(8); + let count = random.below(4 * n); + let edges: Vec<_> = (0..count) + .map(|_| (random.below(n), random.below(n))) + .collect(); + // Integers, zero included: every sum is exact, so equality is exact. + let weights: Vec = (0..count).map(|_| random.below(6) as f64).collect(); + let weights = (round % 3 != 0).then_some(&weights[..]); + let source = random.below(n); + let target = (source + 1 + random.below(n - 1)) % n; + for orientation in ORIENTATIONS { + check(n, &edges, weights, orientation, source, target); + } + } +} + +#[test] +fn the_textbook_network_and_the_refusals_are_as_documented() { + // CLRS figure 26.1: maximum flow 23. + let edges = [ + (0, 1), + (0, 2), + (1, 3), + (2, 1), + (2, 4), + (3, 2), + (3, 5), + (4, 3), + (4, 5), + ]; + let weights = [16.0, 13.0, 12.0, 4.0, 14.0, 9.0, 20.0, 7.0, 4.0]; + let context = context(); + let projection = graph(6, &edges, Some(&weights), Orientation::Outgoing, &context); + let result = max_flow(&projection, "n0", "n5").unwrap(); + assert_eq!(result.value(), 23.0); + assert_eq!(result.source_side(), [true, true, true, false, true, false]); + // Nothing flows backwards along a directed network. + assert_eq!(max_flow(&projection, "n5", "n0").unwrap().value(), 0.0); + + let held = context.usage().unwrap().live_bytes; + assert!(matches!( + max_flow(&projection, "n0", "n0"), + Err(AlgorithmError::InvalidArguments(message)) if message.contains("target") + )); + assert!(matches!( + max_flow(&projection, "n0", "nowhere"), + Err(AlgorithmError::InvalidArguments(_)) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); +} + +#[test] +fn a_long_path_and_a_wide_network_finish_without_recursion() { + let n = 300_000; + let edges: Vec<_> = (1..n).map(|node| (node - 1, node)).collect(); + let context = context(); + let projection = graph(n, &edges, None, Orientation::Outgoing, &context); + let result = max_flow(&projection, "n0", &format!("n{}", n - 1)).unwrap(); + assert_eq!(result.value(), 1.0); + assert_eq!(result.flows().count(), n - 1); + + // A thousand disjoint two-hop routes: a thousand units. + let mut edges = Vec::new(); + for middle in 2..1002 { + edges.push((0, middle)); + edges.push((middle, 1)); + } + let context = self::context(); + let projection = graph(1002, &edges, None, Orientation::Outgoing, &context); + assert_eq!(max_flow(&projection, "n0", "n1").unwrap().value(), 1000.0); +} + +#[test] +fn max_flow_observes_cancellation_and_budget_and_releases_scratch() { + let mut random = Xorshift(0x9E37_79B9_7F4A_7C15); + let edges: Vec<_> = (0..40_000) + .map(|_| (random.below(3000), random.below(3000))) + .collect(); + let context = context(); + let projection = graph(3000, &edges, None, Orientation::Outgoing, &context); + let projection_work = context.usage().unwrap().work_units; + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + max_flow(&projection, "n0", "n1"), + Err(AlgorithmError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + let tight = ExecutionContext::new(limits(projection_work + 60_000)).unwrap(); + let projection = graph(3000, &edges, None, Orientation::Outgoing, &tight); + let held = tight.usage().unwrap().live_bytes; + assert!(matches!( + max_flow(&projection, "n0", "n1"), + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + assert_eq!(tight.usage().unwrap().live_bytes, held); +} diff --git a/crates/grust-algorithms/tests/kcore.rs b/crates/grust-algorithms/tests/kcore.rs new file mode 100644 index 00000000..5f470169 --- /dev/null +++ b/crates/grust-algorithms/tests/kcore.rs @@ -0,0 +1,288 @@ +//! k-core against its definition, exhaustively over small multigraphs. + +use grust_algorithms::{ + AlgorithmError as ProcedureError, ExecutionContext, ExecutionLimits, GraphProjection, + Orientation, ProjectionEdge, SnapshotIdentity, k_core, +}; + +fn context() -> ExecutionContext { + ExecutionContext::new(ExecutionLimits { + memory_bytes: 8 * 1024 * 1024, + work_units: 10_000_000, + batch_rows: 1024, + deadline: None, + }) + .unwrap() +} + +fn graph( + n: usize, + edges: &[(usize, usize)], + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + None, + orientation, + context, + ) + .unwrap() +} + +/// The definition, applied directly: the k-core is what survives repeatedly +/// deleting nodes whose degree (parallel edges counted, loops not) is below k. +fn cores_by_definition(n: usize, edges: &[(usize, usize)]) -> Vec { + let mut core = vec![0i64; n]; + for k in 1..=edges.len() as i64 { + let mut alive = vec![true; n]; + loop { + let mut removed = false; + for v in 0..n { + if !alive[v] { + continue; + } + let degree = edges + .iter() + .filter(|&&(a, b)| a != b && alive[a] && alive[b] && (a == v || b == v)) + .count() as i64; + if degree < k { + alive[v] = false; + removed = true; + } + } + if !removed { + break; + } + } + if !alive.iter().any(|&a| a) { + break; + } + for v in 0..n { + if alive[v] { + core[v] = k; + } + } + } + core +} + +#[test] +fn k_core_matches_an_independent_oracle() { + // Every multigraph on four nodes with multiplicity 0, 1 or 2 per pair, with + // and without a loop on every node: 3^6 * 2 = 1458 graphs. + let pairs = [(0, 1), (0, 2), (0, 3), (1, 2), (1, 3), (2, 3)]; + for code in 0..3usize.pow(6) { + for loops in [false, true] { + let mut edges = Vec::new(); + let mut rest = code; + for &pair in &pairs { + for _ in 0..rest % 3 { + edges.push(pair); + } + rest /= 3; + } + if loops { + edges.extend((0..4).map(|v| (v, v))); + } + let context = context(); + let result = k_core(&graph(4, &edges, Orientation::Undirected, &context)).unwrap(); + let expected = cores_by_definition(4, &edges); + assert_eq!(result.core_values(), expected, "edges {edges:?}"); + assert_eq!(result.degeneracy(), *expected.iter().max().unwrap()); + } + } + // Every simple graph on five nodes: 2^10. + let pairs: Vec<(usize, usize)> = (0..5) + .flat_map(|a| (a + 1..5).map(move |b| (a, b))) + .collect(); + for code in 0..1usize << pairs.len() { + let edges: Vec<_> = pairs + .iter() + .enumerate() + .filter(|(bit, _)| code >> bit & 1 == 1) + .map(|(_, &pair)| pair) + .collect(); + let context = context(); + let result = k_core(&graph(5, &edges, Orientation::Undirected, &context)).unwrap(); + assert_eq!( + result.core_values(), + cores_by_definition(5, &edges), + "{edges:?}" + ); + } +} + +#[test] +fn k_core_states_multigraph_and_self_loop_semantics() { + let context = context(); + // Three parallel edges make a 3-core of two nodes. + let parallel = k_core(&graph( + 2, + &[(0, 1), (0, 1), (1, 0)], + Orientation::Undirected, + &context, + )) + .unwrap(); + assert_eq!(parallel.core_values(), [3, 3]); + // Loops never help: a node with only loops has core 0, and a loop on a + // triangle's corner leaves the triangle a 2-core. + let loops = k_core(&graph( + 4, + &[(0, 0), (0, 0), (1, 2), (2, 3), (3, 1), (1, 1)], + Orientation::Undirected, + &context, + )) + .unwrap(); + assert_eq!(loops.core_values(), [0, 2, 2, 2]); +} + +#[test] +fn k_core_handles_empty_single_node_and_isolates() { + let context = context(); + let empty = k_core(&graph(0, &[], Orientation::Undirected, &context)).unwrap(); + assert!(empty.core_values().is_empty()); + assert_eq!(empty.degeneracy(), 0); + let single = k_core(&graph(1, &[], Orientation::Undirected, &context)).unwrap(); + assert_eq!(single.core_values(), [0]); + // A pendant path off a 4-clique, plus an isolate. + let edges = [ + (0, 1), + (0, 2), + (0, 3), + (1, 2), + (1, 3), + (2, 3), + (3, 4), + (4, 5), + ]; + let mixed = k_core(&graph(7, &edges, Orientation::Undirected, &context)).unwrap(); + assert_eq!(mixed.core_values(), [3, 3, 3, 3, 1, 1, 0]); + assert_eq!(mixed.degeneracy(), 3); +} + +#[test] +fn k_core_rejects_directed_projections_without_leaking_admission() { + for orientation in [Orientation::Outgoing, Orientation::Incoming] { + let context = context(); + let projection = graph(3, &[(0, 1), (1, 2)], orientation, &context); + let held = context.usage().unwrap().live_bytes; + assert!(matches!( + k_core(&projection), + Err(ProcedureError::InvalidArguments(message)) if message.contains("undirected") + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + } +} + +#[test] +fn k_core_observes_cancellation_and_budget_and_releases_scratch() { + let edges: Vec<(usize, usize)> = (0..4000).map(|i| (i, i + 1)).collect(); + + let context = context(); + let projection = graph(4001, &edges, Orientation::Undirected, &context); + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + k_core(&projection), + Err(ProcedureError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + // A budget that admits the projection but not the whole peel fails inside + // the kernel. The projection's own cost is measured, not guessed. + let projection_work = context.usage().unwrap().work_units; + let tight = ExecutionContext::new(ExecutionLimits { + memory_bytes: 8 * 1024 * 1024, + work_units: projection_work + 3000, + batch_rows: 1024, + deadline: None, + }) + .unwrap(); + let projection = graph(4001, &edges, Orientation::Undirected, &tight); + let held = tight.usage().unwrap().live_bytes; + assert!(matches!( + k_core(&projection), + Err(ProcedureError::BudgetExceeded { + resource: "work", + .. + }) + )); + assert_eq!(tight.usage().unwrap().live_bytes, held); +} + +#[cfg(feature = "arrow")] +#[test] +fn k_core_arrow_batches_keep_bounds_and_admission() { + use arrow_array::{Array, Int64Array, StringArray}; + let context = ExecutionContext::new(ExecutionLimits { + memory_bytes: 8 * 1024 * 1024, + work_units: 1_000_000, + batch_rows: 2, + deadline: None, + }) + .unwrap(); + let edges = [(0, 1), (1, 2), (2, 0), (2, 3)]; + let projection = graph(5, &edges, Orientation::Undirected, &context); + let mut cursor = k_core(&projection) + .unwrap() + .into_table() + .into_arrow_results(); + drop(projection); + + let mut ids = Vec::new(); + let mut cores = Vec::new(); + let mut retained = Vec::new(); + while let Some(batch) = cursor.next_batch().unwrap() { + let record = batch.record_batch(); + assert!(record.num_rows() <= 2); + let names: Vec<_> = record + .schema() + .fields() + .iter() + .map(|f| f.name().clone()) + .collect(); + assert_eq!(names, ["nodeId", "coreValue", "degeneracy"]); + let id = record + .column(0) + .as_any() + .downcast_ref::() + .unwrap(); + let core = record + .column(1) + .as_any() + .downcast_ref::() + .unwrap(); + let degeneracy = record + .column(2) + .as_any() + .downcast_ref::() + .unwrap(); + for row in 0..record.num_rows() { + ids.push(id.value(row).to_string()); + cores.push(core.value(row)); + assert_eq!(degeneracy.value(row), 2); + } + assert!(batch.reserved_bytes() > 0); + retained.push(batch); + } + assert_eq!(ids, ["n0", "n1", "n2", "n3", "n4"]); + assert_eq!(cores, [2, 2, 2, 1, 0]); + // Retained batches keep their admission after the cursor is gone. + drop(cursor); + let live = context.usage().unwrap().live_bytes; + assert!(live >= retained.iter().map(|b| b.reserved_bytes()).sum::()); + drop(retained); + assert_eq!(context.usage().unwrap().live_bytes, 0); +} diff --git a/crates/grust-algorithms/tests/label_propagation.rs b/crates/grust-algorithms/tests/label_propagation.rs new file mode 100644 index 00000000..5c4d122e --- /dev/null +++ b/crates/grust-algorithms/tests/label_propagation.rs @@ -0,0 +1,314 @@ +//! Label propagation: fixed points, planted cliques, schedule and limits. + +use grust_algorithms::{ + AlgorithmError, ExecutionContext, ExecutionLimits, GraphProjection, LabelPropagationOptions, + Orientation, ProjectionEdge, SnapshotIdentity, label_propagation, +}; +use std::collections::BTreeMap; + +const ORIENTATIONS: [Orientation; 3] = [ + Orientation::Outgoing, + Orientation::Incoming, + Orientation::Undirected, +]; + +fn limits(work_units: usize) -> ExecutionLimits { + ExecutionLimits { + memory_bytes: 64 * 1024 * 1024, + work_units, + batch_rows: 1024, + deadline: None, + } +} + +fn context() -> ExecutionContext { + ExecutionContext::new(limits(500_000_000)).unwrap() +} + +fn graph( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + weights.map(<[f64]>::to_vec), + orientation, + context, + ) + .unwrap() +} + +/// Weight arriving at each node per community, straight from the edge list. +fn arriving( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, + communities: &[usize], +) -> Vec> { + let mut support = vec![BTreeMap::new(); n]; + for (index, &(a, b)) in edges.iter().enumerate() { + let w = weights.map_or(1.0, |w| w[index]); + let mut arc = |from: usize, to: usize| { + *support[to].entry(communities[from]).or_insert(0.0) += w; + }; + match orientation { + Orientation::Outgoing => arc(a, b), + Orientation::Incoming => arc(b, a), + Orientation::Undirected => { + arc(a, b); + if a != b { + arc(b, a); + } + } + } + } + support +} + +struct Xorshift(u64); +impl Xorshift { + fn below(&mut self, bound: usize) -> usize { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 % bound as u64) as usize + } +} + +#[test] +fn a_converged_run_is_a_fixed_point_and_undirected_runs_always_converge() { + let mut random = Xorshift(0x2545_F491_4F6C_DD1D); + let mut converged_directed = 0; + for round in 0..3000 { + let n = 1 + random.below(12); + let count = random.below(3 * n); + let edges: Vec<_> = (0..count) + .map(|_| (random.below(n), random.below(n))) + .collect(); + let weights: Vec = (0..count).map(|_| random.below(4) as f64).collect(); + let weights = (round % 2 == 1).then_some(&weights[..]); + let seed = (round % 3 == 0).then_some(round as u64); + for orientation in ORIENTATIONS { + let context = context(); + let result = label_propagation( + &graph(n, &edges, weights, orientation, &context), + LabelPropagationOptions { + max_iterations: 1000, + seed, + }, + ) + .unwrap(); + let communities = result.communities(); + // Named by the smallest member, which is therefore a member. + for (node, &community) in communities.iter().enumerate() { + assert!(community <= node); + assert_eq!(communities[community], community); + } + if orientation == Orientation::Undirected { + assert!(result.converged(), "{edges:?} {weights:?}"); + } + if !result.converged() { + assert_eq!(result.iterations(), 1000); + continue; + } + converged_directed += usize::from(orientation != Orientation::Undirected); + // Nothing arriving at a node outweighs the community it holds. + let support = arriving(n, &edges, weights, orientation, communities); + for node in 0..n { + let held = support[node] + .get(&communities[node]) + .copied() + .unwrap_or(0.0); + let heaviest = support[node].values().copied().fold(0.0, f64::max); + assert!( + held >= heaviest, + "{orientation:?} {edges:?} {weights:?}: node {node} holds {held} of {heaviest}" + ); + } + } + } + assert!( + converged_directed > 1000, + "directed runs should mostly converge" + ); +} + +#[test] +fn disjoint_cliques_separate_and_isolates_stay_alone() { + let mut edges = Vec::new(); + for base in [0, 5, 10] { + for a in base..base + 5 { + for b in a + 1..base + 5 { + edges.push((a, b)); + } + } + } + for seed in [None, Some(1), Some(99)] { + let context = context(); + let result = label_propagation( + &graph(17, &edges, None, Orientation::Undirected, &context), + LabelPropagationOptions { + max_iterations: 10, + seed, + }, + ) + .unwrap(); + assert!(result.converged()); + let expected: Vec = (0..17) + .map(|node| if node < 15 { node / 5 * 5 } else { node }) + .collect(); + assert_eq!(result.communities(), expected, "seed {seed:?}"); + } + let context = context(); + let empty = label_propagation( + &graph(0, &[], None, Orientation::Undirected, &context), + LabelPropagationOptions::default(), + ) + .unwrap(); + assert!(empty.communities().is_empty() && empty.converged()); + assert_eq!(empty.iterations(), 1); +} + +#[test] +fn the_schedule_ties_and_direction_are_as_documented() { + let context = context(); + // A directed 3-cycle in row order: 0 takes 2's label, 1 takes it from 0, + // and the second pass changes nothing. + let cycle = [(0, 1), (1, 2), (2, 0)]; + let run = |max_iterations| { + label_propagation( + &graph(3, &cycle, None, Orientation::Outgoing, &context), + LabelPropagationOptions { + max_iterations, + seed: None, + }, + ) + .unwrap() + }; + let two = run(10); + assert_eq!(two.communities(), [0, 0, 0]); + assert_eq!((two.iterations(), two.converged()), (2, true)); + let one = run(1); + assert_eq!((one.iterations(), one.converged()), (1, false)); + + // Labels flow along arcs: a source with no in-arcs keeps its own label and + // gives it to what it points at. + let fan = label_propagation( + &graph(3, &[(0, 1), (0, 2)], None, Orientation::Outgoing, &context), + LabelPropagationOptions::default(), + ) + .unwrap(); + assert_eq!(fan.communities(), [0, 0, 0]); + let reversed = label_propagation( + &graph(3, &[(0, 1), (0, 2)], None, Orientation::Incoming, &context), + LabelPropagationOptions::default(), + ) + .unwrap(); + // Node 0 hears 1 and 2 equally and takes the smaller; nothing reaches 1 or 2. + assert_eq!(reversed.communities(), [0, 0, 2]); + + // Weight decides before the tie-break does, and a zero weight carries nothing. + let weighted = label_propagation( + &graph( + 4, + &[(1, 0), (2, 0), (3, 0)], + Some(&[1.0, 5.0, 0.0]), + Orientation::Outgoing, + &context, + ), + LabelPropagationOptions::default(), + ) + .unwrap(); + assert_eq!(weighted.communities(), [0, 1, 0, 3]); + + assert!(matches!( + label_propagation( + &graph(1, &[], None, Orientation::Undirected, &context), + LabelPropagationOptions { + max_iterations: 0, + seed: None + } + ), + Err(AlgorithmError::InvalidArguments(_)) + )); +} + +fn scrambled(n: usize, edges: usize) -> Vec<(usize, usize)> { + let mut random = Xorshift(0x9E37_79B9_7F4A_7C15); + (0..edges) + .map(|_| (random.below(n), random.below(n).pow(2) / n)) + .collect() +} + +#[test] +fn a_seed_reproduces_its_result_at_any_pool_width() { + let edges = scrambled(2000, 6000); + let run = |threads: usize, seed| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .unwrap(); + pool.install(|| { + let context = context(); + let projection = graph(2000, &edges, None, Orientation::Undirected, &context); + let before = context.usage().unwrap().work_units; + let result = label_propagation( + &projection, + LabelPropagationOptions { + max_iterations: 50, + seed, + }, + ) + .unwrap(); + ( + result.communities().to_vec(), + context.usage().unwrap().work_units - before, + ) + }) + }; + let first = run(1, Some(7)); + assert_eq!(run(8, Some(7)), first); + assert_ne!(run(1, Some(8)).0, first.0, "another seed, another order"); + assert_eq!(run(1, None), run(3, None)); +} + +#[test] +fn label_propagation_observes_cancellation_and_budget_and_releases_scratch() { + let edges = scrambled(2000, 20_000); + let context = context(); + let projection = graph(2000, &edges, None, Orientation::Outgoing, &context); + let projection_work = context.usage().unwrap().work_units; + context.cancel().unwrap(); + assert!(matches!( + label_propagation(&projection, LabelPropagationOptions::default()), + Err(AlgorithmError::Cancelled) + )); + + let tight = ExecutionContext::new(limits(2 * projection_work + 30_000)).unwrap(); + let projection = graph(2000, &edges, None, Orientation::Outgoing, &tight); + assert!(matches!( + label_propagation(&projection, LabelPropagationOptions::default()), + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + // The in-arcs a directed run builds stay with the projection; scratch does not. + drop(projection); + assert_eq!(tight.usage().unwrap().live_bytes, 0); +} diff --git a/crates/grust-algorithms/tests/leiden.rs b/crates/grust-algorithms/tests/leiden.rs new file mode 100644 index 00000000..7fc19940 --- /dev/null +++ b/crates/grust-algorithms/tests/leiden.rs @@ -0,0 +1,352 @@ +//! Leiden: every community connected, modularity honest, structure recovered. + +use grust_algorithms::{ + AlgorithmError, ExecutionContext, ExecutionLimits, GraphProjection, LeidenOptions, + LouvainOptions, Orientation, ProjectionEdge, SnapshotIdentity, leiden, louvain, +}; + +const ORIENTATIONS: [Orientation; 3] = [ + Orientation::Undirected, + Orientation::Outgoing, + Orientation::Incoming, +]; + +fn limits(work_units: usize) -> ExecutionLimits { + ExecutionLimits { + memory_bytes: 256 * 1024 * 1024, + work_units, + batch_rows: 1024, + deadline: None, + } +} + +fn context() -> ExecutionContext { + ExecutionContext::new(limits(2_000_000_000)).unwrap() +} + +type Edges = Vec<(usize, usize, f64)>; + +fn graph( + n: usize, + edges: &Edges, + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target, _))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + Some(edges.iter().map(|e| e.2).collect()), + orientation, + context, + ) + .unwrap() +} + +fn matrix(n: usize, edges: &Edges, orientation: Orientation) -> Vec> { + let mut a = vec![vec![0.0; n]; n]; + for &(s, t, w) in edges { + match orientation { + Orientation::Undirected => { + a[s][t] += w; + a[t][s] += w; + } + Orientation::Outgoing => a[s][t] += w, + Orientation::Incoming => a[t][s] += w, + } + } + a +} + +/// Q = (1/M) Σ_ij [A_ij − γ k_out_i k_in_j / M] δ(c_i, c_j), over the dense matrix. +fn modularity(a: &[Vec], communities: &[usize], gamma: f64) -> f64 { + let n = a.len(); + let total: f64 = a.iter().flatten().sum(); + if total == 0.0 { + return 0.0; + } + let k_out: Vec = a.iter().map(|row| row.iter().sum()).collect(); + let k_in: Vec = (0..n).map(|j| a.iter().map(|row| row[j]).sum()).collect(); + let mut q = 0.0; + for i in 0..n { + for j in 0..n { + if communities[i] == communities[j] { + q += a[i][j] - gamma * k_out[i] * k_in[j] / total; + } + } + } + q / total +} + +fn optimum(a: &[Vec]) -> f64 { + fn go(a: &[Vec], labels: &mut Vec, used: usize, best: &mut f64) { + if labels.len() == a.len() { + *best = best.max(modularity(a, labels, 1.0)); + return; + } + for label in 0..=used { + labels.push(label); + go(a, labels, used.max(label + 1), best); + labels.pop(); + } + } + let mut best = f64::NEG_INFINITY; + go(a, &mut Vec::new(), 0, &mut best); + best +} + +/// Communities that are not connected through positive-weight arcs, either way. +fn disconnected(n: usize, edges: &Edges, communities: &[usize]) -> usize { + let mut parent: Vec = (0..n).collect(); + fn find(parent: &mut [usize], mut x: usize) -> usize { + while parent[x] != x { + parent[x] = parent[parent[x]]; + x = parent[x]; + } + x + } + for &(s, t, w) in edges { + if w > 0.0 && communities[s] == communities[t] { + let (a, b) = (find(&mut parent, s), find(&mut parent, t)); + parent[a] = b; + } + } + // A connected community has exactly one root among its members. + let mut roots = vec![0usize; n]; + for v in 0..n { + if find(&mut parent, v) == v { + roots[communities[v]] += 1; + } + } + roots.iter().filter(|&&count| count > 1).count() +} + +struct Xorshift(u64); +impl Xorshift { + fn below(&mut self, bound: usize) -> usize { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 % bound as u64) as usize + } +} + +fn random_edges(random: &mut Xorshift, n: usize, count: usize) -> Edges { + (0..count) + .map(|_| { + // Zero weights too: such an arc must attach nothing. + (random.below(n), random.below(n), random.below(5) as f64) + }) + .collect() +} + +#[test] +fn every_community_is_connected_and_the_modularity_is_the_partitions_own() { + let mut random = Xorshift(0x2545_F491_4F6C_DD1D); + for round in 0..1500 { + let n = 1 + random.below(40); + let count = random.below(3 * n); + let edges = random_edges(&mut random, n, count); + let options = LeidenOptions { + seed: (round % 2 == 0).then_some(round as u64), + resolution: [1.0, 1.0, 0.5, 2.0][round % 4], + ..Default::default() + }; + for orientation in ORIENTATIONS { + let context = context(); + let result = leiden(&graph(n, &edges, orientation, &context), options).unwrap(); + let communities = result.communities(); + let what = format!("{orientation:?} {options:?} {edges:?}"); + assert_eq!(disconnected(n, &edges, communities), 0, "{what}"); + let a = matrix(n, &edges, orientation); + let truth = modularity(&a, communities, options.resolution); + assert!((result.modularity() - truth).abs() < 1e-12, "{what}"); + let singletons: Vec = (0..n).collect(); + assert!( + truth >= modularity(&a, &singletons, options.resolution) - 1e-12, + "{what}: below where it started" + ); + for (node, &community) in communities.iter().enumerate() { + assert!(community <= node && communities[community] == community); + } + } + } +} + +#[test] +fn leiden_never_exceeds_the_true_optimum_and_often_reaches_it() { + let mut random = Xorshift(0x9E37_79B9_7F4A_7C15); + let (mut runs, mut optimal) = (0, 0); + for _ in 0..120 { + let n = 4 + random.below(4); + let count = 4 + random.below(9); + let edges = random_edges(&mut random, n, count); + for orientation in ORIENTATIONS { + let context = context(); + let result = leiden( + &graph(n, &edges, orientation, &context), + LeidenOptions::default(), + ) + .unwrap(); + let best = optimum(&matrix(n, &edges, orientation)); + assert!( + result.modularity() <= best + 1e-12, + "{orientation:?} {edges:?}" + ); + runs += 1; + optimal += usize::from(result.modularity() >= best - 1e-12); + } + } + // A heuristic, so not always; but a refinement that merged too little would + // show up here as a collapse. + assert!(2 * optimal > runs, "optimal in only {optimal} of {runs}"); +} + +#[test] +fn leiden_recovers_planted_cliques_in_every_orientation() { + let mut edges = Edges::new(); + for base in [0, 6, 12, 18] { + for a in base..base + 6 { + for b in base..base + 6 { + if a < b { + edges.push((a, b, 1.0)); + edges.push((b, a, 1.0)); + } + } + } + } + // A ring of single bridges between the cliques. + for base in [0, 6, 12, 18] { + edges.push((base, (base + 6) % 24, 1.0)); + } + let expected: Vec = (0..24).map(|node| node / 6 * 6).collect(); + for orientation in ORIENTATIONS { + for seed in [None, Some(3), Some(11)] { + let context = context(); + let result = leiden( + &graph(24, &edges, orientation, &context), + LeidenOptions { + seed, + ..Default::default() + }, + ) + .unwrap(); + assert_eq!(result.communities(), expected, "{orientation:?} {seed:?}"); + assert!(result.converged()); + } + } +} + +#[test] +fn leiden_handles_the_degenerate_graphs_and_rejects_bad_options() { + let context = context(); + for (n, edges) in [ + (0, vec![]), + (1, vec![]), + (3, vec![]), + (2, vec![(0, 1, 0.0)]), + ] { + let result = leiden( + &graph(n, &edges, Orientation::Undirected, &context), + LeidenOptions::default(), + ) + .unwrap(); + assert_eq!(result.communities(), (0..n).collect::>()); + assert_eq!(result.modularity(), 0.0); + assert!(result.converged()); + } + assert!(matches!( + leiden( + &graph(2, &vec![(0, 1, 1.0)], Orientation::Undirected, &context), + LeidenOptions { + max_levels: 0, + ..Default::default() + } + ), + Err(AlgorithmError::InvalidArguments(_)) + )); +} + +#[test] +fn leiden_is_reproducible_and_stays_connected_on_a_larger_graph() { + let mut random = Xorshift(0xD1B5_4A32_D192_ED03); + let n = 3000; + let mut edges = Edges::new(); + // Thirty loose groups of a hundred, sparsely joined. + for _ in 0..24_000 { + let group = random.below(30) * 100; + edges.push((group + random.below(100), group + random.below(100), 1.0)); + } + for _ in 0..1500 { + edges.push((random.below(n), random.below(n), 1.0)); + } + let run = |threads: usize, seed| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .unwrap(); + pool.install(|| { + let context = context(); + let projection = graph(n, &edges, Orientation::Undirected, &context); + let options = LouvainOptions { + seed, + ..Default::default() + }; + let result = leiden(&projection, options).unwrap(); + let plain = louvain(&projection, options).unwrap(); + ( + result.communities().to_vec(), + result.modularity().to_bits(), + plain.modularity(), + ) + }) + }; + let first = run(1, Some(5)); + assert_eq!(run(8, Some(5)), first); + assert_eq!(disconnected(n, &edges, &first.0), 0); + let modularity = f64::from_bits(first.1); + // The planted structure is found, and Leiden is in Louvain's neighbourhood. + assert!(modularity > 0.8, "{modularity}"); + assert!( + (modularity - first.2).abs() < 0.05, + "{modularity} vs {}", + first.2 + ); +} + +#[test] +fn leiden_observes_cancellation_and_budget_and_releases_scratch() { + let mut random = Xorshift(0x9E37_79B9_7F4A_7C15); + let edges = random_edges(&mut random, 2000, 12_000); + let context = context(); + let projection = graph(2000, &edges, Orientation::Outgoing, &context); + let projection_work = context.usage().unwrap().work_units; + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + leiden(&projection, LeidenOptions::default()), + Err(AlgorithmError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + let tight = ExecutionContext::new(limits(projection_work + 40_000)).unwrap(); + let projection = graph(2000, &edges, Orientation::Outgoing, &tight); + let held = tight.usage().unwrap().live_bytes; + assert!(matches!( + leiden(&projection, LeidenOptions::default()), + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + assert_eq!(tight.usage().unwrap().live_bytes, held); +} diff --git a/crates/grust-algorithms/tests/louvain.rs b/crates/grust-algorithms/tests/louvain.rs new file mode 100644 index 00000000..dec052ee --- /dev/null +++ b/crates/grust-algorithms/tests/louvain.rs @@ -0,0 +1,417 @@ +//! Louvain against a dense-matrix modularity and the brute-force optimum. + +use grust_algorithms::{ + AlgorithmError, ExecutionContext, ExecutionLimits, GraphProjection, LouvainOptions, + Orientation, ProjectionEdge, SnapshotIdentity, louvain, +}; + +fn context() -> ExecutionContext { + ExecutionContext::new(ExecutionLimits { + memory_bytes: 64 * 1024 * 1024, + work_units: 500_000_000, + batch_rows: 1024, + deadline: None, + }) + .unwrap() +} + +type Edges = Vec<(usize, usize, f64)>; + +fn graph( + n: usize, + edges: &Edges, + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target, _))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + Some(edges.iter().map(|e| e.2).collect()), + orientation, + context, + ) + .unwrap() +} + +/// The adjacency matrix the orientation defines. Undirected: symmetric, and a +/// loop of weight w is 2w on the diagonal (Newman). Directed: A[i][j] is the +/// weight of arcs i -> j as traversed, so `Incoming` transposes. +fn matrix(n: usize, edges: &Edges, orientation: Orientation) -> Vec> { + let mut a = vec![vec![0.0; n]; n]; + for &(s, t, w) in edges { + match orientation { + Orientation::Undirected => { + a[s][t] += w; + a[t][s] += w; + } + Orientation::Outgoing => a[s][t] += w, + Orientation::Incoming => a[t][s] += w, + } + } + a +} + +/// Q = (1/M) Σ_ij [A_ij − γ k_out_i k_in_j / M] δ(c_i, c_j), straight from the +/// papers, over the dense matrix. Shares nothing with the kernel. +fn modularity(a: &[Vec], communities: &[usize], gamma: f64) -> f64 { + let n = a.len(); + let total: f64 = a.iter().flatten().sum(); + if total == 0.0 { + return 0.0; + } + let k_out: Vec = a.iter().map(|row| row.iter().sum()).collect(); + let k_in: Vec = (0..n).map(|j| a.iter().map(|row| row[j]).sum()).collect(); + let mut q = 0.0; + for i in 0..n { + for j in 0..n { + if communities[i] == communities[j] { + q += a[i][j] - gamma * k_out[i] * k_in[j] / total; + } + } + } + q / total +} + +/// The best modularity over every partition, by restricted-growth strings. +fn optimum(a: &[Vec], gamma: f64) -> f64 { + fn go(a: &[Vec], gamma: f64, labels: &mut Vec, used: usize, best: &mut f64) { + if labels.len() == a.len() { + *best = best.max(modularity(a, labels, gamma)); + return; + } + for label in 0..=used { + labels.push(label); + go(a, gamma, labels, used.max(label + 1), best); + labels.pop(); + } + } + let mut best = f64::NEG_INFINITY; + go(a, gamma, &mut Vec::new(), 0, &mut best); + best +} + +fn pseudo_random_edges(n: usize, count: usize, seed: u64) -> Edges { + let mut state = seed | 1; + let mut next = move || { + state ^= state << 13; + state ^= state >> 7; + state ^= state << 17; + state + }; + (0..count) + .map(|_| { + let s = (next() % n as u64) as usize; + let t = (next() % n as u64) as usize; + (s, t, 1.0 + (next() % 4) as f64) + }) + .collect() +} + +#[test] +fn louvain_matches_an_independent_oracle() { + // On many small graphs, loops and parallel edges included, in every + // orientation: the reported modularity is the partition's true modularity; + // it is at least where the search started (singletons) and never above the + // true optimum. Louvain is a heuristic, so equality with the optimum is not + // asserted here. + for orientation in [ + Orientation::Undirected, + Orientation::Outgoing, + Orientation::Incoming, + ] { + for seed in 1..60u64 { + let n = 4 + (seed % 4) as usize; + let edges = pseudo_random_edges(n, 4 + (seed % 9) as usize, seed * 7919); + let context = context(); + let result = louvain( + &graph(n, &edges, orientation, &context), + LouvainOptions::default(), + ) + .unwrap(); + let a = matrix(n, &edges, orientation); + let truth = modularity(&a, result.communities(), 1.0); + assert!( + (result.modularity() - truth).abs() < 1e-12, + "{orientation:?} seed {seed}: reported {} true {truth}", + result.modularity() + ); + let singletons: Vec = (0..n).collect(); + assert!( + truth >= modularity(&a, &singletons, 1.0) - 1e-12, + "{orientation:?} {seed}" + ); + assert!(truth <= optimum(&a, 1.0) + 1e-12, "{orientation:?} {seed}"); + // Canonical ids: a community is named by its smallest member. + for (node, &community) in result.communities().iter().enumerate() { + assert!(community <= node); + assert_eq!(result.communities()[community], community); + } + } + } +} + +fn two_cliques() -> Edges { + let mut edges = Vec::new(); + for base in [0, 5] { + for a in 0..5 { + for b in a + 1..5 { + edges.push((base + a, base + b, 1.0)); + } + } + } + edges.push((4, 5, 1.0)); + edges +} + +#[test] +fn louvain_recovers_planted_structure_and_reaches_the_optimum_there() { + let context = context(); + let edges = two_cliques(); + let result = louvain( + &graph(10, &edges, Orientation::Undirected, &context), + LouvainOptions::default(), + ) + .unwrap(); + assert_eq!(result.communities(), [0, 0, 0, 0, 0, 5, 5, 5, 5, 5]); + assert!(result.converged()); + assert!(result.levels() >= 1); + // All 115,975 partitions of ten nodes: here the heuristic is exactly optimal. + let a = matrix(10, &edges, Orientation::Undirected); + assert!((result.modularity() - optimum(&a, 1.0)).abs() < 1e-12); + + // Directed: two 4-cycles joined by one arc. + let mut directed: Edges = Vec::new(); + for base in [0, 4] { + for i in 0..4 { + directed.push((base + i, base + (i + 1) % 4, 1.0)); + } + } + directed.push((3, 4, 1.0)); + for orientation in [Orientation::Outgoing, Orientation::Incoming] { + let result = louvain( + &graph(8, &directed, orientation, &context), + LouvainOptions::default(), + ) + .unwrap(); + assert_eq!( + result.communities(), + [0, 0, 0, 0, 4, 4, 4, 4], + "{orientation:?}" + ); + } +} + +#[test] +fn louvain_states_multigraph_and_self_loop_semantics() { + let context = context(); + // Two nodes, a loop on each, one edge between. A = [[2,1],[1,2]], M = 6. + // Apart: Q = 2*(2/6 - (3/6)^2) = 1/6. Together: Q = 6/6 - 1 = 0. + // A loop counted once instead of twice would make "together" win. + let edges: Edges = vec![(0, 0, 1.0), (1, 1, 1.0), (0, 1, 1.0)]; + let result = louvain( + &graph(2, &edges, Orientation::Undirected, &context), + LouvainOptions::default(), + ) + .unwrap(); + assert_eq!(result.communities(), [0, 1]); + assert!((result.modularity() - 1.0 / 6.0).abs() < 1e-12); + + // Parallel edges add up: three unit edges behave as one edge of weight 3. + let parallel: Edges = vec![ + (0, 1, 1.0), + (0, 1, 1.0), + (1, 0, 1.0), + (1, 2, 1.0), + (2, 3, 3.0), + ]; + let merged: Edges = vec![(0, 1, 3.0), (1, 2, 1.0), (2, 3, 3.0)]; + let a = louvain( + &graph(4, ¶llel, Orientation::Undirected, &context), + LouvainOptions::default(), + ) + .unwrap(); + let b = louvain( + &graph(4, &merged, Orientation::Undirected, &context), + LouvainOptions::default(), + ) + .unwrap(); + assert_eq!(a.communities(), b.communities()); + assert_eq!(a.communities(), [0, 0, 2, 2]); + assert!((a.modularity() - b.modularity()).abs() < 1e-12); +} + +#[test] +fn louvain_handles_empty_single_node_isolates_and_weightless_graphs() { + let context = context(); + let empty = louvain( + &graph(0, &vec![], Orientation::Undirected, &context), + LouvainOptions::default(), + ) + .unwrap(); + assert!(empty.communities().is_empty()); + assert_eq!(empty.modularity(), 0.0); + assert!(empty.converged()); + + let isolates = louvain( + &graph(3, &vec![], Orientation::Outgoing, &context), + LouvainOptions::default(), + ) + .unwrap(); + assert_eq!(isolates.communities(), [0, 1, 2]); + assert_eq!(isolates.modularity(), 0.0); + + // All weights zero: no structure to find, nothing divides by zero. + let weightless: Edges = vec![(0, 1, 0.0), (1, 2, 0.0)]; + let result = louvain( + &graph(3, &weightless, Orientation::Undirected, &context), + LouvainOptions::default(), + ) + .unwrap(); + assert_eq!(result.communities(), [0, 1, 2]); + assert_eq!(result.modularity(), 0.0); + + // An edge plus an isolate: the isolate stays alone. + let result = louvain( + &graph(3, &vec![(0, 1, 1.0)], Orientation::Undirected, &context), + LouvainOptions::default(), + ) + .unwrap(); + assert_eq!(result.communities(), [0, 0, 2]); +} + +#[test] +fn resolution_moves_between_components_and_singletons() { + let context = context(); + let edges = two_cliques(); + let projection = graph(10, &edges, Orientation::Undirected, &context); + // gamma = 0 rewards only internal weight: each connected component merges. + let merged = louvain( + &projection, + LouvainOptions { + resolution: 0.0, + ..Default::default() + }, + ) + .unwrap(); + assert!(merged.communities().iter().all(|&c| c == 0)); + // A very large gamma makes every merge a loss. + let apart = louvain( + &projection, + LouvainOptions { + resolution: 1e6, + ..Default::default() + }, + ) + .unwrap(); + assert_eq!(apart.communities(), (0..10).collect::>()); + assert_eq!(apart.levels(), 0); +} + +#[test] +fn louvain_is_deterministic_under_a_fixed_seed_and_any_pool_width() { + let edges = pseudo_random_edges(300, 1500, 99); + let run = |seed: Option, threads: usize| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .unwrap(); + pool.install(|| { + let context = context(); + let projection = graph(300, &edges, Orientation::Undirected, &context); + let result = louvain( + &projection, + LouvainOptions { + seed, + ..Default::default() + }, + ) + .unwrap(); + (result.communities().to_vec(), result.modularity().to_bits()) + }) + }; + assert_eq!(run(None, 1), run(None, 8)); + assert_eq!(run(Some(7), 1), run(Some(7), 8)); + assert_eq!(run(Some(7), 1), run(Some(7), 1)); + // Different visiting orders are allowed to find different partitions; each + // must still be a real improvement over singletons. + let (_, q) = run(Some(8), 1); + assert!(f64::from_bits(q) > 0.2); +} + +#[test] +fn louvain_rejects_invalid_options_without_leaking_admission() { + let context = context(); + let projection = graph(3, &vec![(0, 1, 1.0)], Orientation::Undirected, &context); + let held = context.usage().unwrap().live_bytes; + for options in [ + LouvainOptions { + resolution: -1.0, + ..Default::default() + }, + LouvainOptions { + resolution: f64::NAN, + ..Default::default() + }, + LouvainOptions { + tolerance: -1e-9, + ..Default::default() + }, + LouvainOptions { + max_levels: 0, + ..Default::default() + }, + LouvainOptions { + max_iterations: 0, + ..Default::default() + }, + ] { + assert!(matches!( + louvain(&projection, options), + Err(AlgorithmError::InvalidArguments(_)) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + } +} + +#[test] +fn louvain_observes_cancellation_and_budget_and_releases_scratch() { + let edges = pseudo_random_edges(500, 4000, 5); + let context = context(); + let projection = graph(500, &edges, Orientation::Undirected, &context); + let projection_work = context.usage().unwrap().work_units; + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + louvain(&projection, LouvainOptions::default()), + Err(AlgorithmError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + let tight = ExecutionContext::new(ExecutionLimits { + memory_bytes: 64 * 1024 * 1024, + work_units: projection_work + 12_000, + batch_rows: 1024, + deadline: None, + }) + .unwrap(); + let projection = graph(500, &edges, Orientation::Undirected, &tight); + let held = tight.usage().unwrap().live_bytes; + assert!(matches!( + louvain(&projection, LouvainOptions::default()), + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + assert_eq!(tight.usage().unwrap().live_bytes, held); +} diff --git a/crates/grust-algorithms/tests/similarity.rs b/crates/grust-algorithms/tests/similarity.rs new file mode 100644 index 00000000..dadcf07c --- /dev/null +++ b/crates/grust-algorithms/tests/similarity.rs @@ -0,0 +1,394 @@ +//! Node similarity against an O(n^2) recomputation from the edge list. + +use grust_algorithms::{ + AlgorithmError, ExecutionContext, ExecutionLimits, GraphProjection, NodeSimilarityOptions, + Orientation, ProjectionEdge, SimilarityMetric, SnapshotIdentity, node_similarity, +}; +use std::collections::BTreeMap; + +const ORIENTATIONS: [Orientation; 3] = [ + Orientation::Outgoing, + Orientation::Incoming, + Orientation::Undirected, +]; +const METRICS: [SimilarityMetric; 3] = [ + SimilarityMetric::Jaccard, + SimilarityMetric::Overlap, + SimilarityMetric::Cosine, +]; + +fn limits(work_units: usize) -> ExecutionLimits { + ExecutionLimits { + memory_bytes: 256 * 1024 * 1024, + work_units, + batch_rows: 1024, + deadline: None, + } +} + +fn context() -> ExecutionContext { + ExecutionContext::new(limits(2_000_000_000)).unwrap() +} + +fn graph( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + weights.map(<[f64]>::to_vec), + orientation, + context, + ) + .unwrap() +} + +type Rows = Vec<(usize, usize, f64)>; + +/// Every ordered pair, from neighbour maps built straight off the edge list. +/// Weights in these tests are small integers, so every sum is exact and the +/// kernel must agree to the bit, order included. +fn by_definition( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, + options: NodeSimilarityOptions, +) -> Rows { + let mut sets = vec![BTreeMap::::new(); n]; + for (index, &(a, b)) in edges.iter().enumerate() { + let w = weights.map_or(1.0, |w| w[index]); + if a == b { + continue; + } + // Without weights a neighbour is in the set or not; with them, parallel + // edges add up. + let mut add = |from: usize, to: usize| { + let entry = sets[from].entry(to).or_insert(0.0); + *entry = if weights.is_some() { *entry + w } else { 1.0 }; + }; + if orientation != Orientation::Incoming { + add(a, b); + } + if orientation != Orientation::Outgoing { + add(b, a); + } + } + let admitted = |v: usize| { + sets[v].len() >= options.degree_cutoff + && options + .upper_degree_cutoff + .is_none_or(|upper| sets[v].len() <= upper) + }; + let mut rows = Rows::new(); + for a in (0..n).filter(|&a| admitted(a)) { + let mut mine = Rows::new(); + for b in (0..n).filter(|&b| b != a && admitted(b)) { + let (mut low, mut dot, mut any) = (0.0, 0.0, false); + for (key, &x) in &sets[a] { + if let Some(&y) = sets[b].get(key) { + any = true; + low += x.min(y); + dot += x * y; + } + } + if !any { + continue; + } + let total = |v: usize| sets[v].values().sum::(); + let norm = |v: usize| sets[v].values().map(|w| w * w).sum::().sqrt(); + let union = total(a) + total(b) - low; + let similarity = match options.metric { + SimilarityMetric::Jaccard => low / union, + SimilarityMetric::Overlap => low / total(a).min(total(b)), + SimilarityMetric::Cosine => dot / (norm(a) * norm(b)), + }; + if similarity > 0.0 && similarity.is_finite() && similarity >= options.similarity_cutoff + { + mine.push((a, b, similarity)); + } + } + mine.sort_by(|x, y| y.2.total_cmp(&x.2).then(x.1.cmp(&y.1))); + mine.truncate(options.top_k); + rows.extend(mine); + } + if options.top_n > 0 { + rows.sort_by(|x, y| y.2.total_cmp(&x.2).then(x.0.cmp(&y.0)).then(x.1.cmp(&y.1))); + rows.truncate(options.top_n); + } + rows +} + +fn run(projection: &GraphProjection, options: NodeSimilarityOptions) -> Rows { + let result = node_similarity(projection, options).unwrap(); + (0..result.first().len()) + .map(|row| { + ( + result.first()[row], + result.second()[row], + result.similarity()[row], + ) + }) + .collect() +} + +struct Xorshift(u64); +impl Xorshift { + fn below(&mut self, bound: usize) -> usize { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 % bound as u64) as usize + } +} + +#[test] +fn node_similarity_matches_the_definition_row_for_row() { + let mut random = Xorshift(0x2545_F491_4F6C_DD1D); + let mut rows_checked = 0; + for round in 0..1200 { + let n = 1 + random.below(12); + let count = random.below(4 * n); + let edges: Vec<_> = (0..count) + .map(|_| (random.below(n), random.below(n))) + .collect(); + let weights: Vec = (0..count).map(|_| random.below(4) as f64).collect(); + let weights = (round % 2 == 1).then_some(&weights[..]); + let options = NodeSimilarityOptions { + metric: METRICS[round % 3], + top_k: [1, 2, 3, 100][random.below(4)], + top_n: [0, 0, 1, 5][random.below(4)], + similarity_cutoff: [0.0, 0.0, 0.3, 0.5][random.below(4)], + degree_cutoff: 1 + random.below(2), + upper_degree_cutoff: [None, None, Some(3)][random.below(3)], + }; + for orientation in ORIENTATIONS { + let context = context(); + let projection = graph(n, &edges, weights, orientation, &context); + let actual = run(&projection, options); + let expected = by_definition(n, &edges, weights, orientation, options); + assert_eq!( + actual.len(), + expected.len(), + "{orientation:?} {options:?} {edges:?} {weights:?}" + ); + for (a, e) in actual.iter().zip(&expected) { + assert_eq!( + (a.0, a.1, a.2.to_bits()), + (e.0, e.1, e.2.to_bits()), + "{orientation:?} {options:?} {edges:?} {weights:?}: {a:?} vs {e:?}" + ); + } + rows_checked += actual.len(); + } + } + assert!( + rows_checked > 10_000, + "only {rows_checked} rows were compared" + ); +} + +#[test] +fn node_similarity_states_its_textbook_values_and_its_set_semantics() { + let context = context(); + // People 0,1,2 like items 3,4,5: 0 -> {3,4}, 1 -> {3,4,5}, 2 -> {5}. + // The doubled 0->3 and the loop on 2 change nothing: these are sets. + let edges = [ + (0, 3), + (0, 3), + (0, 4), + (1, 3), + (1, 4), + (1, 5), + (2, 5), + (2, 2), + ]; + let projection = graph(6, &edges, None, Orientation::Outgoing, &context); + let with = |metric| { + run( + &projection, + NodeSimilarityOptions { + metric, + ..Default::default() + }, + ) + }; + let third = 1.0 / 3.0; + assert_eq!( + with(SimilarityMetric::Jaccard), + [ + (0, 1, 2.0 / 3.0), + (1, 0, 2.0 / 3.0), + (1, 2, third), + (2, 1, third) + ] + ); + assert_eq!( + with(SimilarityMetric::Overlap), + [(0, 1, 1.0), (1, 0, 1.0), (1, 2, 1.0), (2, 1, 1.0)] + ); + let cosine = with(SimilarityMetric::Cosine); + assert_eq!(cosine[0].2, 2.0 / (2.0f64.sqrt() * 3.0f64.sqrt())); + + // topK is per node1, so a pair can survive from one side only. + let top = run( + &projection, + NodeSimilarityOptions { + top_k: 1, + ..Default::default() + }, + ); + assert_eq!(top, [(0, 1, 2.0 / 3.0), (1, 0, 2.0 / 3.0), (2, 1, third)]); + // topN orders by similarity, then node1, then node2. + let best = run( + &projection, + NodeSimilarityOptions { + top_n: 3, + ..Default::default() + }, + ); + assert_eq!(best, [(0, 1, 2.0 / 3.0), (1, 0, 2.0 / 3.0), (1, 2, third)]); + + // Weighted Jaccard sums the smaller weight over the larger. + let weighted = graph( + 3, + &[(0, 2), (1, 2), (1, 2)], + Some(&[1.0, 1.0, 2.0]), + Orientation::Outgoing, + &context, + ); + // 0 -> {2: 1}, 1 -> {2: 3}: min 1 over max 3. + assert_eq!( + run(&weighted, NodeSimilarityOptions::default()), + [(0, 1, third), (1, 0, third)] + ); +} + +#[test] +fn node_similarity_rejects_options_that_mean_nothing() { + let context = context(); + let projection = graph(2, &[(0, 1)], None, Orientation::Outgoing, &context); + let held = context.usage().unwrap().live_bytes; + for options in [ + NodeSimilarityOptions { + top_k: 0, + ..Default::default() + }, + NodeSimilarityOptions { + similarity_cutoff: 1.5, + ..Default::default() + }, + NodeSimilarityOptions { + similarity_cutoff: f64::NAN, + ..Default::default() + }, + NodeSimilarityOptions { + degree_cutoff: 0, + ..Default::default() + }, + NodeSimilarityOptions { + degree_cutoff: 3, + upper_degree_cutoff: Some(2), + ..Default::default() + }, + ] { + assert!(matches!( + node_similarity(&projection, options), + Err(AlgorithmError::InvalidArguments(_)) + )); + } + assert_eq!(context.usage().unwrap().live_bytes, held); +} + +fn scrambled(n: usize, edges: usize) -> Vec<(usize, usize)> { + let mut random = Xorshift(0x9E37_79B9_7F4A_7C15); + (0..edges) + .map(|_| (random.below(n), random.below(n).pow(2) / n)) + .collect() +} + +#[test] +fn node_similarity_is_identical_at_any_pool_width_and_charges_the_same_work() { + let edges = scrambled(600, 4000); + let weights: Vec = (0..edges.len()).map(|i| (1 + i % 5) as f64 / 3.0).collect(); + for metric in METRICS { + let run_with = |threads: usize| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .unwrap(); + pool.install(|| { + let context = context(); + let projection = + graph(600, &edges, Some(&weights), Orientation::Outgoing, &context); + let before = context.usage().unwrap().work_units; + let rows: Vec<_> = run( + &projection, + NodeSimilarityOptions { + metric, + top_k: 5, + ..Default::default() + }, + ) + .into_iter() + .map(|(a, b, s)| (a, b, s.to_bits())) + .collect(); + (rows, context.usage().unwrap().work_units - before) + }) + }; + let first = run_with(1); + assert!(first.0.len() > 1000); + // Both directions of a pair carry the same bits. + let lookup: BTreeMap<_, _> = first.0.iter().map(|&(a, b, s)| ((a, b), s)).collect(); + for (&(a, b), s) in &lookup { + if let Some(back) = lookup.get(&(b, a)) { + assert_eq!(s, back); + } + } + for threads in [2, 3, 8] { + assert_eq!(run_with(threads), first, "{threads} threads, {metric:?}"); + } + } +} + +#[test] +fn node_similarity_observes_cancellation_and_budget_and_releases_scratch() { + // A hub everyone points at: the quadratic case the work budget must stop. + let edges: Vec<_> = (1..3000).map(|node| (node, 0)).collect(); + let context = context(); + let projection = graph(3000, &edges, None, Orientation::Outgoing, &context); + let projection_work = context.usage().unwrap().work_units; + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + node_similarity(&projection, NodeSimilarityOptions::default()), + Err(AlgorithmError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + let tight = ExecutionContext::new(limits(projection_work + 1_000_000)).unwrap(); + let projection = graph(3000, &edges, None, Orientation::Outgoing, &tight); + let held = tight.usage().unwrap().live_bytes; + assert!(matches!( + node_similarity(&projection, NodeSimilarityOptions::default()), + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + assert_eq!(tight.usage().unwrap().live_bytes, held); +} diff --git a/crates/grust-algorithms/tests/spanning.rs b/crates/grust-algorithms/tests/spanning.rs new file mode 100644 index 00000000..06725060 --- /dev/null +++ b/crates/grust-algorithms/tests/spanning.rs @@ -0,0 +1,269 @@ +//! Spanning forests against an exhaustive search over edge subsets. + +use grust_algorithms::{ + AlgorithmError, ExecutionContext, ExecutionLimits, GraphProjection, Orientation, + ProjectionEdge, SnapshotIdentity, SpanningObjective, SpanningTreeOptions, spanning_tree, +}; + +fn limits(work_units: usize) -> ExecutionLimits { + ExecutionLimits { + memory_bytes: 256 * 1024 * 1024, + work_units, + batch_rows: 1024, + deadline: None, + } +} + +fn context() -> ExecutionContext { + ExecutionContext::new(limits(500_000_000)).unwrap() +} + +fn graph( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + // Descending, so "smaller ordinal" is not "earlier slot". + ordinal: usize::MAX / 2 - ordinal, + id: None, + }) + .collect(), + weights.map(<[f64]>::to_vec), + orientation, + context, + ) + .unwrap() +} + +fn find(parent: &mut [usize], mut x: usize) -> usize { + while parent[x] != x { + parent[x] = parent[parent[x]]; + x = parent[x]; + } + x +} + +/// Components, or `None` if the chosen edges contain a cycle. +fn forest_components(n: usize, edges: &[(usize, usize)], chosen: &[usize]) -> Option { + let mut parent: Vec = (0..n).collect(); + for &slot in chosen { + let (a, b) = ( + find(&mut parent, edges[slot].0), + find(&mut parent, edges[slot].1), + ); + if a == b { + return None; + } + parent[a] = b; + } + Some((0..n).filter(|&v| find(&mut parent, v) == v).count()) +} + +/// Of all spanning forests, the one whose edges, ranked by (weight, ordinal) and +/// sorted, come first: the greedy basis, found without being greedy. +fn by_search( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + maximum: bool, +) -> Vec { + let mut ranked: Vec = (0..edges.len()).collect(); + ranked.sort_by(|&a, &b| { + let (wa, wb) = (weights.map_or(1.0, |w| w[a]), weights.map_or(1.0, |w| w[b])); + let by_weight = if maximum { + wb.total_cmp(&wa) + } else { + wa.total_cmp(&wb) + }; + // Ordinals descend with the slot. + by_weight.then(b.cmp(&a)) + }); + let mut rank = vec![0; edges.len()]; + for (position, &slot) in ranked.iter().enumerate() { + rank[slot] = position; + } + let all: Vec = (0..edges.len()).collect(); + let pieces = { + let mut parent: Vec = (0..n).collect(); + for &(a, b) in edges { + let (a, b) = (find(&mut parent, a), find(&mut parent, b)); + parent[a] = b; + } + (0..n).filter(|&v| find(&mut parent, v) == v).count() + }; + let mut best: Option<(Vec, Vec)> = None; + for mask in 0u32..1 << edges.len() { + let chosen: Vec = all.iter().copied().filter(|s| mask >> s & 1 == 1).collect(); + if forest_components(n, edges, &chosen) != Some(pieces) { + continue; + } + let mut key: Vec = chosen.iter().map(|&slot| rank[slot]).collect(); + key.sort_unstable(); + if best.as_ref().is_none_or(|(least, _)| key < *least) { + best = Some((key, chosen)); + } + } + best.unwrap().1 +} + +struct Xorshift(u64); +impl Xorshift { + fn below(&mut self, bound: usize) -> usize { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 % bound as u64) as usize + } +} + +#[test] +fn the_forest_is_the_one_an_exhaustive_search_finds() { + let mut random = Xorshift(0x2545_F491_4F6C_DD1D); + for round in 0..1500 { + let n = 1 + random.below(7); + let count = random.below(11); + let edges: Vec<_> = (0..count) + .map(|_| (random.below(n), random.below(n))) + .collect(); + // Few distinct weights, so ties are the rule. + let weights: Vec = (0..count).map(|_| random.below(3) as f64).collect(); + let weights = (round % 3 != 0).then_some(&weights[..]); + for objective in [SpanningObjective::Minimum, SpanningObjective::Maximum] { + let context = context(); + let result = spanning_tree( + &graph(n, &edges, weights, Orientation::Undirected, &context), + SpanningTreeOptions { + objective, + source: None, + }, + ) + .unwrap(); + let expected = by_search(n, &edges, weights, objective == SpanningObjective::Maximum); + assert_eq!( + result.edges(), + expected, + "{objective:?} {edges:?} {weights:?}" + ); + let total: f64 = expected + .iter() + .map(|&s| weights.map_or(1.0, |w| w[s])) + .sum(); + assert_eq!(result.total_weight(), total); + assert_eq!(result.weights().len(), expected.len()); + } + } +} + +#[test] +fn a_source_keeps_the_tree_of_its_own_component() { + let context = context(); + // Two components: a weighted triangle 0-1-2 and an edge 3-4; 5 is alone. + let edges = [(0, 1), (1, 2), (2, 0), (3, 4)]; + let weights = [5.0, 1.0, 2.0, 7.0]; + let projection = graph(6, &edges, Some(&weights), Orientation::Undirected, &context); + let with = |objective, source| { + let result = spanning_tree(&projection, SpanningTreeOptions { objective, source }).unwrap(); + (result.edges().to_vec(), result.total_weight()) + }; + assert_eq!( + with(SpanningObjective::Minimum, None), + (vec![1, 2, 3], 10.0) + ); + assert_eq!( + with(SpanningObjective::Maximum, None), + (vec![0, 2, 3], 14.0) + ); + assert_eq!( + with(SpanningObjective::Minimum, Some("n2")), + (vec![1, 2], 3.0) + ); + assert_eq!(with(SpanningObjective::Minimum, Some("n4")), (vec![3], 7.0)); + assert_eq!(with(SpanningObjective::Minimum, Some("n5")), (vec![], 0.0)); + assert!(matches!( + spanning_tree( + &projection, + SpanningTreeOptions { + source: Some("nowhere"), + ..Default::default() + } + ), + Err(AlgorithmError::InvalidArguments(_)) + )); + assert!(matches!( + spanning_tree( + &graph(2, &[(0, 1)], None, Orientation::Outgoing, &context), + SpanningTreeOptions::default() + ), + Err(AlgorithmError::InvalidArguments(message)) if message.contains("undirected") + )); +} + +#[test] +fn the_forest_is_identical_at_any_pool_width_and_charges_the_same_work() { + let mut random = Xorshift(0x9E37_79B9_7F4A_7C15); + let n = 20_000; + let edges: Vec<_> = (0..120_000) + .map(|_| (random.below(n), random.below(n))) + .collect(); + let weights: Vec = (0..edges.len()).map(|_| random.below(50) as f64).collect(); + let run = |threads: usize| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .unwrap(); + pool.install(|| { + let context = context(); + let projection = graph(n, &edges, Some(&weights), Orientation::Undirected, &context); + let before = context.usage().unwrap().work_units; + let result = spanning_tree(&projection, SpanningTreeOptions::default()).unwrap(); + ( + result.edges().to_vec(), + result.total_weight().to_bits(), + context.usage().unwrap().work_units - before, + ) + }) + }; + let first = run(1); + assert!(first.0.len() > 19_000); + for threads in [2, 8] { + assert_eq!(run(threads), first, "{threads} threads"); + } +} + +#[test] +fn spanning_tree_observes_cancellation_and_budget_and_releases_scratch() { + let edges: Vec<_> = (1..5000).map(|node| (node / 2, node)).collect(); + let context = context(); + let projection = graph(5000, &edges, None, Orientation::Undirected, &context); + let projection_work = context.usage().unwrap().work_units; + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + spanning_tree(&projection, SpanningTreeOptions::default()), + Err(AlgorithmError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + let tight = ExecutionContext::new(limits(projection_work + 3000)).unwrap(); + let projection = graph(5000, &edges, None, Orientation::Undirected, &tight); + let held = tight.usage().unwrap().live_bytes; + assert!(matches!( + spanning_tree(&projection, SpanningTreeOptions::default()), + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + assert_eq!(tight.usage().unwrap().live_bytes, held); +} diff --git a/crates/grust-algorithms/tests/spectral.rs b/crates/grust-algorithms/tests/spectral.rs new file mode 100644 index 00000000..05a36fb5 --- /dev/null +++ b/crates/grust-algorithms/tests/spectral.rs @@ -0,0 +1,438 @@ +//! Eigenvector, Katz and HITS against dense linear algebra done in the test. + +use grust_algorithms::{ + AlgorithmError, ExecutionContext, ExecutionLimits, GraphProjection, IterationOptions, + KatzOptions, Orientation, ProjectionEdge, SnapshotIdentity, eigenvector, hits, katz, +}; + +const ORIENTATIONS: [Orientation; 3] = [ + Orientation::Outgoing, + Orientation::Incoming, + Orientation::Undirected, +]; + +fn limits(work_units: usize) -> ExecutionLimits { + ExecutionLimits { + memory_bytes: 256 * 1024 * 1024, + work_units, + batch_rows: 1024, + deadline: None, + } +} + +fn context() -> ExecutionContext { + ExecutionContext::new(limits(2_000_000_000)).unwrap() +} + +fn graph( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + weights.map(<[f64]>::to_vec), + orientation, + context, + ) + .unwrap() +} + +/// `m[v][u]`: total weight of arcs u -> v, so `m x` pulls along arcs. +fn pull_matrix( + n: usize, + edges: &[(usize, usize)], + weights: Option<&[f64]>, + orientation: Orientation, +) -> Vec> { + let mut m = vec![vec![0.0; n]; n]; + for (index, &(a, b)) in edges.iter().enumerate() { + let w = weights.map_or(1.0, |w| w[index]); + match orientation { + Orientation::Outgoing => m[b][a] += w, + Orientation::Incoming => m[a][b] += w, + Orientation::Undirected => { + m[b][a] += w; + if a != b { + m[a][b] += w; + } + } + } + } + m +} + +fn times(m: &[Vec], x: &[f64]) -> Vec { + m.iter() + .map(|row| row.iter().zip(x).map(|(a, b)| a * b).sum()) + .collect() +} + +fn transposed(m: &[Vec]) -> Vec> { + (0..m.len()) + .map(|i| (0..m.len()).map(|j| m[j][i]).collect()) + .collect() +} + +fn dot(a: &[f64], b: &[f64]) -> f64 { + a.iter().zip(b).map(|(x, y)| x * y).sum() +} + +/// `m x = λ x` with `λ = x·mx`, for a unit `x`; zero vectors pass trivially. +fn assert_eigenvector(m: &[Vec], x: &[f64], what: &str) { + let mx = times(m, x); + let lambda = dot(x, &mx); + for (row, (&left, &value)) in mx.iter().zip(x).enumerate() { + assert!( + (left - lambda * value).abs() <= 1e-6 * (1.0 + lambda), + "{what}: row {row}: (Mx) = {left}, λx = {}", + lambda * value + ); + } +} + +fn assert_unit_nonnegative(x: &[f64], what: &str) { + assert!(x.iter().all(|&v| v >= 0.0), "{what}: {x:?}"); + assert!((dot(x, x).sqrt() - 1.0).abs() < 1e-12, "{what}: {x:?}"); +} + +/// Solve `(I - alpha m) x = beta` by Gaussian elimination with pivoting. +fn katz_by_elimination(m: &[Vec], alpha: f64, beta: f64) -> Vec { + let n = m.len(); + let mut a: Vec> = (0..n) + .map(|i| { + let mut row: Vec = (0..n) + .map(|j| f64::from(u8::from(i == j)) - alpha * m[i][j]) + .collect(); + row.push(beta); + row + }) + .collect(); + for column in 0..n { + let pivot = (column..n) + .max_by(|&i, &j| a[i][column].abs().total_cmp(&a[j][column].abs())) + .unwrap(); + a.swap(column, pivot); + for row in 0..n { + if row != column { + let factor = a[row][column] / a[column][column]; + let pivot_row = a[column].clone(); + for (value, pivot) in a[row].iter_mut().zip(&pivot_row).skip(column) { + *value -= factor * pivot; + } + } + } + } + (0..n).map(|i| a[i][n] / a[i][i]).collect() +} + +struct Xorshift(u64); +impl Xorshift { + fn below(&mut self, bound: usize) -> usize { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + (self.0 % bound as u64) as usize + } +} + +#[test] +fn converged_runs_satisfy_their_defining_equations() { + let mut random = Xorshift(0x2545_F491_4F6C_DD1D); + let (mut eigen, mut hubs) = (0, 0); + for round in 0..600 { + let n = 1 + random.below(10); + let count = random.below(4 * n); + let edges: Vec<_> = (0..count) + .map(|_| (random.below(n), random.below(n))) + .collect(); + let weights: Vec = (0..count).map(|_| (1 + random.below(4)) as f64).collect(); + let weights = (round % 2 == 1).then_some(&weights[..]); + for orientation in ORIENTATIONS { + let what = format!("{orientation:?} {edges:?} {weights:?}"); + let context = context(); + let projection = graph(n, &edges, weights, orientation, &context); + let m = pull_matrix(n, &edges, weights, orientation); + + let result = eigenvector(&projection, IterationOptions::default()).unwrap(); + let x = result.values("score").unwrap(); + assert_unit_nonnegative(x, &what); + if result.converged() { + assert_eigenvector(&m, x, &what); + eigen += 1; + } else { + assert_eq!(result.iterations(), 1000, "{what}"); + } + + let result = hits(&projection, IterationOptions::default()).unwrap(); + let (hub, authority) = ( + result.values("hub").unwrap(), + result.values("authority").unwrap(), + ); + if count > 0 && m.iter().flatten().any(|&w| w > 0.0) { + assert_unit_nonnegative(hub, &what); + assert_unit_nonnegative(authority, &what); + } + if result.converged() { + // Authorities are an eigenvector of M Mᵀ, hubs of Mᵀ M. + let mt = transposed(&m); + // Column i of M Mᵀ is M times row i of M; both products are symmetric. + let mmt: Vec> = (0..n).map(|i| times(&m, &m[i])).collect(); + let mtm: Vec> = (0..n).map(|i| times(&mt, &mt[i])).collect(); + assert_eigenvector(&mmt, authority, &what); + assert_eigenvector(&mtm, hub, &what); + hubs += 1; + } + + // An alpha under the sufficient bound always converges to the solve. + let strongest = m + .iter() + .map(|row| row.iter().sum::()) + .fold(0.0, f64::max); + let alpha = 0.5 / strongest.max(1.0); + let options = KatzOptions { + alpha, + beta: 1.5, + ..Default::default() + }; + let result = katz(&projection, options).unwrap(); + assert!(result.converged(), "{what}"); + let expected = katz_by_elimination(&m, alpha, 1.5); + for (row, (a, e)) in result + .values("score") + .unwrap() + .iter() + .zip(&expected) + .enumerate() + { + assert!( + (a - e).abs() <= 1e-6 * (1.0 + e.abs()), + "{what}: row {row}: {a} vs {e}" + ); + } + } + } + assert!( + eigen > 800 && hubs > 1500, + "{eigen} eigenvector and {hubs} HITS runs converged" + ); +} + +#[test] +fn the_textbook_cases_are_as_documented() { + let context = context(); + // A star is bipartite: plain power iteration oscillates, the shifted one + // converges. Hub to leaf is sqrt(leaves) to one. + let star = [(0, 1), (0, 2), (0, 3), (0, 4)]; + let result = eigenvector( + &graph(5, &star, None, Orientation::Undirected, &context), + IterationOptions::default(), + ) + .unwrap(); + assert!(result.converged()); + let x = result.values("score").unwrap(); + assert!((x[0] / x[1] - 2.0).abs() < 1e-6, "{x:?}"); + assert!((x[1] - x[4]).abs() < 1e-12); + + // Directed: influence flows along arcs, so the pointed-at node gains. + let result = katz( + &graph(3, &[(0, 2), (1, 2)], None, Orientation::Outgoing, &context), + KatzOptions::default(), + ) + .unwrap(); + assert_eq!(result.values("score").unwrap(), [1.0, 1.0, 1.2]); + let normalized = katz( + &graph(3, &[(0, 2), (1, 2)], None, Orientation::Outgoing, &context), + KatzOptions { + normalized: true, + ..Default::default() + }, + ) + .unwrap(); + let x = normalized.values("score").unwrap(); + assert!((dot(x, x) - 1.0).abs() < 1e-12); + + // HITS: 0 and 1 point at 2. They are the hubs, 2 the authority. + let result = hits( + &graph(3, &[(0, 2), (1, 2)], None, Orientation::Outgoing, &context), + IterationOptions::default(), + ) + .unwrap(); + assert!(result.converged()); + let half = 0.5f64.sqrt(); + let close = |a: &[f64], e: [f64; 3]| a.iter().zip(e).all(|(a, e)| (a - e).abs() < 1e-9); + assert!(close(result.values("hub").unwrap(), [half, half, 0.0])); + assert!(close(result.values("authority").unwrap(), [0.0, 0.0, 1.0])); + + // No arcs: HITS is zero everywhere, eigenvector stays uniform, both settle. + let empty = graph(4, &[], None, Orientation::Outgoing, &context); + let result = hits(&empty, IterationOptions::default()).unwrap(); + assert!(result.converged()); + assert_eq!(result.values("authority").unwrap(), [0.0; 4]); + let result = eigenvector(&empty, IterationOptions::default()).unwrap(); + assert!(result.converged()); + assert_eq!(result.values("score").unwrap(), [0.5; 4]); +} + +#[test] +fn a_katz_alpha_that_is_too_large_is_reported_not_hidden() { + let context = context(); + // A triangle has λmax = 2, so alpha must be below 0.5. + let triangle = graph( + 3, + &[(0, 1), (1, 2), (2, 0)], + None, + Orientation::Undirected, + &context, + ); + let with = |alpha, max_iterations| { + katz( + &triangle, + KatzOptions { + alpha, + iteration: IterationOptions { + max_iterations, + ..Default::default() + }, + ..Default::default() + }, + ) + }; + assert!(with(0.45, 1000).unwrap().converged()); + let stalled = with(0.5, 50).unwrap(); + assert!(!stalled.converged() && stalled.iterations() == 50); + assert!(matches!( + with(3.0, 5000), + Err(AlgorithmError::Numerical(message)) if message.contains("alpha") + )); + for options in [ + KatzOptions { + alpha: 0.0, + ..Default::default() + }, + KatzOptions { + beta: f64::NAN, + ..Default::default() + }, + ] { + assert!(matches!( + katz(&triangle, options), + Err(AlgorithmError::InvalidArguments(_)) + )); + } + assert!(matches!( + eigenvector( + &triangle, + IterationOptions { + max_iterations: 0, + ..Default::default() + } + ), + Err(AlgorithmError::InvalidArguments(_)) + )); +} + +fn scrambled(n: usize, edges: usize) -> Vec<(usize, usize)> { + let mut random = Xorshift(0x9E37_79B9_7F4A_7C15); + (0..edges) + .map(|_| (random.below(n), random.below(n).pow(2) / n)) + .collect() +} + +#[test] +fn the_iterations_are_identical_at_any_pool_width_and_charge_the_same_work() { + // More than one chunk of 4096, so the chunked sums are exercised. + let n = 10_000; + let edges = scrambled(n, 60_000); + let weights: Vec = (0..edges.len()).map(|i| (1 + i % 7) as f64 / 3.0).collect(); + let run = |threads: usize| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .unwrap(); + pool.install(|| { + let context = context(); + let projection = graph(n, &edges, Some(&weights), Orientation::Outgoing, &context); + let before = context.usage().unwrap().work_units; + let options = IterationOptions { + max_iterations: 30, + ..Default::default() + }; + let results = [ + eigenvector(&projection, options).unwrap(), + hits(&projection, options).unwrap(), + katz( + &projection, + KatzOptions { + alpha: 0.001, + iteration: options, + ..Default::default() + }, + ) + .unwrap(), + ]; + let bits: Vec = results + .iter() + .flat_map(|result| { + ["score", "hub", "authority"] + .into_iter() + .filter_map(|name| result.values(name)) + .flatten() + .map(|v| v.to_bits()) + .chain([result.residual().to_bits(), result.iterations() as u64]) + .collect::>() + }) + .collect(); + (bits, context.usage().unwrap().work_units - before) + }) + }; + let first = run(1); + for threads in [2, 3, 8] { + assert_eq!(run(threads), first, "{threads} threads"); + } +} + +#[test] +fn the_iterations_observe_cancellation_and_budget_and_release_scratch() { + let edges = scrambled(5000, 40_000); + let context = context(); + let projection = graph(5000, &edges, None, Orientation::Undirected, &context); + let projection_work = context.usage().unwrap().work_units; + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + eigenvector(&projection, IterationOptions::default()), + Err(AlgorithmError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + let tight = ExecutionContext::new(limits(projection_work + 200_000)).unwrap(); + let projection = graph(5000, &edges, None, Orientation::Undirected, &tight); + let held = tight.usage().unwrap().live_bytes; + for outcome in [ + eigenvector(&projection, IterationOptions::default()).map(drop), + hits(&projection, IterationOptions::default()).map(drop), + ] { + assert!(matches!( + outcome, + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + } + assert_eq!(tight.usage().unwrap().live_bytes, held); +} diff --git a/crates/grust-algorithms/tests/triangles.rs b/crates/grust-algorithms/tests/triangles.rs new file mode 100644 index 00000000..e91e484b --- /dev/null +++ b/crates/grust-algorithms/tests/triangles.rs @@ -0,0 +1,296 @@ +//! Triangles and clustering coefficients against an O(n^3) definition. + +use grust_algorithms::{ + AlgorithmError, ExecutionContext, ExecutionLimits, GraphProjection, Orientation, + ProjectionEdge, SnapshotIdentity, TriangleOptions, triangles, +}; + +fn context() -> ExecutionContext { + ExecutionContext::new(ExecutionLimits { + memory_bytes: 64 * 1024 * 1024, + work_units: 200_000_000, + batch_rows: 1024, + deadline: None, + }) + .unwrap() +} + +fn graph( + n: usize, + edges: &[(usize, usize)], + orientation: Orientation, + context: &ExecutionContext, +) -> GraphProjection { + GraphProjection::from_topology( + SnapshotIdentity::new("g".into(), "r".into(), "reader".into()).unwrap(), + (0..n).map(|i| format!("n{i}").into()).collect(), + edges + .iter() + .enumerate() + .map(|(ordinal, &(source, target))| ProjectionEdge { + source, + target, + ordinal, + id: None, + }) + .collect(), + None, + orientation, + context, + ) + .unwrap() +} + +/// The definition: a triangle is three distinct, pairwise adjacent nodes. +/// Adjacency is a yes-or-no matrix, so multiplicity and loops cannot matter. +fn by_definition(n: usize, edges: &[(usize, usize)]) -> (Vec, Vec>, i64) { + let mut adjacent = vec![vec![false; n]; n]; + for &(a, b) in edges { + if a != b { + adjacent[a][b] = true; + adjacent[b][a] = true; + } + } + let mut counts = vec![0i64; n]; + let mut total = 0; + for a in 0..n { + for b in a + 1..n { + for c in b + 1..n { + if adjacent[a][b] && adjacent[b][c] && adjacent[a][c] { + total += 1; + for corner in [a, b, c] { + counts[corner] += 1; + } + } + } + } + } + let coefficients = (0..n) + .map(|v| { + let d = adjacent[v].iter().filter(|&&x| x).count() as f64; + (d >= 2.0).then(|| 2.0 * counts[v] as f64 / (d * (d - 1.0))) + }) + .collect(); + (counts, coefficients, total) +} + +fn assert_matches(n: usize, edges: &[(usize, usize)]) { + let context = context(); + let result = triangles( + &graph(n, edges, Orientation::Undirected, &context), + TriangleOptions::default(), + ) + .unwrap(); + let (counts, coefficients, total) = by_definition(n, edges); + assert_eq!(result.triangles(), counts, "{edges:?}"); + assert_eq!(result.triangle_count(), total, "{edges:?}"); + for (v, expected) in coefficients.iter().enumerate() { + match expected { + Some(value) => assert!( + (result.coefficients()[v] - value).abs() < 1e-12, + "{edges:?}" + ), + None => assert!(result.coefficients()[v].is_nan(), "{edges:?}"), + } + } +} + +#[test] +fn triangles_match_an_independent_oracle() { + // Every simple graph on six nodes: 2^15 = 32,768. + let pairs: Vec<(usize, usize)> = (0..6) + .flat_map(|a| (a + 1..6).map(move |b| (a, b))) + .collect(); + for code in 0..1usize << pairs.len() { + let edges: Vec<_> = pairs + .iter() + .enumerate() + .filter(|(bit, _)| code >> bit & 1 == 1) + .map(|(_, &pair)| pair) + .collect(); + assert_matches(6, &edges); + } +} + +#[test] +fn triangles_state_multigraph_and_self_loop_semantics() { + // Every four-node multigraph with multiplicity 0..=2, in both edge + // directions, with loops: the simple graph underneath decides. + let pairs = [(0, 1), (0, 2), (0, 3), (1, 2), (1, 3), (2, 3)]; + for code in 0..3usize.pow(6) { + let mut edges = vec![(0, 0), (3, 3)]; + let mut rest = code; + for &(a, b) in &pairs { + for copy in 0..rest % 3 { + edges.push(if copy == 0 { (a, b) } else { (b, a) }); + } + rest /= 3; + } + assert_matches(4, &edges); + } + // One triangle drawn with doubled edges and loops is still one triangle. + let context = context(); + let doubled = [(0, 1), (1, 0), (1, 2), (1, 2), (2, 0), (0, 0), (2, 2)]; + let result = triangles( + &graph(3, &doubled, Orientation::Undirected, &context), + TriangleOptions::default(), + ) + .unwrap(); + assert_eq!(result.triangles(), [1, 1, 1]); + assert_eq!(result.triangle_count(), 1); + assert_eq!(result.coefficients(), [1.0, 1.0, 1.0]); +} + +#[test] +fn triangles_handle_empty_single_node_and_isolates() { + let context = context(); + let empty = triangles( + &graph(0, &[], Orientation::Undirected, &context), + TriangleOptions::default(), + ) + .unwrap(); + assert!(empty.triangles().is_empty()); + assert_eq!(empty.triangle_count(), 0); + assert_eq!(empty.average_coefficient(), 0.0); + + // A triangle with a pendant, plus an isolate. + let result = triangles( + &graph( + 5, + &[(0, 1), (1, 2), (2, 0), (2, 3)], + Orientation::Undirected, + &context, + ), + TriangleOptions::default(), + ) + .unwrap(); + assert_eq!(result.triangles(), [1, 1, 1, 0, 0]); + assert_eq!(result.coefficients()[0], 1.0); + assert!((result.coefficients()[2] - 1.0 / 3.0).abs() < 1e-12); + assert!(result.coefficients()[3].is_nan() && result.coefficients()[4].is_nan()); + assert!((result.average_coefficient() - (1.0 + 1.0 + 1.0 / 3.0) / 3.0).abs() < 1e-12); +} + +#[test] +fn max_degree_leaves_hubs_and_their_triangles_out() { + let context = context(); + // Node 0 is a hub closing triangles 0-1-2 and 0-3-4; 5-6-7 stands alone. + let edges = [ + (0, 1), + (0, 2), + (1, 2), + (0, 3), + (0, 4), + (3, 4), + (5, 6), + (6, 7), + (7, 5), + ]; + let result = triangles( + &graph(8, &edges, Orientation::Undirected, &context), + TriangleOptions { + max_degree: Some(2), + }, + ) + .unwrap(); + assert_eq!(result.triangles(), [-1, 0, 0, 0, 0, 1, 1, 1]); + assert!(result.coefficients()[0].is_nan()); + assert_eq!(result.triangle_count(), 1); +} + +#[test] +fn triangles_reject_directed_projections_without_leaking_admission() { + for orientation in [Orientation::Outgoing, Orientation::Incoming] { + let context = context(); + let projection = graph(3, &[(0, 1), (1, 2), (2, 0)], orientation, &context); + let held = context.usage().unwrap().live_bytes; + assert!(matches!( + triangles(&projection, TriangleOptions::default()), + Err(AlgorithmError::InvalidArguments(message)) if message.contains("undirected") + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + } +} + +/// A deterministic pseudo-random graph with hubs, so blocks are uneven. +fn scrambled(n: usize, edges: usize) -> Vec<(usize, usize)> { + let mut state = 0x9E37_79B9_7F4A_7C15u64; + let mut next = move || { + state ^= state << 13; + state ^= state >> 7; + state ^= state << 17; + state + }; + (0..edges) + .map(|_| { + let a = (next() % n as u64) as usize; + // Squaring skews endpoints toward low rows: a few heavy nodes. + let b = ((next() % n as u64).pow(2) / n as u64) as usize; + (a, b) + }) + .collect() +} + +#[test] +fn triangles_are_identical_at_any_pool_width_and_charge_the_same_work() { + let edges = scrambled(400, 6000); + let run = |threads: usize| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .unwrap(); + pool.install(|| { + let context = context(); + let projection = graph(400, &edges, Orientation::Undirected, &context); + let before = context.usage().unwrap().work_units; + let result = triangles(&projection, TriangleOptions::default()).unwrap(); + let work = context.usage().unwrap().work_units - before; + (result.triangles().to_vec(), result.triangle_count(), work) + }) + }; + let (counts, total, work) = run(1); + assert!(total > 0, "the fixture should contain triangles"); + let (expected, _, expected_total) = by_definition(400, &edges); + assert_eq!(counts, expected); + assert_eq!(total, expected_total); + for threads in [2, 3, 8] { + let (other, other_total, other_work) = run(threads); + assert_eq!(other, counts, "{threads} threads"); + assert_eq!(other_total, total); + // Block buffers aside, the counted work is the graph's, not the pool's. + assert_eq!(other_work, work, "{threads} threads charged differently"); + } +} + +#[test] +fn triangles_observe_cancellation_and_budget_and_release_scratch() { + let edges = scrambled(300, 5000); + let context = context(); + let projection = graph(300, &edges, Orientation::Undirected, &context); + let projection_work = context.usage().unwrap().work_units; + let held = context.usage().unwrap().live_bytes; + context.cancel().unwrap(); + assert!(matches!( + triangles(&projection, TriangleOptions::default()), + Err(AlgorithmError::Cancelled) + )); + assert_eq!(context.usage().unwrap().live_bytes, held); + + let tight = ExecutionContext::new(ExecutionLimits { + memory_bytes: 64 * 1024 * 1024, + work_units: projection_work + 20_000, + batch_rows: 1024, + deadline: None, + }) + .unwrap(); + let projection = graph(300, &edges, Orientation::Undirected, &tight); + let held = tight.usage().unwrap().live_bytes; + assert!(matches!( + triangles(&projection, TriangleOptions::default()), + Err(AlgorithmError::BudgetExceeded { + resource: "work", + .. + }) + )); + assert_eq!(tight.usage().unwrap().live_bytes, held); +} diff --git a/crates/grust/Cargo.toml b/crates/grust/Cargo.toml index 79e6d2d7..c97d98a2 100644 --- a/crates/grust/Cargo.toml +++ b/crates/grust/Cargo.toml @@ -16,7 +16,7 @@ name = "grust" [features] default = [] algorithms = ["dep:grust-algorithms", "dep:grust-algorithm-procedures", "dep:grust-procedures"] -arrow = ["dep:grust-arrow", "grust-algorithms?/arrow"] +arrow = ["dep:grust-arrow", "grust-algorithms?/arrow", "grust-algorithm-procedures?/arrow"] adbc = ["arrow", "grust-arrow/adbc"] datafusion = ["arrow", "dep:grust-datafusion"] cocoindex = ["dep:grust-cocoindex"] diff --git a/docs/goals/graph-analytics-catalog.md b/docs/goals/graph-analytics-catalog.md index 7b63cb31..ceb7e260 100644 --- a/docs/goals/graph-analytics-catalog.md +++ b/docs/goals/graph-analytics-catalog.md @@ -561,20 +561,128 @@ M4 (Tier A), then per milestone.** Each is a minor version: new public API. not hidden. - **Two implementations.** See D1. -## Open decisions for the operator - -- **D1** — one implementation in Grust, icecat ports only as differential - checks? (recommended: yes) -- **D2** — Grust option names with a GDS alias table applied by Nutmeg? - (recommended: yes) -- **D3** — Louvain/Leiden/triangles/k-core/bridges require an `Undirected` - projection in v1 and reject others? (recommended: yes; directed modularity - later) -- **D4** — Dinic for max flow with Edmonds–Karp as its oracle? (recommended) -- **D5** — `rayon` behind a `parallel` feature, thread count in - `ExecutionLimits`, decided at M2 with measurements? (recommended) -- **D6** — group 35: port trainers, or hand features to an ML library under - Sail? (recommended: decide at M9, lean to the latter) +## Decisions (operator, 2026-09-20) + +- **D1 — one implementation, in `grust-algorithms`.** icecat gets a port only + where the benchmark wants a second independent implementation. +- **D2 — Grust option names.** Nutmeg applies the GDS alias table. +- **D3 — directed modularity now.** Louvain and Leiden accept every + orientation: undirected modularity on `Undirected` projections, Leicht–Newman + directed modularity otherwise. Triangles, k-core and the bridges family are + defined on undirected graphs only and still reject other orientations. This + supersedes the "require `Undirected` in v1" sentences in groups 1 and 7. +- **D4 — Dinic for max flow**, Edmonds–Karp as its oracle (recommendation taken). +- **D5 — rayon from the start.** Kernels are written parallel where the + algorithm allows, behind a `parallel` feature that is on by default, with the + thread count taken from `ExecutionLimits`. Rule 4 still binds: a kernel whose + parallel schedule cannot be made deterministic runs that phase sequentially. + Every parallel kernel keeps a one-thread path, and the one-thread result is + the oracle for the many-thread result. This supersedes P7's "sequential + first". +- **D6 — group 35 decided at M9** (recommendation taken). + +## Notes from the build + +- **`NodeTable` (added with group 5) replaces P3's bespoke community type and + every other per-node result type.** A kernel returns a typed wrapper for Rust + callers (`KCore::core_values()`) and `into_table()` for the adapters; the Arrow + and row adapters have one arm for all of them. Communities are a `Node` column + (the canonical member's row). Pair-shaped results (node similarity, link + prediction) and edge-shaped results (bridges, spanning forest, flow) need the + same treatment once: a `PairTable` and an `EdgeTable`. +- **`Meter`** (`meter.rs`) is the block-charging helper rule 5 asks for: `tick` + in the loop, `flush` before returning. +- **Thread count comes from the caller's rayon pool**, not from + `ExecutionLimits`: adding a field there breaks 33 construction sites and every + downstream crate. A caller installs a pool; rule 4 makes the result identical + at any width, and tests run each parallel kernel in a one-thread and a + many-thread pool. +- **Undirected-only kernels refuse a directed projection with a message that + contains "undirected".** Nutmeg's schema probe and Grust's catalog test both + rely on that word to retry on an undirected projection. Keep it. +- `run_on_projection` matches names case-insensitively, as the registry does. +- **Louvain's move phase is sequential by rule 4.** NetworKit's PLM moves nodes + asynchronously in parallel and is not reproducible. The kernel visits nodes in + row order or a seeded order and applies one move at a time. What can be + parallel later without touching the result: strength sums, coarsening's + per-community aggregation, the final modularity pass. Leiden (group 7) reuses + `Level`, `move_nodes`, `coarsen` and `modularity_of`; make them `pub(crate)` + there. `modularity_of` is group 22's kernel. `includeIntermediateCommunities` + waits for a list column on `NodeTable`. +- **`parallel.rs` is the pattern for every parallel kernel**: split nodes into + contiguous blocks of equal estimated work (`balanced_ranges`), run one task per + block (`map_ranges`), combine in block order. Cross-block writes go through + atomic counters (`Buffer::filled_with`), never per-task vectors, so memory and + charged work do not depend on the pool width. Each task owns a `Meter`. + Triangles proves it: identical counts *and* identical work at 1, 2, 3, 8 threads. +- **Per-source kernels use `parallel::ordered_blocks`.** Betweenness splits its + sources into fixed blocks of 64, runs `width()` blocks at a time, and merges + per-block vectors in block order: float sums and charged work are + bit-identical at any width, but live memory is one workspace per running + block, so it does grow with the pool. Closeness and harmonic (group 6) should + reuse it and `shortest::MinHeap` (now `pub(crate)`). Betweenness needs only + out-arcs: the backward pass rescans them instead of storing predecessors. + GDS's `samplingSeed` maps to `seed`; GDS does not rescale a sampled score and + Grust does, which Nutmeg's alias layer must state. +- **Label propagation keeps the current label on a tie**, then the smallest. + The plan said "ties to the smallest"; keeping the current label is what makes + the undirected convergence argument go through (each change strictly raises + same-label edge weight), and the test asserts that convergence. It is the + first user of `GraphProjection::incoming()`. +- **Pair results are a keyed `NodeTable`, not a new `PairTable`.** + `NodeTable::keyed(graph, "node1", rows)` makes a table of any length led by a + node column the kernel names; every later pair or list kernel (link + prediction, K-nearest neighbours, bridges, articulation points) uses it. A + zero-row table emits one empty Arrow batch so the schema is never lost. +- **Node similarity dropped `candidateLimit`.** Every triple is charged, so the + work budget is already that guard. Without weights a neighbour set is a true + set; the oracle first shared the kernel's mistake of summing parallel unit + weights, and only the hand-computed case caught it. Keep hand-computed cases + beside every oracle. +- **Edge-shaped rows lead with `sourceNodeId`.** A bare `edgeOrdinal` means + nothing to a Cypher caller, and a keyed table needs a node first, so bridges + and biconnected components return `sourceNodeId`, `targetNodeId`, + `edgeOrdinal` (then `componentId`). Spanning tree and max flow should do the + same. +- **Iterative kernels pull, in fixed chunks.** `parallel::for_chunks` cuts a + vector into chunks of 4,096 whatever the pool, each node pulls its next value + from `incoming()`, and per-chunk partial sums are folded in chunk order. That + is what makes a float reduction bit-identical at any width; `balanced_ranges` + cannot, because its boundaries move with the width. `articleRank` is not done: + it is an option inside `pagerank.rs`, which is push-based and sequential and + which the Nutmeg agent may be parallelising. Do it when that file's owner is + settled, preferably by moving PageRank onto `spectral::pull`. +- **Leiden returns the refined partition, always.** In the paper the final + answer is the move phase's partition of the last aggregate graph, which is + connected only because the loop ran until that partition was all singletons. + Under a level cap that is not so; returning the composition of refined + partitions keeps the guarantee unconditionally. Refinement is greedy (no + `theta`). `louvain.rs` exposes `Level`, `move_nodes` (now with an initial + partition), `renumber`, `coarsen`, `validate` and `finish` as `pub(super)`. +- **Two-ended kernels use `Spec::with_target()`**: a second positional String + argument after `source`. Schema probes must pass different nodes for the two + (Grust's catalog test and Nutmeg's `probe_args` now do). Max flow did not need + P2's reverse-arc index: it builds its own residual CSR, grouped by tail, in + which each arc knows its pair. +- **Vector results are `NodeColumn::Vector { values, dimension }`**: row-major + `f32`, Arrow `FixedSizeList`, `Value::FloatArray` to Cypher. Node2Vec, + HashGNN, GraphSAGE and k-means centroids reuse it. A variable-length list + column (SLLPA's `communityIds`, Louvain's intermediate communities) is still + missing. +- **Group 9 is blocked on a decision, not on effort.** Bellman–Ford needs a + projection that admits negative weights. Today `from_buffers` rejects them and + every kernel relies on that. Admitting them means (1) a way to ask for it — + a new field on `ProjectionOptions` breaks every literal construction of that + public struct, in Nutmeg too, so prefer a new `WeightSelection::SignedProperty` + variant; (2) a `signed` flag on `GraphProjection`; and (3) **every existing + kernel refusing a signed projection**, which edits the ten original kernel + files whose ownership is open in `codex-to-codex.md`. The safe central form + of (3) is one `GraphProjection::require_nonnegative()` called first in each + kernel, with a catalog test that runs every registered kernel on a signed + projection and demands `InvalidArguments` from all but `bellmanFord`. The + negative cycle is a result with a witness (P4), not an error. A\* needs P6 + (latitude/longitude node properties) to be registered at all; a Rust-only + `astar` over `Fn(usize) -> f64` is possible now but reaches no caller. ## Progress ledger @@ -582,11 +690,11 @@ Update in the same commit as the work. `—` not started, `wip`, `done ` | Item | State | Item | State | Item | State | | --- | --- | --- | --- | --- | --- | -| P1 | — | P2 | — | P3 | — | -| P4 | — | P5 | — | P6 | — | -| P7 | — | 1 Louvain | — | 2 Betweenness | — | -| 3 Node similarity | — | 4 Triangles/LCC | — | 5 k-core | — | -| 6 Closeness/harmonic | — | 7 Leiden | — | 8 Label propagation | — | -| 9 A\*/Bellman–Ford | — | 10 Eigenvector family | — | 11 Bridges family | — | -| 12 Spanning forest | — | 13 Max flow | — | 14 FastRP | — | +| P1 | done (Grust and Nutmeg) | P2 | done (`incoming()`) | P3 | done (`NodeTable`) | +| P4 | — | P5 | done (`random.rs`) | P6 | — | +| P7 | done (`parallel.rs`) | 1 Louvain | done | 2 Betweenness | done | +| 3 Node similarity | done | 4 Triangles/LCC | done | 5 k-core | done | +| 6 Closeness/harmonic | done | 7 Leiden | done | 8 Label propagation | done | +| 9 A\*/Bellman–Ford | blocked: see notes | 10 Eigenvector family | done except `articleRank` | 11 Bridges family | done | +| 12 Spanning forest | done | 13 Max flow | done (`maxFlow`, `minCut`) | 14 FastRP | done | | 15–35 | — (see Tier B) | | | | |