Skip to content

tensor4all/strided-rs-benchmark-suite

BranchesTags

Repository files navigation

strided-rs-benchmark-suite

Benchmark suite for strided-rs based on the einsum benchmark (168 standardized einsum problems across 7 categories).

Overview

This repository provides:

  • A Python pipeline to extract einsum benchmark metadata (shapes, dtypes, contraction paths) into portable JSON
  • A Rust benchmark runner using strided-opteinsum
  • A Julia benchmark runner using OMEinsum.jl

Only metadata is stored — tensors are generated at benchmark time (zero-filled), keeping the repo lightweight.

See tensor4all/strided-rs#63 for the full design discussion.

Benchmark Guides

  • Benchmark index: entry point for all published benchmark result pages.
  • Einsum benchmarks: full-suite strided-opteinsum versus OMEinsum.jl results and focused einsum case studies.
  • Strided benchmarks: kernel-level comparisons for naive, strided-rs, and HPTT where directly comparable.

Project Structure

strided-rs-benchmark-suite/
  benchmarks/
    README.md               # Index for benchmark result pages
    einsum_benchmarks/      # Full-suite einsum results and TN light 415 late-step case study
    strided_benchmarks/     # Strided kernel benchmarks and comparisons
  src/
    main.rs                 # Rust benchmark runner (strided-opteinsum)
    main.jl                 # Julia benchmark runner (OMEinsum.jl)
  scripts/
    run_all.sh              # Run all benchmarks (configurable thread count)
    generate_dataset.py     # Filter & export benchmark instances as JSON
    convert_tensornetwork.py # Convert TensorNetworkBenchmarks format to strided-rs JSON
    create_lightweight_instance.py # Extract BFS-connected subgraph for lighter tensor network
    format_results.py       # Parse logs and output markdown tables
  data/
    instances/              # Exported JSON metadata (one file per instance)
    results/                # Benchmark logs and markdown results
  Cargo.toml                # Rust project
  Project.toml              # Julia project
  pyproject.toml            # Python project

Setup

Python (dataset export)

Requires Python 3.12+ and uv.

uv sync

Rust

Requires a local clone of strided-rs at ../strided-rs.

Two GEMM backends are supported (mutually exclusive):

  • faer (default) — pure Rust GEMM via faer
  • blas — links to OpenBLAS (>= 0.3.29 recommended)

OpenBLAS setup:

  • macOS: brew install openblas
  • Ubuntu: build from source (system libopenblas-dev on Ubuntu 20.04 is 0.3.8 — too old for fair benchmarks)
# Build OpenBLAS 0.3.29 from source (one-time)
curl -L -o /tmp/OpenBLAS-0.3.29.tar.gz \
  https://github.com/OpenMathLib/OpenBLAS/releases/download/v0.3.29/OpenBLAS-0.3.29.tar.gz
cd /tmp && tar xf OpenBLAS-0.3.29.tar.gz && cd OpenBLAS-0.3.29
make -j$(nproc) USE_OPENMP=0 NO_LAPACK=1
make PREFIX=$HOME/opt/openblas-0.3.29 install

# Set environment for building/running
export OPENBLAS_LIB_DIR=$HOME/opt/openblas-0.3.29/lib
export LD_LIBRARY_PATH=$HOME/opt/openblas-0.3.29/lib:$LD_LIBRARY_PATH
cargo build --release                                  # faer (default)
cargo build --release --no-default-features --features blas   # OpenBLAS

Julia

julia --project=. -e 'using Pkg; Pkg.instantiate()'

Usage

1. Export benchmark metadata

uv run python scripts/generate_dataset.py

This selects instances by category with laptop-scale criteria and saves JSON metadata to data/instances/. rnd_mixed_ instances are excluded (not yet supported by strided-rs).

Optional: Convert TensorNetworkBenchmarks format

To add the TensorNetworkBenchmarks tensor network (550 tensors, 2^33.2 complexity) as a strided-rs instance:

python scripts/convert_tensornetwork.py

This requires the TensorNetworkBenchmarks repository cloned as a sibling directory:

cd .. && git clone https://github.com/TensorBFS/TensorNetworkBenchmarks.git && cd -

Optional: Create a lightweight tensor network instance

The full tensornetwork_permutation_optimized instance (550 tensors) takes ~40 seconds. To create a lighter version (~5 seconds), extract a connected subgraph with an optimized contraction path:

uv run python scripts/create_lightweight_instance.py \
  data/instances/tensornetwork_permutation_optimized.json \
  data/instances/tensornetwork_permutation_light_415.json \
  415

The script finds a BFS-connected subset of the given size with minimal free indices, sets the output to scalar, and computes an optimized contraction path via opt_einsum.

Optional: Focus on the dominant late contraction steps (original steps 408/409)

For kernel-level profiling of the most expensive region in tensornetwork_permutation_light_415, this repo also includes:

  • data/instances/tensornetwork_permutation_focus_step409_316.json

This instance is extracted from the original 415-tensor contraction tree by taking the subtree that ends at original step 409. It preserves the expensive late-stage structure (original steps 408 and 409), while reducing total tree size from 415 to 316 tensors for faster iteration. Unlike the lightweight scalar instance, this focused instance has a non-scalar output (rank-18), because it represents an internal contraction state.

Use it when you want to benchmark or profile the bottleneck contraction kernels directly:

BENCH_INSTANCE=tensornetwork_permutation_focus_step409_316 ./scripts/run_all.sh 1

Selection criteria (per category):

Category Prefix log10[FLOPS] log2[SIZE] num_tensors dtype
Language model lm_ < 10 < 25 ≤ 100 float64 or complex128
Graphical model gm_ < 10 < 27 ≤ 200 float64 or complex128
Structured str_ < 11 < 26 ≤ 200 float64 or complex128

2. Run all benchmarks

./scripts/run_all.sh          # 1 thread (default), canonical ids off
./scripts/run_all.sh 4        # 4 threads, canonical ids off
./scripts/run_all.sh 1 1      # 1 thread, canonical [lo, ro, batch] ids on

Runs Rust (faer + blas) and Julia benchmarks.

  • First argument sets OMP_NUM_THREADS, RAYON_NUM_THREADS, and JULIA_NUM_THREADS.
  • Second argument sets STRIDED_OPTEINSUM_CANONICAL_BINARY_IDS (0 or 1). If OpenBLAS is not installed, the blas benchmark is skipped with a warning. Results are saved to data/results/. To run only one instance, see Run a single instance below.

Instance JSON files that fail to read or parse are skipped with a warning; the suite continues with the rest. Instances that trigger a backend error (e.g. duplicate axis labels in strided-opteinsum) are reported as SKIP in the table with the reason on stderr.

3. Run a single instance

To run the benchmark for one instance only, set the environment variable BENCH_INSTANCE to the instance name. Useful for heavy instances (e.g. gm_queen5_5_3.wcsp, str_nw_mera_closed_120, tensornetwork_permutation_optimized), lighter tensor network tests (e.g. tensornetwork_permutation_light_415), or focused bottleneck testing (e.g. tensornetwork_permutation_focus_step409_316).

With the full script (Rust + Julia):

BENCH_INSTANCE=str_nw_mera_closed_120 ./scripts/run_all.sh 1
BENCH_INSTANCE=gm_queen5_5_3.wcsp ./scripts/run_all.sh 4
BENCH_INSTANCE=tensornetwork_permutation_light_415 ./scripts/run_all.sh 1 1
BENCH_INSTANCE=tensornetwork_permutation_focus_step409_316 ./scripts/run_all.sh 1

Rust or Julia alone:

BENCH_INSTANCE=str_nw_mera_closed_120 cargo run --release
BENCH_INSTANCE=str_nw_mera_closed_120 julia --project=. src/main.jl
  • Instance name must match the name field in the JSON (i.e. the filename without .json). To list available names: ls data/instances/ → e.g. str_nw_mera_closed_120.json → use str_nw_mera_closed_120.
  • Both the Rust and Julia runners respect BENCH_INSTANCE; if set, only that instance is loaded and run for every strategy and mode.

3.1 Binary Diagnostic Datasets (data/instances)

The following binary-only benchmark datasets were added to make backend differences easier to isolate:

  • bin_matmul_256.json (ij,jk->ik)
    • Dense matrix multiplication baseline.
  • bin_batched_matmul_b32_m64_n64_k64.json (bij,bjk->bik)
    • Batched GEMM behavior with moderate batch size.
  • bin_outer_product_4096.json (i,j->ij)
    • Outer-product path (sensitive to broadcast/pack overhead).
  • bin_elementwise_mul_2048x2048.json (ij,ij->ij)
    • Pure elementwise multiplication throughput.

These are intended for quick, reproducible profiling before running heavier LM/structured/network cases.

Example:

RAYON_NUM_THREADS=1 OMP_NUM_THREADS=1 \
  BENCH_INSTANCE=bin_outer_product_4096 cargo run --release

4. Run individually

Rust (strided-opteinsum, faer backend):

RAYON_NUM_THREADS=1 OMP_NUM_THREADS=1 cargo run --release

Rust (strided-opteinsum, blas backend):

RAYON_NUM_THREADS=1 OMP_NUM_THREADS=1 cargo run --release --no-default-features --features blas

Julia (OMEinsum.jl):

OMP_NUM_THREADS=1 JULIA_NUM_THREADS=1 julia --project=. src/main.jl

Julia benchmark modes:

  • omeinsum_path — follows the same pre-computed contraction path as Rust (fair kernel-level comparison)
  • omeinsum_opt — OMEinsum.jl with optimize_code and TreeSA() (optimizer-chosen path)

5. Profiling

CPU flamegraph (requires cargo install flamegraph):

BENCH_INSTANCE=tensornetwork_permutation_light_415 RAYON_NUM_THREADS=1 OMP_NUM_THREADS=1 \
  cargo flamegraph --profile release-with-debug -o flamegraph.svg --no-default-features --features faer

Opens an interactive SVG showing where CPU time is spent. On macOS, uses xctrace; on Linux, uses perf.

Internal profiler (plan/bgemm/buffer stats):

BENCH_INSTANCE=tensornetwork_permutation_light_415 STRIDED_EINSUM2_PROFILE=1 \
  RAYON_NUM_THREADS=1 OMP_NUM_THREADS=1 cargo run --release --no-default-features --features faer

Prints strided-einsum2 dispatch stats to stderr for each strategy:

  • total_calls, direct_calls, packed_calls
  • m_packed_hist, n_packed_hist, k_packed_hist (bucketed packed-path dimensions)

Row-major to Column-major Conversion

NumPy arrays are row-major (C order). strided-rs uses column-major (Fortran order). The conversion is metadata-only:

  • Reverse each tensor's shape: [M, K] -> [K, M]
  • Reverse each operand's index labels: "ij,jk->ik" -> "ji,kj->ki"

Both the original (format_string, shapes) and converted (format_string_colmajor, shapes_colmajor) metadata are stored in each JSON file.

Reproducing Benchmarks

Run all benchmarks (Rust faer + Rust blas + Julia):

./scripts/run_all.sh        # 1 thread, canonical ids off
./scripts/run_all.sh 4      # 4 threads, canonical ids off
./scripts/run_all.sh 1 1    # 1 thread, canonical ids on

This script:

  1. Sets OMP_NUM_THREADS, RAYON_NUM_THREADS, JULIA_NUM_THREADS to the given thread count (default: 1) and STRIDED_OPTEINSUM_CANONICAL_BINARY_IDS (default: 0)
  2. Builds and runs the Rust benchmark with the faer backend
  3. Builds and runs the Rust benchmark with the blas (OpenBLAS) backend (skipped if OpenBLAS is not installed)
  4. Runs the Julia benchmark (julia --project=. src/main.jl)
  5. Formats results as a markdown table via scripts/format_results.py
  6. Saves all outputs to data/results/ with timestamps

To format existing log files into a markdown table:

uv run python scripts/format_results.py data/results/rust_*.log data/results/julia_*.log

See Benchmark index for the current published result pages. The root README intentionally keeps large result tables out of the front page.

References

About

No description, website, or topics provided.

Resources

Readme

License

MIT license
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages