perf(csr): run each contraction chunk on its used columns only

[FabianHofmann]

Note

The following content was generated by AI.

Changes proposed in this Pull Request

CSRLinearExpression.contracted (the sparse @/dot against a constant, used in both dense and sparse models) multiplies the expression in chunks of 64 kept rows. Each chunk's CSR has one column per variable in the whole model (model._xCounter). scipy's sparse matmul allocates its scratch arrays sized to the column count, so on a model with millions of variables every chunk allocated ~175 MB, although it touched only a few thousand nonzeros. With 100+ chunks this caused millions of page faults and made build times vary 2–3x depending on the malloc heap state.

• New helper column_compacted_matmul in linopy/csr.py: it compacts the chunk to the columns it uses (np.unique on the indices), multiplies, and maps the column indices back. The result is unchanged.
• Release note under Sparse model.

No new test: the existing chunk-boundary contraction tests exercise the index remapping (a deliberately broken map-back fails test_contracted_partial_matches_dense).

Benchmark: build time, best of 3 in a fresh process

Power-system-like model (ramps via shift, storage via roll, groupby(bus).sum(), f @ incidence.T, CO2 sum), size = (snapshots, buses, gens per bus):

size	mode	master	this PR
2190, 200, 5	dense	1.26 s	0.79 s
2190, 200, 5	sparse	1.78 s	1.62 s
8760, 100, 5	dense	7.34 s	1.60 s
8760, 100, 5	sparse	9.31 s	3.30 s

Microbenchmark of the chunked product at the large size: 9.8 s / 7.0M page faults with chunk 64 before, 0.14 s with compacted columns.

Checklist

• AI-generated content is marked (see AGENTS.md).
• Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
• Unit tests for new features were added (if applicable).
• A note for the release notes doc/release_notes.rst of the upcoming release is included.
• I consent to the release of this PR's code under the MIT license.

[codspeed]

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks¹

---

_{Comparing perf/contract-columns (7578d85) with master (26e000d)²}

Footnotes

1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on master (5e53a10) during the generation of this report, so 26e000d was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[github-actions]

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04738 (1.13)   │   15.03 (1.00) 
 (v1)        0.04179 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05055 (1.14)   │   12.04 (1.00) 
 (v1)        0.04449 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.01953 (1.11)   │   15.20 (1.0) 
 (v1)        0.01766 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03281 (1.10)   │   44.93 (1.0) 
 (v1)         0.0298 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0219 (1.06)   │   11.51 (1.0) 
 (v1)       0.02062 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05128 (1.03)   │   24.34 (1.06) 
 (v1)        0.04999 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05551 (1.00)   │   16.12 (1.00) 
 (v1)        0.05545 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.01183 (1.0)   │   752.18 (1.10) 
 (v1)       0.01243 (1.05)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.01168 (1.03)   │   3.12 (1.33) 
 (v1)        0.01136 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03784 (1.19)   │   126.16 (1.44) 
 (v1)        0.03179 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04191 (1.35)   │   126.16 (1.44) 
 (v1)        0.03095 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04126 (1.29)   │   126.16 (1.44) 
 (v1)        0.03195 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.02814 (1.0)   │    715.12 (1.0) 
 (v1)       0.02849 (1.01)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)     0.0264 (1.0)   │   4.54 (1.27) 
 (v1)       0.02652 (1.00)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.2014 (1.06)   │   704.12 (1.09) 
 (v1)        0.1908 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.2185 (1.08)   │   18.34 (1.0) 
 (v1)         0.203 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.2086 (1.05)   │   9.54 (1.0) 
 (v1)        0.1994 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03931 (1.02)   │   3.77 (1.12) 
 (v1)        0.03841 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03735 (1.02)   │   216.59 (1.10) 
 (v1)        0.03654 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.02046 (1.08)   │   938.49 (1.0) 
 (v1)        0.01899 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02176 (1.11)   │   9.66 (1.0) 
 (v1)        0.01963 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02152 (1.09)   │   5.32 (1.0) 
 (v1)        0.01965 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01067 (1.04)   │   1.28 (1.0) 
 (v1)        0.01031 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01118 (1.13)   │   1.28 (1.0) 
 (v1)       0.009917 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01054 (1.03)   │   1.28 (1.0) 
 (v1)        0.01028 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.09416 (1.03)   │   946.85 (1.06) 
 (v1)         0.0911 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09129 (1.02)   │   12.01 (1.00) 
 (v1)        0.08944 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02585 (1.13)   │   147.70 (1.06) 
 (v1)        0.02285 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02659 (1.18)   │   2.60 (1.09) 
 (v1)         0.0225 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02121 (1.14)   │   696.75 (1.03) 
 (v1)        0.01868 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05551 (1.10)   │   137.97 (1.0) 
 (v1)         0.0503 (1.0)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03703 (1.03)   │   69.22 (1.0) 
 (v1)        0.03609 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02341 (1.13)   │   402.33 (1.00) 
 (v1)        0.02075 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02296 (1.10)   │   3.19 (1.19) 
 (v1)        0.02087 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.02523 (1.02)   │   29.00 (1.54) 
 (v1)        0.02472 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03827 (1.28)   │   37.95 (1.43) 
 (v1)           0.03 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.04778 (1.0)   │    410.93 (1.0) 
 (v1)       0.04829 (1.01)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.05146 (1.0)   │     9.94 (1.0) 
 (v1)       0.05202 (1.01)   │   10.22 (1.03) 

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}