From 7814ac57e36bef5c8d4762ad50757b21e9248cbd Mon Sep 17 00:00:00 2001 From: Fabian Date: Fri, 9 Oct 2026 15:48:10 +0200 Subject: [PATCH] docs: prepare release notes for v0.10.0; v1 becomes the only convention in 1.0 --- doc/design/goals.rst | 7 +- doc/migrating-to-v1.rst | 25 +++--- doc/release_notes.rst | 179 ++++++++++++++++++++++++---------------- 3 files changed, 123 insertions(+), 88 deletions(-) diff --git a/doc/design/goals.rst b/doc/design/goals.rst index f321b5560..2135037c1 100644 --- a/doc/design/goals.rst +++ b/doc/design/goals.rst @@ -41,7 +41,6 @@ Transitioning goals **Schedule:** -1. Introduce v1 as opt-in — warn about behaviour changes on legacy, raise if - opted into v1. -2. Make v1 the default, allow opt-out. -3. linopy 1.0 — drop the legacy convention entirely. +1. linopy 0.10 — introduce v1 as opt-in. Warn about behaviour changes on + legacy, raise if opted into v1. +2. linopy 1.0 — make v1 the only convention and drop legacy entirely. diff --git a/doc/migrating-to-v1.rst b/doc/migrating-to-v1.rst index 44595f2a6..ea3d9c2ca 100644 --- a/doc/migrating-to-v1.rst +++ b/doc/migrating-to-v1.rst @@ -3,9 +3,10 @@ Migrating to the v1 arithmetic convention .. note:: - v1 is **opt-in** in this release and legacy remains the default. Nothing - changes until you set ``linopy.options["semantics"] = "v1"``. This guide is - for deciding *when* to opt in and *what* to change when you do. + v1 is **opt-in** in linopy 0.10 and legacy remains the default. linopy 1.0 + removes legacy, so 0.10 is the release to migrate in. Nothing changes until + you set ``linopy.options["semantics"] = "v1"``. This guide tells you *what* + to change when you do. Why v1 exists ------------- @@ -28,16 +29,18 @@ per-case bug catalogue is in `issue #714 `. +**Strict "v1" arithmetic (opt-in)** +Enable it with ``linopy.options["semantics"] = "v1"``. Legacy stays the default in 0.10 but is removed in 1.0, so we recommend switching now. Under legacy, every operation whose result changes under v1 emits a ``LinopySemanticsWarning`` that names the fix, so a model can migrate step by step. The full rules are in :doc:`the arithmetic convention `. Under v1: -*Numerical scaling* +* **Exact alignment.** Shared dimensions align by label with ``join="exact"``. Differing labels or a pure reorder raise. Resolve with ``.sel`` / ``.reindex`` / ``.assign_coords``, or pass ``join=`` to ``.add`` / ``.sub`` / ``.mul`` / ``.div`` / ``.le`` / ``.ge`` / ``.eq``. +* **Unlabeled operands.** Numpy arrays, lists and polars ``Series`` pair with dimensions by size. An ambiguous match (square array, equal-length dimensions) or no match raises. Wrap the operand in a ``DataArray`` to name its dimensions. (https://github.com/PyPSA/linopy/issues/736) +* **NaN constants raise** instead of being filled with a value that differed per operator. +* **Absence propagates.** Masked, reindexed and shifted-in slots stay absent through every operator instead of collapsing to zero. +* **Explicit fill on joins.** A reindexing ``join=`` fills the positions it creates per side: the linopy operand with the zero expression, the constant with the new ``fill_value=`` of ``.add`` / ``.sub`` / ``.mul`` / ``.div``. The default means "does not apply": a missing divisor zeroes the term, and a missing numerator no longer gives ``1 / divisor``. ``fill_value=linopy.ABSENT`` keeps all created positions absent, also in ``linopy.merge``. (https://github.com/PyPSA/linopy/issues/890) +* **Conflicting auxiliary coordinates raise** instead of being dropped silently. +* **No MultiIndex dimensions.** Use a flat dimension with the levels as auxiliary coordinates. +* **Label-aligned groupby.** A grouper must match the grouped dimension's labels in set and order. A multi-key grouper gives a flat ``group`` dimension with the keys as auxiliary coordinates. (https://github.com/PyPSA/linopy/issues/827) -* Variables, constraints and the objective accept a ``scaling`` factor that rewrites the problem into better-behaved units for the solver, without changing the answer. Variable scaling is column-like, constraint and objective scaling are row-like, and primal values, duals and the objective are transformed back to the original units after solving. See the :doc:`numerical-scaling` tutorial and the *Numerical scaling* section of the :doc:`user-guide`. +**Sparse models** -*New solver: NVIDIA cuOpt* +* **One switch.** ``Model(sparse=True)`` turns on the sparse (CSR, compressed sparse row) path for a model. ``groupby(...).sum()`` and ``@``/``dot`` against a constant return CSR-backed expressions, and ``add_constraints`` freezes every constraint unless ``freeze=False`` is passed. The key requires v1 and rejects ``chunk``. ``Model.copy`` and the netcdf round trip keep it. (`#976 `__) +* **Sparse groupby-sum.** It builds from the ungrouped terms, so no padded term rectangle exists and build memory no longer depends on the largest group: 6-9x less peak memory on the hub-bus case. Multi-key groupers work; ``observed=True`` keeps only observed key combinations. (`#745 `__, `#870 `__, `#940 `__) +* **Stays sparse.** A CSR-backed expression is no longer densified by: -* Added support for the GPU-accelerated `NVIDIA cuOpt `__ solver for linear, mixed-integer and convex quadratic problems, via ``model.solve("cuopt", io_api="direct")``. Install it with ``pip install "linopy[gpu]"`` — Linux only, and requires an NVIDIA GPU of compute capability 7.0 or higher with a CUDA 12 driver (525.60.13 or newer). See :doc:`gpu-acceleration` for the supported problem classes and the known limitations. + * metadata reads: ``repr()``, ``shape``, ``sizes``, ``dims``, ``coords``, ``indexes``, ``isnull()``, ``const``, ``has_terms`` and ``drop_vars`` of auxiliary coordinates (`#962 `__, `#1003 `__) + * arithmetic with a non-scalar constant, aligned as on the dense path (`#965 `__) + * ``sum(dim=...)`` and a further ``groupby(...).sum()`` (`#964 `__) + * ``where``, ``sel``, ``isel``, ``loc`` and ``[]``, with the dense semantics; masked cells become absent (`#966 `__) + * ``linopy.merge`` of differing grids with auxiliary coordinates (`#966 `__) + * ``@``/``dot`` against a constant, which returns the compact canonical form (`#748 `__, `#756 `__, `#925 `__) + * ``add_constraints(..., mask=..., freeze=True)`` and an expression on the right-hand side (`#970 `__) + * ``flat``, ``to_polars`` and ``solution`` (`#968 `__) + * use as linear objective, through ``matrices.c``, LP/MPS, netcdf, ``Model.copy``, persistent snapshots and direct solver APIs (`#967 `__) -*New feature: constraint softening* + An operand or indexer that adds dimensions still densifies. +* **Visibility.** ``LinearExpression.is_sparse`` tells if an expression is CSR-backed, and its repr reads ``LinearExpression (sparse)``. The opt-in ``linopy.options["warn_on_densify"]`` emits a ``PerformanceWarning`` with the reason whenever a sparse backing is dropped. (`#969 `__) -* A constraint can now be softened with ``Constraint.soften(penalty, max_violation=None, name=None)``, which adds a slack variable (a positive/negative pair for equality constraints) to the constraint's ``lhs`` and a penalty term to the objective, returning a ``Slack`` named tuple. It is not supported on detached copies from ``.mutable()``, ``.sel()``, or ``.isel()``. ``model.add_constraints(..., penalty=...)`` is a shortcut that softens the constraint right after creation. -* A frozen ``CSRConstraint`` is softened natively, without densifying: the slack terms are appended to its sparse rows in place, so ``add_constraints(..., penalty=..., freeze=True)`` works as well. The slack is kept by ``to_dense()``/``mutable()``, ``Constraint.freeze()`` and the netcdf round trip. (`#975 `__) -* The slack variable(s) created by ``soften()`` can be retrieved afterwards via the new ``slack`` property of ``Constraint`` and ``CSRConstraint``. +**Numerical scaling** -*Sparse model* +* Variables, constraints and the objective accept a ``scaling`` factor that moves the problem into better-behaved units for the solver. Variable scaling acts on columns, constraint and objective scaling on rows. +* Primal values, duals and the objective are mapped back to the original units after the solve. Solver tolerances apply to the scaled problem. +* See the :doc:`numerical-scaling` tutorial and the *Numerical scaling* section of the :doc:`user-guide`. -* New read-only model key ``Model(sparse=True)`` turns on the whole sparse (CSR) path for one model: ``groupby(...).sum()`` and ``@``/``dot`` against a constant return CSR-backed expressions, and ``add_constraints`` freezes every constraint unless ``freeze=False`` is passed. It requires the v1 semantics: ``Model(sparse=True)`` raises under legacy, and so do ``groupby(...).sum()`` and ``@`` on a sparse model, and ``read_netcdf`` of one, once the semantics are switched back to legacy. It rejects ``chunk``. The key is kept by ``Model.copy`` and the netcdf round trip. ``@`` no longer reads ``linopy.options["sparse_groupby"]``. (`#976 `__) -* ``@``/``dot`` against a sparse constant no longer scales with the total number of variables in the model: each chunk of the sparse product now runs on only the variables it uses, which removes several seconds of allocation overhead on models with millions of variables. (`#990 `__) -* Deprecated in favour of ``Model(sparse=True)``, each with a ``FutureWarning`` and to be removed with the legacy semantics: ``Model(freeze_constraints=...)`` and the ``Model.freeze_constraints`` setter, ``groupby(...).sum(sparse=...)`` and ``linopy.options["sparse_groupby"]``. They keep their current behaviour until then, except that ``@`` ignores ``sparse_groupby``, and netcdf files that store ``freeze_constraints`` still load. (`#976 `__) -* Adding a frozen constraint from a sparse expression no longer copies the lhs matrix when every row stays active, and picks the mask and the row scaling at the active rows without expanding them over the full coordinate grid. This roughly halves the peak memory of ``add_constraints`` on a sparse model. (`#977 `__) -* Building ``Model.matrices`` no longer allocates label-sized scaling lookups and skips ``eliminate_zeros`` for frozen constraints. On a model with 2M variables this makes each build of a sparse or frozen model about 20 to 35 ms faster. (`#1008 `__) -* Freezing a dense constraint builds the CSR matrix directly from the term rectangle and sorts it only when a row has more than one term. On a model with 2M variables this cuts the freeze of the nodal balance from about 465 to 185 ms and its peak memory from 1.74 GB to 0.25 GB. The export of a mutable constraint also no longer copies a strided term array whose terms are mostly zero. (`#1009 `__) +**Constraint softening** -*Other* +* ``Constraint.soften(penalty, max_violation=None, name=None)`` adds a slack variable to the ``lhs`` (a positive/negative pair for equality constraints) and a penalty term to the objective. It returns a ``Slack`` named tuple; the ``slack`` property gives it back later. +* ``add_constraints(..., penalty=...)`` softens the constraint on creation, also with ``freeze=True``. +* A frozen ``CSRConstraint`` is softened natively by appending the slack terms to its sparse rows. ``to_dense()``/``mutable()``, ``Constraint.freeze()`` and netcdf keep the slack. (`#975 `__) +* Detached copies from ``.mutable()``, ``.sel()`` or ``.isel()`` cannot be softened. + +**New solver: NVIDIA cuOpt** -* New method :meth:`linopy.Model.assign_coords` reassigns coordinate values across an existing model — variables, constraints (dense and CSR-backed), expressions and parameters — without changing the model's shape: ``m.assign_coords(snapshot=new_snapshots)``. Values-only: the new values must match the length of the dimension's full-index container, and containers holding subsets of the dimension are mapped by label, preserving the subset relation. Dataset variable order is preserved. Typical use is advancing the window in rolling-horizon optimization with the persistent solver interface. (https://github.com/PyPSA/linopy/issues/767) +* GPU-accelerated `NVIDIA cuOpt `__ for linear, mixed-integer and convex quadratic problems, via ``model.solve("cuopt", io_api="direct")``. +* Install with ``pip install "linopy[gpu]"``. Requires Linux, an NVIDIA GPU of compute capability 7.0+ and a CUDA 12 driver (525.60.13+). See :doc:`gpu-acceleration` for supported problem classes and limitations. -* ``add_piecewise_formulation`` gained a ``mask`` parameter declaring which breakpoint slots hold a real breakpoint. It is needed for **ragged** curves — entities with different numbers of breakpoints — which are stored densely with the surplus slots left absent. Under v1 that absence must be declared (``mask=x_pts.notnull()``) rather than read off the NaN padding. (https://github.com/PyPSA/linopy/issues/884) -* ``Model.compute_infeasibilities`` now also supports HiGHS, via ``Highs.getIis``. (https://github.com/PyPSA/linopy/issues/956) +**Other features** -*Internal* +* :meth:`linopy.Model.assign_coords` replaces coordinate values across a model (variables, constraints, expressions, parameters) without changing its shape, e.g. ``m.assign_coords(snapshot=new_snapshots)``. Containers on a subset of the dimension are mapped by label. Typical use: advancing the window in rolling-horizon runs with the persistent solver interface. (https://github.com/PyPSA/linopy/issues/767) +* ``add_piecewise_formulation(..., mask=...)`` declares which breakpoint slots are real, for **ragged** curves with different breakpoint counts per entity. Under v1 the mask is required (``mask=x_pts.notnull()``) instead of reading NaN padding. (https://github.com/PyPSA/linopy/issues/884) +* ``Model.compute_infeasibilities`` supports HiGHS, via ``Highs.getIis``. (https://github.com/PyPSA/linopy/issues/956) +* ``Model.to_netcdf(compression=...)`` applies one encoding to every array, so compression no longer needs linopy's internal variable names. ``True`` (default) means ``{"zlib": True, "complevel": 3}``, ``False`` turns it off, and explicit ``encoding`` entries win. (`#1007 `__) -* The sparse backing of a ``LinearExpression`` moved from ``linopy.sparse_expression`` to ``linopy.csr`` and the class ``CSRExpression`` was renamed to ``CSRLinearExpression``. Dense/sparse conversion is now spelled the same way on both CSR types: ``CSRLinearExpression.from_dense`` / ``.to_dense`` and ``CSRConstraint.from_dense`` (previously ``CSRConstraint.from_mutable``) / ``.to_dense``. ``Constraint.freeze()`` and ``CSRConstraint.mutable()`` are unchanged. -* The first constructor argument of ``LinearExpressionGroupby`` is now named ``obj`` (previously ``data``), as it holds a ``Dataset`` or an expression; ``LinearExpressionGroupby(data=...)`` must be spelled ``LinearExpressionGroupby(obj=...)``. +**Performance** -*Documentation* +*Model build* -* The example notebooks now opt into the v1 arithmetic convention (``linopy.options["semantics"] = "v1"``). The coordinate-alignment and expression tutorials were reworked to teach strict label-based alignment: a mismatch on a shared dimension raises rather than silently filling or pairing by position, and is resolved explicitly with ``.sel`` / ``.reindex`` / ``.assign_coords`` or an explicit ``join=`` on the named ``.add`` / ``.mul`` / ``.le`` / … methods. +* ``@``/``dot`` against a constant with zeros drops the zero terms instead of keeping one term per contracted member: 852 to 3 terms on PyPSA's KVL (Kirchhoff Voltage Law) constraint. (`#748 `__) +* Sparse ``@``/``dot`` runs each chunk on only the variables it uses, which removes seconds of overhead on models with millions of variables. Chunks are sized by nonzeros. (`#990 `__) +* ``densify_terms`` (``sum(drop_zeros=True)``) is vectorised: 127 s to 3 ms on a (2000 x 60) expression. +* An N-way ``linopy.merge`` of CSR-backed expressions concatenates once and scales with the nonzeros, not quadratically with the operand count. +* ``add_constraints`` on a sparse model no longer copies the lhs matrix or expands the mask and scaling over the full grid: about half the peak memory. (`#977 `__) +* Freezing a dense constraint builds the CSR matrix directly and sorts only rows with more than one term: 465 to 185 ms and 1.74 to 0.25 GB on a 2M-variable nodal balance. (`#1009 `__) -**Performance** +*Matrices, export and solvers* -* Direct solver APIs no longer set variable and constraint names by default: ``Model.set_names_in_solver_io`` now defaults to ``False``, and ``to_highspy``, ``to_gurobipy``, ``to_mosek`` and ``to_xpress`` follow it when ``set_names`` is not given. The default names ``x{label}``/``c{label}`` carry no information, and the solution is mapped back by position. ``Model.solve`` still sets them when ``warmstart_fn`` or ``basis_fn`` is given, as basis files refer to names. Setting them made ``to_highspy`` about 2x slower and cost ~100 ms in ``to_gurobipy`` on a model with 200k variables. **Behaviour change:** pass ``set_names=True`` or ``Model(set_names_in_solver_io=True)`` to keep names on the native solver model, e.g. to write it to a file. (`#978 `__) -* ``@``/``dot`` against a constant matrix that holds zeros no longer densifies the result to one term per contracted member. The zero-coefficient terms are dropped, so the term dimension shrinks to the widest non-zero cell. On PyPSA's Kirchhoff Voltage Law constraint (a cycle matrix with ~3 branches per cycle) this cuts the expression from 852 to 3 terms — 284x fewer cells — which in turn shrinks the downstream ``merge``. A constant without zeros is unaffected. (`#748 `__) -* ``densify_terms`` (used by ``sum(drop_zeros=True)`` and the sparse ``@`` path) is now fully vectorised. It previously counted the non-zero positions with a Python loop that scaled quadratically in the number of non-zero terms — 127 s for a (2000 x 60) expression, now 3 ms — and allocated the compacted output at the full original term width. It now allocates only the compacted width and returns the expression unchanged when it holds no zeros. -* Under v1, ``@``/``dot`` against a constant now runs as sparse linear algebra instead of building the dense broadcast intermediate (``self * other`` then ``.sum()``): peak memory scales with ``nnz(C) x nterm`` rather than with the full broadcast shape. A ``CSRLinearExpression``-backed operand (from ``groupby(...).sum(sparse=True)``) stays CSR-backed through ``@``, and the result is the compact canonical form (duplicate variables summed, terms label-ordered, explicit zeros pruned — cell activeness is carried by ``const`` alone). (`#748 `__, `#756 `__, `#925 `__) -* Reading metadata of a CSR-backed ``LinearExpression`` (``repr()``, ``shape``, ``sizes``, ``dims``, ``coords``, ``indexes``, ``isnull()``) no longer converts it to dense; it is served from the sparse backing, auxiliary coordinates included. (`#962 `__) -* New ``LinearExpression.is_sparse`` tells whether an expression is CSR-backed, and its repr header reads ``LinearExpression (sparse)``. The opt-in option ``linopy.options["warn_on_densify"]`` (default ``False``) emits a ``PerformanceWarning`` naming the reason whenever a sparse backing is dropped, e.g. on ``.data``, an operation without a sparse path, or ``CSRConstraint.mutable()``. (`#969 `__) -* Multiplying, dividing, adding or subtracting a non-scalar constant (``DataArray``, ``Series``, ``ndarray``, …) keeps a CSR-backed ``LinearExpression`` sparse: the coefficients are scaled per cell and only the constant changes, instead of expanding the dense rectangle. The constant is aligned exactly as on the dense path (NaN, label mismatch, joins and auxiliary coordinates); an operand that adds dimensions still densifies. ``expr.const`` is now served from the sparse backing as well. (`#965 `__) -* ``sum(dim=...)`` and a further ``groupby(...).sum()`` keep a CSR-backed ``LinearExpression`` sparse: summed rows are merged in the sparse backing instead of stacking the summed dimensions into the term dimension of the dense rectangle. Auxiliary coordinates, absent cells and the constant follow the dense path; ``groupby(...).sum(sparse=False)`` still densifies. (`#964 `__) -* ``where``, ``sel``, ``isel``, ``loc`` and ``[]`` keep a CSR-backed ``LinearExpression`` sparse: the selection is evaluated on the grid's row numbers with the dense (xarray) semantics, including ``drop=``, ``method=``, scalar coordinates left by a scalar selection and auxiliary coordinates, and the selected rows are gathered from the sparse backing; masked cells become absent. A condition or indexer that introduces dimensions still densifies. A merge of differing grids now stays sparse when operands carry auxiliary coordinates, which are checked for conflicts as on the dense path. (`#966 `__) -* ``Model.add_constraints(..., mask=..., freeze=True)`` with a CSR-backed left-hand side, or with a ``CSRConstraint`` built beforehand, applies the mask sparsely and returns a ``CSRConstraint`` instead of expanding the dense rectangle. An expression on the right-hand side (``lhs <= other_expr`` or ``add_constraints(lhs, "<=", other_expr)``) no longer densifies a CSR-backed left-hand side: it is moved to the left-hand side, its constant becoming the right-hand side, as on the dense path. ``penalty=`` stays unsupported together with ``freeze=True`` and raises. (`#970 `__) -* ``LinearExpression.flat`` and ``LinearExpression.to_polars`` on a CSR-backed expression are emitted directly from the sparse backing instead of expanding the dense rectangle; the rows equal those of the dense path, with absent cells and zero coefficients dropped. (`#968 `__) -* A CSR-backed linear objective stays sparse when set with ``Model.add_objective`` and when the model is exported or solved: the objective vector ``matrices.c``, LP/MPS files, netcdf output, ``Model.copy``, persistent snapshots and direct solver APIs all read it without expanding the dense rectangle, with results equal to the dense objective. The objective name is now owned by the ``Objective`` itself (``Objective.name`` and ``Objective.attrs``) instead of being written into the expression's attributes. (`#967 `__) -* Persistent snapshots of tz-aware ``DatetimeIndex`` coordinates no longer materialise an object array of ``Timestamp`` per container per capture and diff. Coordinates are stored as UTC-ns arrays with the timezone identity carried alongside, making snapshot capture ~24x and warm-start diffs ~33x faster on tz-aware models, while naive and tz-aware coordinates — and differing timezones — stay correctly unequal. (`#960 `__) -* The MOSEK direct API builds bound keys, bounds and constraint names vectorised instead of looping over every row and column in Python. -* The LP writer computes the scaling lookups once per file, skips scaling when every factor is 1, does not re-sort rows that are already grouped (as frozen constraints deliver them) and builds each constraint line from fewer string branches. Constraint writing is about 2x faster; the output is byte-identical. -* ``linopy.merge`` of CSR-backed expressions concatenates all operands once instead of folding them pairwise, so an N-way merge scales with the total number of nonzeros instead of quadratically with the operand count. -* ``LinearExpression.solution`` on a CSR-backed expression is evaluated on its sparse backing without densifying. -* ``model.matrices`` assembles the constraint blocks into preallocated CSR buffers in one pass, without ``vstack``, ``eliminate_zeros`` or a full-grid scaling round trip; scaling is skipped when it is identity. Label lookups use the model's label dtype, so ``matrices.clabels`` and ``matrices.indicator_binvar`` are now ``int32`` by default. On an 8M-row model this is 45% faster with 200 MB less peak memory, with bit-identical output. -* Duals of frozen constraints are read back on the active rows directly instead of through three full-grid arrays, and freezing a masked constraint gathers rows once instead of up to three times. Mixed-sign constraints convert their sense vectorised instead of in a Python loop. +* ``Model.matrices`` assembles constraint blocks in one pass into preallocated CSR buffers, without full-grid scaling lookups or ``eliminate_zeros``: 45% faster with 200 MB less peak memory on an 8M-row model, with identical output. (`#1008 `__) +* Duals of frozen constraints are read back on the active rows only. A masked freeze gathers rows once, and mixed-sign senses convert vectorised. +* The LP writer writes constraints about 2x faster with byte-identical output. +* The MOSEK direct API builds bounds and names vectorised. +* Direct solver APIs no longer set variable and constraint names by default, which made ``to_highspy`` about 2x faster (see **Breaking changes**). (`#978 `__) +* Persistent snapshots of tz-aware ``DatetimeIndex`` coordinates store UTC-ns arrays plus the timezone: capture ~24x and warm-start diffs ~33x faster. (`#960 `__) **Bug fixes** -* ``Variable.get_solver_attribute`` for Gurobi now maps values by position for the direct API. It parsed labels from the solver's variable names, which returned the values of the wrong variables when names were off and a variable was masked. (`#978 `__) -* ``densify_terms`` no longer raises on expressions without coordinate dimensions (``expr.sum(drop_zeros=True)`` over all dimensions) and now works on ``QuadraticExpression``, where it previously indexed the ``_factor`` axis as the term axis. -* ``sum()`` over a dimension no longer raises when another dimension of the expression has size 0; it returns an expression without terms over the kept coordinates, as summing over the empty dimension itself already did. (https://github.com/PyPSA/linopy/issues/906) -* A multi-key ``groupby`` now returns its groups sorted by key tuple, like the single-key path. The key combinations were numbered by iterating a ``set``, so the group order was arbitrary and changed between processes with ``PYTHONHASHSEED``. -* The ``linopy.options`` context manager now restores the values that were active on entry instead of resetting all options to their defaults. -* ``Model.copy`` (and the ``copy.copy``/``copy.deepcopy`` protocols) no longer downgrades a quadratic objective to a linear one. The objective was rebuilt as a ``LinearExpression`` regardless of its type, so the copy silently solved a different problem. ``linopy.testing.assert_model_equal`` now compares the objective by expression type as well, which it could not do before. (`#903 `__) -* ``Model.to_netcdf``/``linopy.read_netcdf`` downgraded a quadratic objective the same way; the expression type is now stored alongside the objective and restored on read. Files written by earlier versions are read as before. (`#903 `__) -* ``linopy.read_netcdf`` now restores variables, expressions and constraints in their original insertion order instead of alphabetically, so ``matrices.A`` of a round-tripped model is no longer a row/column permutation of the original. The order is stored in the file; files written by earlier versions still load in sorted order. ``linopy.testing.assert_model_equal`` now also compares container order. (`#934 `__) -* Adding or removing variables after a constraint was added with ``freeze=True`` no longer breaks ``model.matrices``. The frozen constraint stored dense variable positions as its matrix columns, so blocks frozen at different times disagreed on their width and stacking them raised ``ValueError: inconsistent shapes``. Raw variable labels are stored instead and mapped to positions when the matrix is assembled. Frozen constraints in netCDF files written by earlier versions are read as before. (`#926 `__) -* A ``CSRConstraint`` built from a CSR-backed expression now keeps its auxiliary coordinates, e.g. the key levels of a multi-key ``groupby(..., observed=True).sum(sparse=True)`` or a scalar coordinate left by a scalar selection, so its ``coords`` equal those of the dense constraint. They are kept by ``to_dense()``/``mutable()`` and written to and read from netcdf, and ``Constraint.freeze()`` keeps them as well. (`#941 `__) -* ``linopy.read_netcdf`` no longer fails for a whole model when a frozen constraint has a dimension or MultiIndex level whose name contains ``-``: the coordinate metadata of a ``CSRConstraint`` is now stored under positional variable names with the names as attributes. Files written by earlier versions are read as before. -* ``CSRConstraint.loc``, ``update`` and ``from_rule``, and assigning to ``coeffs``, ``vars`` or ``sign``, now raise an ``AttributeError`` saying the operation is not supported on a frozen constraint and naming ``.mutable()`` as the way out (for ``from_rule``: build with ``Constraint.from_rule`` and ``.freeze()`` the result), like the read-only ``rhs``, ``lhs`` and ``scaling`` setters, instead of a bare ``AttributeError``. (`#963 `__) -* ``linopy.merge`` with ``join="outer"``, ``"left"`` or ``"right"`` no longer raises an xarray ``MergeError`` when only one operand carries an auxiliary coordinate on a joined dimension. The constant was reindexed with a fill value of ``0`` that was also written into the auxiliary coordinate, so it no longer matched the coefficients' copy. The coordinate now follows its rows onto the joined grid and is NaN where no operand provides it, under both semantics. The same holds for ``.add`` / ``.sub`` / ``.mul`` / ``.div`` with a constant and a reindexing ``join=``. -* A frozen constraint caches the label-to-position mapping of its matrix columns by weak reference and only rebuilds it when the constraint or the set of variables changes. While a persistent snapshot holds the arrays, repeated matrix assembly on an unchanged model returns the same objects, so the snapshot diff can again skip the comparison of untouched frozen constraints by object identity; one-off exports such as ``to_file`` retain no extra memory. (`#933 `__) -* ``Model.copy`` (and the ``copy.copy``/``copy.deepcopy`` protocols) keeps frozen constraints as ``CSRConstraint`` instead of rebuilding them as dense ``Constraint``; ``deep=True`` copies their arrays, and duals are copied only with ``include_solution=True``. A CSR-backed objective is now deep-copied with ``deep=True`` as well. The auxiliary coordinates of a CSR grid are stored read-only, so a copy can share them safely. (`#981 `__) -* ``Solver.close()`` no longer leaves dangling native handles behind. The solver model is now dropped before the environment that owns it, instead of after. And the COPT and MindOpt file interfaces no longer hand back a model they already disposed: after a file-based COPT or MindOpt solve, ``model.solver_model`` is ``None`` rather than a handle into freed memory. (`#899 `__) +*Copy and I/O* -**Breaking Changes** +* ``Model.copy`` (and ``copy.copy``/``copy.deepcopy``) no longer turns a quadratic objective into a linear one. ``assert_model_equal`` now compares the objective type. (`#903 `__) +* ``to_netcdf``/``read_netcdf`` keep a quadratic objective quadratic. (`#903 `__) +* ``read_netcdf`` restores containers in insertion order, so a round trip no longer permutes ``matrices.A``. ``assert_model_equal`` now compares the order. (`#934 `__) +* ``read_netcdf`` no longer fails when a frozen constraint has a dimension or level name with ``-``. +* ``Model.copy`` keeps frozen constraints frozen; ``deep=True`` copies their arrays and a CSR-backed objective. (`#981 `__) + +Files from earlier versions load as before. + +*Frozen constraints* + +* Adding or removing variables after ``freeze=True`` no longer breaks ``model.matrices`` with ``inconsistent shapes``. (`#926 `__) +* A ``CSRConstraint`` from a CSR-backed expression keeps its auxiliary coordinates through ``to_dense()``, ``freeze()`` and netcdf. (`#941 `__) +* Unsupported ``CSRConstraint`` operations (``loc``, ``update``, ``from_rule``, setting ``coeffs``/``vars``/``sign``) raise an ``AttributeError`` that points to ``.mutable()``. (`#963 `__) +* Repeated matrix builds on an unchanged model reuse the cached column mapping again, so persistent snapshot diffs skip untouched frozen constraints. (`#933 `__) + +*Expressions* + +* ``sum()`` no longer raises when another dimension has size 0. (https://github.com/PyPSA/linopy/issues/906) +* A multi-key ``groupby`` returns groups sorted by key, not in an order that changed with ``PYTHONHASHSEED``. +* ``linopy.merge`` with ``join="outer"``/``"left"``/``"right"`` no longer raises a ``MergeError`` when only one operand has an auxiliary coordinate on a joined dimension. The same holds for ``.add`` / ``.sub`` / ``.mul`` / ``.div`` with a reindexing ``join=``. +* ``densify_terms`` works on expressions without coordinate dimensions and on ``QuadraticExpression``. + +*Solvers and options* + +* ``Variable.get_solver_attribute`` for Gurobi maps values by position. It returned values of the wrong variables when names were off and a variable was masked. (`#978 `__) +* ``Solver.close()`` drops the solver model before its environment. After a file-based COPT or MindOpt solve, ``model.solver_model`` is ``None`` instead of a handle into freed memory. (`#899 `__) +* The ``linopy.options`` context manager restores the values active on entry, not the defaults. + +**Breaking changes** + +* **No automatic solver release.** Garbage collection no longer closes the solver, as the finalizer could abort the interpreter. Call ``model.solver.close()``, set ``model.solver = None`` or use ``contextlib.closing(model.solver)``. A new ``solve()`` still closes the previous solver. (`#899 `__) +* **No solver names by default.** ``Model.set_names_in_solver_io`` defaults to ``False`` for ``to_highspy``, ``to_gurobipy``, ``to_mosek`` and ``to_xpress``. ``solve()`` still sets names with ``warmstart_fn`` or ``basis_fn``. Pass ``set_names=True`` or ``Model(set_names_in_solver_io=True)`` to keep them. (`#978 `__) +* **Compressed netcdf by default.** ``Model.to_netcdf`` compresses with zlib level 3; pass ``compression=False`` for the old output. (`#1007 `__) +* **Renamed sparse internals.** ``linopy.sparse_expression`` is now ``linopy.csr``, ``CSRExpression`` is ``CSRLinearExpression`` and ``CSRConstraint.from_mutable`` is ``CSRConstraint.from_dense``. +* **Other API changes.** ``LinearExpressionGroupby(data=...)`` is now ``LinearExpressionGroupby(obj=...)``. The objective name lives on ``Objective.name``/``Objective.attrs``, not in the expression's attributes (`#967 `__). ``matrices.clabels`` and ``matrices.indicator_binvar`` are ``int32`` by default. + +**Deprecations** + +* In favour of ``Model(sparse=True)``, with a ``FutureWarning``: ``Model(freeze_constraints=...)``, the ``Model.freeze_constraints`` setter, ``groupby(...).sum(sparse=...)`` and ``linopy.options["sparse_groupby"]``. They keep working until linopy 1.0 removes them together with the legacy semantics; only ``@`` ignores ``sparse_groupby`` already. (`#976 `__) + +**Documentation** -* A solver is no longer released automatically when the model is garbage-collected, withdrawing a guarantee announced in 0.8.0. The finalizer that did so ran mid-collection and could tear down native state inside an unrelated call stack, aborting the interpreter. Release the solver explicitly instead, with ``model.solver.close()``, ``model.solver = None``, or ``contextlib.closing(model.solver)``. A new ``solve()`` still closes the solver it replaces, so a re-solve loop needs no change. (`#899 `__) +* The example notebooks use v1 semantics. The coordinate-alignment and expression tutorials teach strict label-based alignment and explicit ``join=``. Version 0.9.1 -------------