Build whole models from mathspec (whole-model-only) - #958
Open
FabianHofmann wants to merge 67 commits into
Open
FabianHofmann wants to merge 67 commits into
FabianHofmann wants to merge 67 commits into
Conversation
Turns a lowered math-spec Program plus user data into master coordinates, padded lookups and on-demand parameter arrays under the three binding rules. Missing rows stay NaN for the builder. Sources are pulled by key, never iterated, and aligned arrays keep their buffer.
…s under the right dimension
Validate retain, check a scalar's dtype before casting, bind empty sources as all-NaN, check label-space lookup dtypes, report unknown labels in source order on every path, re-stamp coordinates onto the master dtype without copying, and pin the remaining binder rules.
…ype the test spec dicts
… group math-spec is not on PyPI and needs Python >= 3.12. A PEP 735 dependency group keeps the git pin out of the wheel metadata; the 3.12 and 3.13 test jobs install it so the binder tests and their coverage run in CI.
…ssions Port lpspec's linopy lane onto the binder: builder, where, operators, coverage and curves, wired to Bound and SpecDataError. Add Model.add_spec, Model.from_spec and the model.spec accessor with expressions and evaluate.
…en windows Coverage and the retain closure now descend into a Power's operands; evaluate() refuses sources labelled unlike the model; an all-null window width is a window of nothing; cases fold through the aligned combine.
Persist the spec text, the master coordinates and the lookups alongside the model, re-lowering the program from the text on read; math-spec is imported only for a file that carries a spec. Lookups and arrays of labels are stored as codes into a category table, so partial maps keep their holes and dtypes.
Write the in-memory dtype of every parameter and cast it back on read, and stamp the master coordinates onto every container, so no engine leaves a model disagreeing with itself. assert_model_equal now compares dataset dtypes, and synthetic_sources moves to linopy/spec/testing.py for both users.
A missing parameter row was read as a silent zero when it stood as a coefficient, while a bound, constant side or divisor already refused it. Refuse it as a coefficient too, so every position behaves alike and a hole is never filled without the modeller saying so: mask the coordinate out with a where, or fill the value into the data.
A runnable, nbconvert-clean walkthrough of the spec feature: the dispatch program, binding data, folding named expressions, retain and evaluate, the uniform absence rule, lookups and grouped sums, temporal shift, and the netCDF round trip.
m.spec.expressions[name] returns a NamedExpression bundling .node (the lowered formula), .expression (the unsolved linopy expression) and .solution (the fold over the model's solution). evaluate() returns the same object. Add ModelSpec.to_latex/to_markdown/to_typst for whole-model typesetting, rendered as Markdown in a notebook.
building-models-from-specs.ipynb imports math_spec, which the docs CI environment does not install, so the notebook job failed on import. Skip it like the other special-setup notebooks.
# Conflicts: # linopy/io.py
pandas 3 hands strings over as StringDtype, Arrow-backed when pyarrow is installed. xarray keeps the extension array, refuses it in positional indexing and reports no np.dtype, so the netcdf dtype round trip broke.
…rrow in the spec group
…repair moves to io parameters.py owns resolution and derivation, groups.py the axis partition, nodes.amounts_of the parameter-named amounts, Context.lookup the lookups. io records and restores parameter dtypes for every model and owns restamp_coords and the module-level prefix helpers spec/netcdf reuses.
…e walk evaluate.py holds the recursive evaluator, builder.py the declarations. check_coverage collects divisor, constant-side and coefficient obligations in one walk, so cases: masks are evaluated once per declaration. Public docstrings in numpy style so the API pages render.
…shared material in conftest
…in CI with the spec group
…rom_spec and bind warn_evolving_api moves to linopy.constants so piecewise and spec share the once-per-key dedup; the pytest filter silences the spec prefix.
…ift notes Point the math-spec ImportError at the spec dependency group and the 3.12 floor, fix the notebook's stale model.parameters check, document sparse_groupby/freeze_constraints for skewed topologies in api.rst and the notebook, and add release notes for the spec hardening work.
A str is YAML text when it holds a newline, opens a mapping or a sequence, or holds a ':' and names no file; every other str is a path, and a missing one raises FileNotFoundError instead of a read error. An open file is refused by name. A spec that declares no dimension, parameter or variable, or that is not a mapping of sections, is refused before any data is read.
…ing read sources is wrapped once, keys() is called once and everything after that is read through it, so what a build actually read is known. An extra key is ignored by default rather than refused, so one mapping can feed several specs, and Attached.unused reports it; strict=True restores the refusal. A key close to a declared name warns as a likely typo either way, and a sources without keys() is refused by name instead of by AttributeError.
A dual has no symbolic form, so reading .expression says so instead of asking for a solve the model may already have had; folding one the model does not hold raises the spec's own 'no dual yet' rather than linopy's AttributeError. NamedExpression.solution folds afresh on every read.
A dimension index is one flat axis: a MultiIndex, or labels pandas tuple-izes into one, is refused by name. Its dtype is checked against the declaration with the same rule parameter values pass, so a declared type is a claim about the labels too.
…n the model Variables, constraints and expressions a spec builds carry the spec's name in attrs and a spec property. Variable-bearing named expressions are built into model.expressions at add_spec time; build_expressions=False keeps them lazy. remove_variables/constraints/expressions refuse spec-built names; repr tags read the stamp; the name round trips through netcdf.
Merging this PR will improve performance by 13.91%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | test_build[piecewise-n=10] |
15.2 KB | 13.4 KB | +13.91% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing spec-whole-model (fd3c8d5) with master (1b2ea76)
Footnotes
-
181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Build cost — v1 vs legacyv1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).
Full table (time + peak, mean)📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run. Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped. |
FabianHofmann
added
enhancement
FabianHofmann
marked this pull request as ready for review
September 18, 2026 07:45
Drop parameters_of/walk re-exports, present/live_rows, Context.relation and the unreachable raises; collapse coverage checks into one kind table; move wording helpers into errors.py; share dtype helpers with io.py.
…odel.__repr__ ModelSpec._schema becomes a cached_property, and ModelSpec.drift takes the piecewise variables and constraints Model.__repr__ has already computed.
…coverage.py Context, Parameters and the Term/Array/Value aliases live in context.py; amounts_of and dims_of in coverage.py. 17 modules become 14.
This was referenced Sep 19, 2026
Renamed program/expression/predicate types, split Walk into Direction and Partition, and switched program lookups to mapping access.
5 tasks
…ns API Replace the deleted piecewise-checks/derivation layer with a generic check over Program.assumptions; handle the new predicate kinds and drop SosDeclaration.big_m.
…extra
Lower via to_spec().expand('piecewise'), open bounds as None, sos along, Named and at() nodes.
Rows emptied by absence are dropped as mathspec specifies (tracked in #993); non-finite
arithmetic on present data is refused. Old-layout netcdf files load as plain models.
FabianHofmann
changed the title
# Conflicts: # linopy/constraints.py # linopy/io.py
…cated freeze_constraints
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
The following content was generated by AI.
Changes proposed in this Pull Request
A mathspec spec (a YAML declaration of a linear model) builds a whole linopy model from
linopy/spec/. mathspec (>= 0.2, from PyPI) is the optional extralinopy[spec](Python >= 3.12) and imported lazily, soimport linopynever pulls it in.How this differs from #922: this branch drops that PR's layer and binding machinery; a spec builds one whole model into an empty
Modeland cannot be layered onto, or bound into, an existing one.Building
Model.add_spec(spec, sources, retain="report", build_expressions=True)builds variables, SOS, constraints and the objective into an empty v1 model;Model.from_spec(spec, sources, retain=..., **model_kwargs)is sugar over it on a fresh model.specis a path, YAML text, dict ormathspec.Spec.add_specon a non-empty model is refused.linopy.spec.attach(program, sources, retain=...)turns a loweredProgramplus user data into anAttached: master coordinates, lookups and on-demand parameters.sourcesis anyMappingpulled by key and never iterated, or a singlexr.Dataset. mathspec's attachment rules are enforced; unknown labels, duplicate rows, wrong rank or dtype raiseSpecDataError. Aligned data is never copied.Sourcesadapter reads each key once; a key naming nothing the spec declares is ignored and listed onAttached.unused, and a near-miss key warns as a likely typo.assumptions:entry of the spec is checked against the data before building; a failing one raisesSpecDataErrorwith its first coordinates.shift) is not built, as mathspec specifies; a present row left with no variable is refused. Whether this should become strict is tracked in spec: rows emptied by absence are dropped silently; revisit strict refusal #993. Arithmetic on present data that turns non-finite (0/0,1/0,inf - inf) is refused rather than read as absence.Stamping and named expressions
attrs["spec"], read through aspecproperty (str | None) onVariable,Constraint,LinearExpressionandQuadraticExpression. A derivative of a stamped object (expr * 2,linopy.merge,add_constraints(expr >= 0)) carriesNone. The name is the spec file's stem, else"spec".model.expressionsunder its declared name atadd_spectime and stamped, somodel.expressions[name]andmodel.spec.expressions[name].expressionare one object. A data-only body or one reading a constraint'sdualstays spec-only and folds on read.build_expressions=Falsekeeps them lazy; the flag is not persisted.remove_variables,remove_constraintsandremove_expressionsrefuse to drop a name the spec built.unspecifiedreports what the spec does not declare, so a whole-spec model's repr carries no tags; tags appear only once hand-built items sit beside the spec.Reading back and typesetting
model.specis aModelSpecexposingprogram,text,parameters,coords,lookupsandexpressions.model.spec.expressions[name]is aNamedExpressionwith.node,.dims,.expressionand.solution(the fold over the solved model).model.spec.evaluate(name, sources)reattaches parameters afresh forretain="none".model.spec.to_latex/to_markdown/to_typstrender the whole model;declaration(name)renders one line. In a notebook the accessor renders itself for MathJax.Persistence
to_netcdfwrites the spec under aspec-prefix: the spec text and name, master coordinates and lookups, and object-dtype parameters aspandas.factorizecodes and categories, so partial maps keep holes and dtypes on both netcdf engines. A version header names the mathspec version and layout; a version mismatch warns on read, and a file of another layout warns and loads as a plain model withoutmodel.spec.read_netcdfre-lowers theProgramfrom its text, and a file without a spec loads without mathspec installed.Model.copy()carries the spec;assert_model_equalcompares the spec text, name and parameters including dtypes and the per-object stamps.Docs, CI, benchmarks
examples/building-models-from-specs.ipynb, wired into the user guide and executed on Read the Docs and in the notebook CI job with thespecextra installed. API pages forlinopy.specand release notes added.specextra; mypy runs with mathspec installed. Tests skip without it. The sweep over mathspec's example specs runs whenMATHSPEC_EXAMPLESpoints at a mathspec checkout; CI clones the examples at the installed mathspec's tag.Verification (81c8eed, mathspec 0.2.0)
Checklist
AGENTS.md).doc.doc/release_notes.rstof the upcoming release is included.