Add EIC-first deterministic matching before Duke fuzzy matching#289
Add EIC-first deterministic matching before Duke fuzzy matching#289MaykThewessen wants to merge 2 commits into
Conversation
|
Two developments that raise the value of this PR:
Happy to rebase/extend if there's interest in moving this forward. |
Implements the architecture proposed in PyPSA#287: deterministic matching on EIC (Energy Identification Code) runs before Duke fuzzy matching in compare_two_datasets(). Plants sharing an EIC code are matched with certainty, then removed from the Duke input so the fuzzy matcher only handles the residual. This provides a robustness guarantee against co-located plant confusion. For example, in the Eemshaven harbour area (Netherlands): OPSD "Eemscentrale Ec" (Natural Gas, 1929 MW, 6 EIC codes) ENTSOE "Eems" (Natural Gas, 1931 MW, same 6 EIC codes) ENTSOE "Eemshaven" (Hard Coal, 1580 MW, different EIC) ENTSOE "Eemshaven" (Natural Gas, 1410 MW, different EIC) EIC matching deterministically pairs the gas plants via their shared codes, preventing any possibility of Duke merging them with the nearby coal plant based on name/geo similarity. Similarly, Borssele nuclear (OPSD: 492 MW, ENTSOE: 485 MW) is locked to its correct cross-source pair via EIC 49W000000000054X, independent of fuzzy name matching against nearby Borssele wind/coal entries. Integration test: 431 deterministic EIC matches between OPSD and ENTSOE (28% of ENTSOE), reducing Duke's workload for those pairs to zero. Closes PyPSA#287 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
for more information, see https://pre-commit.ci
ec4364a to
25d6a4e
Compare
Summary
Implements the architecture proposed in #287: deterministic matching on EIC (Energy Identification Code) runs before Duke fuzzy matching in
compare_two_datasets(). Plants sharing an EIC code are matched with certainty and removed from the Duke input, so the fuzzy matcher only handles the residual.Motivation
EIC codes are unique European plant identifiers already loaded from ENTSOE, OPSD, and GEM — but previously unused in the matching decision. This change uses them as a first-pass exact match, providing a robustness guarantee against co-located plant confusion.
Example: Eemshaven harbour, Netherlands
49W000000000066Q49W000000000119VEIC matching deterministically pairs the gas plants via their shared codes, preventing any possibility of Duke merging them with the nearby coal plant based on name/geo similarity (~0.86 JaroWinkler, ~400m apart).
Example: Borssele, Netherlands
Borssele nuclear (OPSD: 492 MW, ENTSOE: 485 MW) is locked to its correct cross-source pair via EIC
49W000000000054X, independent of fuzzy name matching against nearby Borssele wind/coal/solar entries.Integration test results
Implementation
_match_by_eic()function inmatching.py— builds{eic_code → row_index}dicts, finds shared codes via set intersection, greedy 1-to-1 pairingcompare_two_datasets()— calls EIC matching first, drops matched rows, then Duke on residualTest plan
pytest test/test_matching.py— 7 tests passpowerplants(update=True)to verify end-to-end outputCloses #287
🤖 Generated with Claude Code