Skip to content

Add EIC-first deterministic matching before Duke fuzzy matching#289

Open
MaykThewessen wants to merge 2 commits into
PyPSA:masterfrom
MaykThewessen:feature/eic-deterministic-matching
Open

Add EIC-first deterministic matching before Duke fuzzy matching#289
MaykThewessen wants to merge 2 commits into
PyPSA:masterfrom
MaykThewessen:feature/eic-deterministic-matching

Conversation

@MaykThewessen

Copy link
Copy Markdown

Summary

Implements the architecture proposed in #287: deterministic matching on EIC (Energy Identification Code) runs before Duke fuzzy matching in compare_two_datasets(). Plants sharing an EIC code are matched with certainty and removed from the Duke input, so the fuzzy matcher only handles the residual.

Before:
  All sources ──→ Duke fuzzy match ──→ reduce

After:
  All sources ──→ Deterministic EIC join ──→ matched set 1 (high confidence)
                  │
                  └→ EIC-unmatched residual ──→ Duke fuzzy match ──→ matched set 2

                  matched set 1 + matched set 2 ──→ reduce

Motivation

EIC codes are unique European plant identifiers already loaded from ENTSOE, OPSD, and GEM — but previously unused in the matching decision. This change uses them as a first-pass exact match, providing a robustness guarantee against co-located plant confusion.

Example: Eemshaven harbour, Netherlands

Source Plant Fuel Capacity EIC
OPSD Eemscentrale Ec Natural Gas 1929 MW 6 shared codes
ENTSOE Eems Natural Gas 1931 MW same 6 codes
ENTSOE Eemshaven Hard Coal 1580 MW 49W000000000066Q
ENTSOE Eemshaven Natural Gas (Magnum) 1410 MW 49W000000000119V

EIC matching deterministically pairs the gas plants via their shared codes, preventing any possibility of Duke merging them with the nearby coal plant based on name/geo similarity (~0.86 JaroWinkler, ~400m apart).

Example: Borssele, Netherlands

Borssele nuclear (OPSD: 492 MW, ENTSOE: 485 MW) is locked to its correct cross-source pair via EIC 49W000000000054X, independent of fuzzy name matching against nearby Borssele wind/coal/solar entries.

Integration test results

  • 431 deterministic EIC matches between OPSD and ENTSOE (28% of ENTSOE resolved without fuzzy matching)
  • Reduces Duke's workload for those pairs to zero
  • Gracefully falls back to Duke-only when EIC data is unavailable (GEM, GEO, GPD have no EIC codes)

Implementation

  • New _match_by_eic() function in matching.py — builds {eic_code → row_index} dicts, finds shared codes via set intersection, greedy 1-to-1 pairing
  • Modified compare_two_datasets() — calls EIC matching first, drops matched rows, then Duke on residual
  • 7 unit tests covering: basic matching, missing EIC columns, empty sets, NaN-in-set filtering, 1-to-1 enforcement, non-set values

Test plan

  • pytest test/test_matching.py — 7 tests pass
  • Integration test with real OPSD + ENTSOE data — 431 matches, correct plant pairings verified
  • Full pipeline run with powerplants(update=True) to verify end-to-end output

Closes #287

🤖 Generated with Claude Code

@MaykThewessen

Copy link
Copy Markdown
Author

Two developments that raise the value of this PR:

  1. Since feat(OSM): upgrade source from frozen Europe snapshot to live global dataset #292, OSM is a live monthly-refreshed source. The Dutch thermal fleet now carries ref:EU:EIC tags in OSM (Amer, Hemweg, Maasstroom, Rijnmond, Schoonebeek, PerGen, ...), and Extract ref:EU:EIC tags into an EIC column for deterministic dataset matching open-energy-transition/osm-powerplants#13 proposes passing those through to an EIC column in osm_global.csv. With that in place, EIC-first matching would cover the OSM source as well - an end-to-end deterministic path from an OSM tag to the matched output.

  2. The NL slice provides concrete test cases where Duke fuzzy matching currently merges distinct plants that EIC-first matching would keep separate: Maasvlakte Uniper (2306 MW = MPP3 1070 + the retired MV1/2) and the duplicate Eem/Eemscentrale group (six ENTSOE EICs).

Happy to rebase/extend if there's interest in moving this forward.

MaykThewessen and others added 2 commits June 13, 2026 00:07
Implements the architecture proposed in PyPSA#287: deterministic matching on
EIC (Energy Identification Code) runs before Duke fuzzy matching in
compare_two_datasets(). Plants sharing an EIC code are matched with
certainty, then removed from the Duke input so the fuzzy matcher only
handles the residual.

This provides a robustness guarantee against co-located plant confusion.
For example, in the Eemshaven harbour area (Netherlands):

  OPSD "Eemscentrale Ec"  (Natural Gas, 1929 MW, 6 EIC codes)
  ENTSOE "Eems"           (Natural Gas, 1931 MW, same 6 EIC codes)
  ENTSOE "Eemshaven"      (Hard Coal,   1580 MW, different EIC)
  ENTSOE "Eemshaven"      (Natural Gas, 1410 MW, different EIC)

EIC matching deterministically pairs the gas plants via their shared
codes, preventing any possibility of Duke merging them with the nearby
coal plant based on name/geo similarity.

Similarly, Borssele nuclear (OPSD: 492 MW, ENTSOE: 485 MW) is locked
to its correct cross-source pair via EIC 49W000000000054X, independent
of fuzzy name matching against nearby Borssele wind/coal entries.

Integration test: 431 deterministic EIC matches between OPSD and ENTSOE
(28% of ENTSOE), reducing Duke's workload for those pairs to zero.

Closes PyPSA#287

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@MaykThewessen MaykThewessen force-pushed the feature/eic-deterministic-matching branch from ec4364a to 25d6a4e Compare June 12, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use EIC codes as deterministic matching key before Duke fuzzy matching

1 participant