Skip to content

Cache dataset & exchange-set metadata via cross-session sidecars (#467)#468

Merged
philliphoff merged 10 commits into
mainfrom
philliphoff-cache-metadata-sidecars
Jul 18, 2026
Merged

Cache dataset & exchange-set metadata via cross-session sidecars (#467)#468
philliphoff merged 10 commits into
mainfrom
philliphoff-cache-metadata-sidecars

Conversation

@philliphoff

@philliphoff philliphoff commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Implements the metadata-caching work tracked in #467 to speed up dataset loading and exchange-set re-open by (a) making metadata a byproduct of the parse we already do, and (b) persisting it in cross-session disk sidecars so a re-open skips re-parsing.

What landed

WS1 — metadata as a parse byproduct

  • Added IDatasetProcessor.Metadata so every processor exposes its DatasetMetadata (extent, name, edition, …) as a byproduct of the load it already performs — no separate metadata read.
  • Implemented across S-101/S-102/S-104/S-111 processors and the shared GmlDatasetProcessorBase.
  • Routed ViewerDatasetCatalog through the canonical ReadMetadata path, removing the per-render extent rescan.

WS3 — cross-session metadata sidecars

  • Slice 1: DatasetMetadataSerializer + DiskDatasetMetadataCache (EncDotNet.S100.Core/Metadata/) — an mtime+size-keyed, atomically-written, LRU-capped disk sidecar for DatasetMetadata.
  • Slice 3: IDatasetMetadataReader / CachingDatasetMetadataReader probe each S-101 base cell's extent (cheap ISO 8211 metadata peek) so a folder of loose ENC cells is framed to its union bounding box up front. S-57 cells are skipped (no cheap extent).
  • Slice 2: S57CatalogCacheSerializer + IS57CatalogCache / DiskS57CatalogCache (Viewer Services/Caching/) cache the S-57 CATALOG.031 base-cell descriptors so lazy registration of a large exchange set skips re-parsing the catalogue on re-open.
  • DI wiring in App.axaml.cs; new cache directories in ViewerDataPaths (DatasetMetadataCacheDirectory, S57CatalogCacheDirectory).

Closed without changes

WS4 — smaller memoizations: investigated and closed. GetPortrayalContentHashAsync is already memoized and the dataset-bytes hash is computed once in the constructor (nothing to eliminate); compiled-Lua caching is infeasible/low-value (no compiled-form hook in ILuaContext; the instruction cache already skips the whole Lua run on a hit).

WS2 — peek→promote object reuse: remains optional/deferred; only worth doing if profiling shows the now-permitted in-session double parse hurts a real eager flow.

Tests

New xunit coverage for every new public API: metadata serializer round-trip + disk cache, per-processor Metadata byproduct, loose-cell union framing, the S-57 catalogue serializer + disk cache, and an end-to-end re-open-hit against the Synthetic-S57-Framed fixture. Real-data tests use Skip.If. Full EncDotNet.S100.Viewer.Tests suite green.

Closes #467

philliphoff and others added 5 commits July 17, 2026 16:01
@philliphoff
Add IDatasetProcessor.Metadata as a parse byproduct (#467 WS1)
97e7fbc 
Expose a memoized, product-agnostic DatasetMetadata (spec, extent, CRS,
display scale, time coverage) derived once from the dataset each processor
already parsed — never a second parse or a static ReadMetadata(path) call.

- IDatasetProcessor gains a default Metadata member (Spec-only) so
  non-overriding implementers still compile.
- GmlDatasetProcessorBase memoizes the raw WGS-84 envelope and shares that
  single feature scan between the padded render extent
  (ComputeGeographicExtent) and Metadata.Extent, removing the per-render
  O(n) coordinate rescan (previously run on every render).
- S101 derives Metadata from the already-parsed dataset; S102/S104/S111
  derive extent + CRS from the coverage source's already-read
  georeferencing metadata (no HDF5 payload re-read). S104/S111 dcf8
  station series union their station coordinates.
- Tests: GML raw extent agrees with the padded render extent and is
  memoized; S102 UTM cell surfaces CRS + extent; S104/S111 dcf8 derive
  the station bounding box.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a68ee5d9-dc24-43b6-b6dc-1d0d843f0eb8
@philliphoff
Route ViewerDatasetCatalog through canonical ReadMetadata (WS1)
791b6fc 
Consolidate the MCP catalog's GML and S-101 projection helpers onto the
canonical dataset ReadMetadata() surface, completing issue #467 WS1's
metadata-as-byproduct goal for the viewer.

- ProjectGml now returns (data, dataset.ReadMetadata()); the declared
  product edition and raw WGS-84 envelope come from GmlDatasetMetadata,
  replacing the hand-rolled ComputeGmlBounds walk. The catalog keeps its
  canonical spec name (S-57 -> S-101 mapping stays upstream in TryProject)
  and only adopts the metadata edition.
- ProjectS101 derives Spec + Extent from S101Dataset.ReadMetadata(),
  replacing ResolveS101Edition + ComputeS101Bounds.
- Delete ResolveS101Edition, ComputeS101Bounds, ComputeGmlBounds and the
  now-unused EncDotNet.S100.Features using.
- GML entries now surface the declared edition (was default) — an
  intentional improvement; no test pinned GML edition to default.

Tests: retarget the four helper tests to the canonical ReadMetadata API
and add a GML bounds parity test. All 15 ViewerDatasetCatalog tests pass;
formatters (whitespace + IDE0005) clean.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a68ee5d9-dc24-43b6-b6dc-1d0d843f0eb8
@philliphoff
Add cross-session dataset metadata sidecar cache (WS3)
29281c3 
Introduce the foundational, format-agnostic per-dataset metadata sidecar
that makes the relaxed parse-once rule pay off (issue #467 WS3): a
persisted DatasetMetadata lets a later session recover a dataset's cheap
"peek" facts (spec, extent, CRS, display-scale window, temporal coverage)
without re-parsing, and is the primary enabler of loose-dataset lazy
registration.

- DatasetMetadataSerializer: versioned binary serialize / total
  TryDeserialize of DatasetMetadata; corruption or FormatVersion mismatch
  yields null (a miss), mirroring DrawingInstructionSerializer.
- IDatasetMetadataCache + DiskDatasetMetadataCache: one .dmeta sidecar per
  dataset keyed by source last-write time + length; a stale, unwritable,
  or corrupt entry is a miss that never breaks loading. Atomic temp+move
  writes, LRU byte cap, thread-safe (single lock) — same robustness
  contract as DiskPortrayalInstructionCache. GetOrRead runs the cold
  producer on a miss and persists; TryGet is a pure probe.

Tests: 14 new (7 serializer round-trip / corruption / version, 7 disk
cache hit-avoids-parse, cross-instance persistence, mtime/size
invalidation, corrupt=miss, missing-source pass-through). Core README
updated; formatters clean.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a68ee5d9-dc24-43b6-b6dc-1d0d843f0eb8
@philliphoff
Frame loose ENC cell folders up front via metadata sidecar (WS3 Slice 3)
c14699a 
Wire the cross-session per-dataset metadata sidecar (Core WS3 Slice 1)
into ExchangeSetService.OpenLooseCellFolderAsync. Each S-101 base cell's
extent is read once through IDatasetMetadataReader /
CachingDatasetMetadataReader (backed by DiskDatasetMetadataCache, keyed
by source mtime+size) and unioned via ComputeLooseCellUnion, so a loose
`.000` folder frames its extent up front and reuses cached extents on
later sessions with no re-parse. S-57 cells have no cheap extent reader
and are skipped from the union (graceful degradation preserving the
prior null-union behaviour when no extent is available).

- Add IDatasetMetadataReader + CachingDatasetMetadataReader (S-101 wired
  via S101Dataset.ReadMetadata; unknown/unsupported product => null, no
  negative caching).
- Register IDatasetMetadataCache (DiskDatasetMetadataCache, 16 MB cap)
  and IDatasetMetadataReader singletons; add
  ViewerDataPaths.DatasetMetadataCacheDirectory to CacheDirectories.
- Thread onFramingReady + UnionBoundingBox through the loose path.
- Tests: ExchangeSetServiceLooseCellUnionTests (union/skip/null) and
  CachingDatasetMetadataReaderTests (empty/unknown => null; S-101 read
  then hit). READMEs updated.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a68ee5d9-dc24-43b6-b6dc-1d0d843f0eb8
@philliphoff
Cache S-57 exchange-set catalogue descriptors across sessions (WS3 Sl…
f9db45f 
…ice 2)

Re-opening a large S-57 / S-63 exchange set re-parsed its binary
CATALOG.031 every time. Add a cross-session sidecar that caches the
base-cell descriptor list (cell name, base-cell relative path, ordered
update relative paths, EPSG:4326 bbox) keyed by the catalogue file's
mtime + size, so a re-open replays descriptors straight into lazy
registration with zero ISO 8211 re-parse. Dataset bytes are still read
on demand, so the cache never serves stale chart content.

- Add S57CatalogCacheSerializer (versioned binary, corruption=miss),
  IS57CatalogCache + DiskS57CatalogCache (atomic write, mtime+size key,
  LRU byte cap, miss-never-breaks) under Viewer Services/Caching/,
  mirroring DiskDatasetMetadataCache.
- ExchangeSetService.ReadBaseCellsCached wraps
  S57ExchangeSetCatalog.ReadBaseCells for both the eager and lazy S-57
  open paths; ResolveCataloguePath keys the entry and preserves the
  existing catalogue-not-found handling. Null cache => direct read.
- Register IS57CatalogCache (16 MB cap) via DI; add
  ViewerDataPaths.S57CatalogCacheDirectory to CacheDirectories.
- Tests: serializer round-trip/corruption (4), disk cache
  hit/invalidate/corrupt/missing (5), and an end-to-end re-open hit
  against the synthetic Synthetic-S57-Framed CATALOG.031 fixture (1).
  README updated.

Scoped to S-57: only the S-57 branch has lazy load and consumes purely
catalogue-derived descriptors. The S-100 and S-57-eager paths still need
the live catalogue object to dispatch datasets, so caching there would
only shift framing marginally.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a68ee5d9-dc24-43b6-b6dc-1d0d843f0eb8
Copilot AI review requested due to automatic review settings July 18, 2026 01:26
@github-actions

github-actions Bot commented Jul 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Performance Gate

✅ PASSED — no regressions.

Threshold: 10.0%, MAD multiplier (k): 3.0, retry-zone mult: 2.0×

Scenario summary

Scenario Status Δ median (%) z (Δ/MAD) Base median (ms) Samples (b/c)
exchange-set-open ✅ pass -0.5 -0.22 1.68 5/5
s101-portray-cold ✅ pass -3.9 -0.60 0.24 5/5
s101-portray-warm ✅ pass -19.4 -1.15 0.26 5/5
s101-real-cold ✅ pass -4.7 -0.45 0.26 5/5
s101-real-warm ✅ pass -10.9 -6.50 0.27 5/5
s101-render-warm ✅ pass -8.7 -0.36 0.21 5/5
s102-coverage ✅ pass -72.2 -4.19 0.51 5/5
s102-coverage-open ✅ pass +1.9 +0.61 3.06 20/20
s102-coverage-render-large ✅ pass +5.2 +1.84 0.17 5/5
s102-real-warm ✅ pass +8.4 +1.16 0.28 5/5
s111-real-warm ✅ pass +6.7 +159.00 0.24 5/5
s124-vector ✅ pass +13.2 +5.75 0.17 5/5
s201-vector ✅ pass -5.8 -0.35 0.18 5/5

exchange-set-open

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 1.68 1.67
Baseline MAD (ms) 0.04 —
Δ median — -0.5%
z (Δ/MAD) — -0.22

Spans (sum of all iterations)

Span Baseline (ms) Candidate (ms) Delta Status
s100.asset.read 4.48 4.53 +1.0% ▫️
s100.exchangeset.parse 44.33 45.57 +2.8% ▫️

Metrics

Metric Baseline Candidate Delta Status
s100.asset.read.duration 0.20 0.21 +6.3% ❌

s101-portray-cold

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 0.24 0.23
Baseline MAD (ms) 0.02 —
Δ median — -3.9%
z (Δ/MAD) — -0.60

s101-portray-warm

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 0.26 0.21
Baseline MAD (ms) 0.04 —
Δ median — -19.4%
z (Δ/MAD) — -1.15

s101-real-cold

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 0.26 0.24
Baseline MAD (ms) 0.03 —
Δ median — -4.7%
z (Δ/MAD) — -0.45

s101-real-warm

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 0.27 0.24
Baseline MAD (ms) 0.00 —
Δ median — -10.9%
z (Δ/MAD) — -6.50

s101-render-warm

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 0.21 0.19
Baseline MAD (ms) 0.05 —
Δ median — -8.7%
z (Δ/MAD) — -0.36

s102-coverage

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 0.51 0.14
Baseline MAD (ms) 0.09 —
Δ median — -72.2%
z (Δ/MAD) — -4.19

s102-coverage-open

Iteration statistics

Stat Baseline Candidate
Samples 20 20
Median (ms) 3.06 3.12
Baseline MAD (ms) 0.10 —
Δ median — +1.9%
z (Δ/MAD) — +0.61

Spans (sum of all iterations)

Span Baseline (ms) Candidate (ms) Delta Status
s100.dataset.open 451.23 453.04 +0.4% ▫️
s100.hdf5.dataset.read 162.68 161.82 -0.5% ▫️
s100.hdf5.file.open 21.21 21.27 +0.3% ▫️
s100.hdf5.open 20.68 20.98 +1.4% ▫️

Metrics

Metric Baseline Candidate Delta Status
s100.hdf5.read.bytes 36456.00 36456.00 +0.0% ▫️
s100.hdf5.read.duration 26.85 26.59 -1.0% ▫️
s100.hdf5.read.duration 31.66 31.11 -1.7% ▫️
s100.hdf5.read.duration 11.46 11.48 +0.2% ▫️

s102-coverage-render-large

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 0.17 0.18
Baseline MAD (ms) 0.00 —
Δ median — +5.2%
z (Δ/MAD) — +1.84

s102-real-warm

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 0.28 0.30
Baseline MAD (ms) 0.02 —
Δ median — +8.4%
z (Δ/MAD) — +1.16

s111-real-warm

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 0.24 0.25
Baseline MAD (ms) 0.00 —
Δ median — +6.7%
z (Δ/MAD) — +159.00

s124-vector

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 0.17 0.20
Baseline MAD (ms) 0.00 —
Δ median — +13.2%
z (Δ/MAD) — +5.75

s201-vector

Iteration statistics

Stat Baseline Candidate
Samples 5 5
Median (ms) 0.18 0.17
Baseline MAD (ms) 0.03 —
Δ median — -5.8%
z (Δ/MAD) — -0.35

Generated by EncDotNet.S100.PerfReport gate command

Copilot AI reviewed Jul 18, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements issue #467’s metadata-caching plan to speed up dataset loading and exchange-set re-open by (1) making DatasetMetadata a byproduct of existing dataset parses and (2) persisting “peek” metadata and S-57 catalogue descriptors as cross-session on-disk sidecars for re-open hits.

Changes:

  • Added IDatasetProcessor.Metadata and implemented memoized, parse-byproduct metadata across S-101/S-102/S-104/S-111 processors and GmlDatasetProcessorBase (single feature scan shared with render extent computation).
  • Introduced disk sidecar caches: DatasetMetadataSerializer + DiskDatasetMetadataCache (Core) and S57CatalogCacheSerializer + DiskS57CatalogCache (Viewer).
  • Updated Viewer services to use canonical ReadMetadata() for bounds, added loose ENC cell union framing via cached metadata probes, and wired new caches via DI + ViewerDataPaths.

Reviewed changes

Copilot reviewed 34 out of 34 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/EncDotNet.S100.VisualRegression.Tests/DatasetProcessorMetadataTests.cs Adds regression test validating IDatasetProcessor.Metadata memoization + extent parity for a GML processor.
tests/EncDotNet.S100.Viewer.Tests/ViewerDatasetCatalogTests.cs Updates tests to use ReadMetadata() for edition/bounds; adds parity guard for GML bounds.
tests/EncDotNet.S100.Viewer.Tests/ViewerDataPathsTests.cs Updates expected cache-directory counts after new cache dirs were added.
tests/EncDotNet.S100.Viewer.Tests/S57CatalogCacheSerializerTests.cs Adds serializer round-trip + corruption/version mismatch tests for S-57 catalogue sidecar payload.
tests/EncDotNet.S100.Viewer.Tests/ExchangeSetServiceLooseCellUnionTests.cs Adds unit tests for loose-cell union framing based on metadata probes.
tests/EncDotNet.S100.Viewer.Tests/ExchangeSetServiceLoaderTests.cs Adds E2E test ensuring S-57 catalogue descriptors are served from cache on reopen.
tests/EncDotNet.S100.Viewer.Tests/DiskS57CatalogCacheTests.cs Adds disk cache behavior tests (hit/miss, invalidation, corrupt sidecar, missing source).
tests/EncDotNet.S100.Viewer.Tests/CachingDatasetMetadataReaderTests.cs Adds tests for path-based metadata reader backed by disk cache (incl. real-data skip).
tests/EncDotNet.S100.Pipelines.Tests/S111Dcf8ProcessorTests.cs Adds test verifying S-111 DCF8 processor metadata extent/memoization.
tests/EncDotNet.S100.Pipelines.Tests/S104Dcf8ProcessorTests.cs Adds test verifying S-104 DCF8 processor metadata extent/memoization.
tests/EncDotNet.S100.Pipelines.Tests/S102Edition21Tests.cs Adds test verifying S-102 processor metadata surfaces CRS/extent and is memoized.
tests/EncDotNet.S100.Core.Tests/DiskDatasetMetadataCacheTests.cs Adds tests for disk-backed dataset metadata cache (hit/miss, invalidation, corruption).
tests/EncDotNet.S100.Core.Tests/DatasetMetadataSerializerTests.cs Adds serializer round-trip + corruption/version mismatch tests for DatasetMetadata.
src/EncDotNet.S100.Viewer/ViewerDataPaths.cs Adds new cache directories for dataset-metadata sidecars and S-57 catalogue descriptors.
src/EncDotNet.S100.Viewer/Services/ViewerDatasetCatalog.cs Switches GML/S-101 bounds+edition to canonical ReadMetadata() and removes hand-rolled scans.
src/EncDotNet.S100.Viewer/Services/IDatasetMetadataReader.cs Introduces internal interface for cheap per-file metadata probing.
src/EncDotNet.S100.Viewer/Services/ExchangeSetService.cs Adds loose-cell union framing via metadata reader; adds cached S-57 base-cell reads via sidecar.
src/EncDotNet.S100.Viewer/Services/CachingDatasetMetadataReader.cs Implements cached path-based metadata reader (currently wired for S-101).
src/EncDotNet.S100.Viewer/Services/Caching/S57CatalogCacheSerializer.cs Implements binary serializer for cached S-57 base-cell descriptors.
src/EncDotNet.S100.Viewer/Services/Caching/IS57CatalogCache.cs Adds abstraction for cross-session S-57 catalogue-descriptor cache.
src/EncDotNet.S100.Viewer/Services/Caching/DiskS57CatalogCache.cs Implements disk-backed S-57 catalogue descriptor cache with atomic writes + LRU eviction.
src/EncDotNet.S100.Viewer/README.md Documents loose-cell framing and S-57 catalogue descriptor caching behavior.
src/EncDotNet.S100.Viewer/App.axaml.cs Wires dataset-metadata and S-57 catalogue caches + reader into DI container.
src/EncDotNet.S100.Datasets.Pipelines/S111DatasetProcessor.cs Adds memoized Metadata derived from gridded source or fixed-station coordinates.
src/EncDotNet.S100.Datasets.Pipelines/S104DatasetProcessor.cs Adds memoized Metadata derived from gridded source or fixed-station coordinates.
src/EncDotNet.S100.Datasets.Pipelines/S102DatasetProcessor.cs Adds memoized Metadata derived from coverage georeferencing + CRS.
src/EncDotNet.S100.Datasets.Pipelines/S101DatasetProcessor.cs Adds memoized Metadata derived from already-parsed S-101 dataset.
src/EncDotNet.S100.Datasets.Pipelines/README.md Documents “metadata as a parse byproduct” behavior and expectations.
src/EncDotNet.S100.Datasets.Pipelines/IDatasetProcessor.cs Adds Metadata to processor interface via default implementation.
src/EncDotNet.S100.Datasets.Pipelines/GmlDatasetProcessorBase.cs Memoizes a raw envelope once and reuses it for metadata extent + padded render extent.
src/EncDotNet.S100.Core/README.md Updates Core README to document the new cross-session metadata sidecar cache.
src/EncDotNet.S100.Core/Metadata/IDatasetMetadataCache.cs Adds public cache interface for persisted DatasetMetadata keyed by mtime/size.
src/EncDotNet.S100.Core/Metadata/DiskDatasetMetadataCache.cs Implements disk-backed metadata cache with atomic writes + LRU eviction + robustness contract.
src/EncDotNet.S100.Core/Metadata/DatasetMetadataSerializer.cs Implements versioned binary serializer for DatasetMetadata used by the disk cache.

Comment thread src/EncDotNet.S100.Datasets.Pipelines/IDatasetProcessor.cs
@philliphoff
Clarify IDatasetProcessor.Metadata default-vs-override doc
7cb54bd 
The XML doc said the value is "derived once ... and cached", but the
default interface implementation returns a fresh Spec-only instance on
each access. Distinguish the overriding derive-once-and-cache behaviour
from the default per-access minimal value.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a68ee5d9-dc24-43b6-b6dc-1d0d843f0eb8
Copilot AI review requested due to automatic review settings July 18, 2026 01:31
Copilot AI reviewed Jul 18, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 1 comment.

Comment thread src/EncDotNet.S100.Core/Metadata/IDatasetMetadataCache.cs Outdated
@philliphoff
Clarify IDatasetMetadataCache.Misses counter semantics
103bb61 
TryGet increments the miss counter without invoking a producer, so the
"fell through to the producer" wording was inaccurate for TryGet misses.
Document Misses as any lookup not served from a valid cached entry, via
either GetOrRead (where the producer runs) or TryGet (returns false).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a68ee5d9-dc24-43b6-b6dc-1d0d843f0eb8
Copilot AI review requested due to automatic review settings July 18, 2026 01:37
Copilot AI reviewed Jul 18, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.

Comment thread src/EncDotNet.S100.Datasets.Pipelines/GmlDatasetProcessorBase.cs Outdated
Comment thread src/EncDotNet.S100.Viewer/Services/Caching/S57CatalogCacheSerializer.cs Outdated
@philliphoff
Harden memoized envelope publication and clarify update-list build
27be1f4 
- GmlDatasetProcessorBase: make the memoized RawEnvelope a reference type
  so the unsynchronized ??= publishes it via an atomic reference write.
  Metadata (catalog framing) and ComputeGeographicExtent (render) can run
  on different threads; a multi-field record struct risked torn reads.
- S57CatalogCacheSerializer.TryDeserialize: build the update paths as a
  List<string> directly and upcast once, dropping the brittle
  IReadOnlyList->List cast in the fill loop.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a68ee5d9-dc24-43b6-b6dc-1d0d843f0eb8
Copilot AI review requested due to automatic review settings July 18, 2026 01:45
Copilot AI reviewed Jul 18, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 3 comments.

Comment thread src/EncDotNet.S100.Core/Metadata/DiskDatasetMetadataCache.cs
Comment thread src/EncDotNet.S100.Viewer/Services/Caching/DiskS57CatalogCache.cs
Comment thread src/EncDotNet.S100.Viewer/Services/ExchangeSetService.cs
@philliphoff
Guard sidecar payload length and probe loose cells off the UI thread
33dce8b 
- DiskDatasetMetadataCache / DiskS57CatalogCache: reject a payload length
  larger than the remaining sidecar bytes before ReadBytes allocates, so a
  corrupt frame degrades to a cache miss instead of a huge allocation/OOM.
- ExchangeSetService.OpenLooseCellFolderAsync: run the up-front loose-cell
  union probe on a worker thread (Task.Run). OpenAsync is dispatched on the
  UI thread, and a first-open cache miss parses every cell's metadata; that
  work must not block the UI. First-open framing is preserved (the probe is
  the intentional, cross-session-cached parse under the relaxed rule).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a68ee5d9-dc24-43b6-b6dc-1d0d843f0eb8
Copilot AI review requested due to automatic review settings July 18, 2026 01:55
Copilot AI reviewed Jul 18, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.

Comment thread src/EncDotNet.S100.Core/Metadata/DatasetMetadataSerializer.cs
Comment thread src/EncDotNet.S100.Viewer/Services/Caching/S57CatalogCacheSerializer.cs
@philliphoff
Bound sidecar string reads against remaining payload bytes
f8c3bd8 
BinaryReader.ReadString() allocates from a 7-bit length prefix before it
discovers EOF, so a corrupt sidecar could trigger a very large allocation
instead of degrading to a cache miss. Route every string read in
DatasetMetadataSerializer and S57CatalogCacheSerializer through a
ReadBoundedString helper that rejects a length prefix larger than the
remaining (in-memory, size-framed) payload before allocating. The read
stays wire-compatible with BinaryWriter.Write(string). Adds a regression
test per serializer for an oversized length prefix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a68ee5d9-dc24-43b6-b6dc-1d0d843f0eb8
Copilot AI review requested due to automatic review settings July 18, 2026 02:09
Copilot AI reviewed Jul 18, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated no new comments.

@philliphoff
philliphoff merged commit 2eba755 into main Jul 18, 2026
13 checks passed
@philliphoff
philliphoff deleted the philliphoff-cache-metadata-sidecars branch July 18, 2026 02:26
@philliphoff philliphoff mentioned this pull request Jul 18, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: Metadata-caching for faster dataset loading (parse-once + cross-session sidecar)

2 participants

@philliphoff