Load logs newest-first for fast first paint and cut resolver memory#615
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reworks the log-load + resolution pipeline to prioritize “newest-first” rendering (fast first paint) and reduces memory/time overhead in message resolution and EVT marshalling.
Changes:
- Load logs newest-first via reverse-direction reading and a direction-agnostic
NewestBookmark(IEventLogReader), plus eager first-paint partial dispatch. - Introduce a two-lane priority gate (
PrioritySemaphore,ResolutionPriority) so first-screenful resolution preempts bulk work across concurrent loads. - Reduce provider message-table overhead via compact/lazy message sources (
ILazyMessageSource,CompactMessageStore,LegacyMessageFileSource,MessageTableReader) and optimize EVT variant/system-property reads.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/Unit/EventLogExpert.Runtime.Tests/EventLog/EffectsTests.cs | Adds reverse eager-load behavior tests and helper fakes/utilities. |
| tests/Unit/EventLogExpert.Runtime.Tests/Concurrency/PrioritySemaphoreTests.cs | Adds correctness/stress tests for priority semaphore behavior. |
| tests/Unit/EventLogExpert.Provider.Tests/Resolution/ProviderDetailsTests.cs | Adds coverage for compact/lazy message storage preserving ordering/fields. |
| tests/Unit/EventLogExpert.Eventing.Tests/PublisherMetadata/MessageTableReaderTests.cs | Adds tests for bounds-checked message-table walking and lazy source parity. |
| tests/Unit/EventLogExpert.Eventing.Tests/Interop/NativeMethodsEvtTests.cs | Updates/extends marshalling tests for pointer-indexed EvtVariant reads and system property extraction. |
| tests/Integration/EventLogExpert.Eventing.IntegrationTests/Readers/EventLogReaderReverseTests.cs | Adds integration tests for reverse reading and NewestBookmark semantics. |
| src/EventLogExpert/EventLogExpert.csproj | Enables Server GC for Windows builds to improve parallel resolve throughput. |
| src/EventLogExpert.Runtime/EventLog/OpenLogEffects.cs | Implements newest-first reading, eager first paint partial dispatch, and priority-gated resolution. |
| src/EventLogExpert.Runtime/EventLog/IEventLogReaderFactory.cs | Introduces factory abstraction to enable test substitution of readers. |
| src/EventLogExpert.Runtime/EventLog/EventLogReaderFactory.cs | Default factory implementation for creating EventLogReader instances. |
| src/EventLogExpert.Runtime/DependencyInjection/RuntimeServiceCollectionExtensions.cs | Registers IEventLogReaderFactory in runtime DI. |
| src/EventLogExpert.Runtime/Concurrency/ResolutionPriority.cs | Defines priority classes for resolution gating. |
| src/EventLogExpert.Runtime/Concurrency/PrioritySemaphore.cs | Adds the priority-aware async semaphore implementation. |
| src/EventLogExpert.Provider/Resolution/ProviderDetails.cs | Switches to lazy/compact message sources and exposes lazy sources for reuse/fallback. |
| src/EventLogExpert.Provider/Resolution/ILazyMessageSource.cs | New interface for lazy message-table lookup/materialization. |
| src/EventLogExpert.Provider/Resolution/CompactMessageStore.cs | New compact in-memory message store with lazy/cached per-id materialization. |
| src/EventLogExpert.Eventing/Readers/IEventLogReader.cs | New reader abstraction exposing NewestBookmark and batch reads. |
| src/EventLogExpert.Eventing/Readers/EventLogReader.cs | Adds reverse-direction querying and NewestBookmark capture for reverse reads; adds handle cleanup on failed reads. |
| src/EventLogExpert.Eventing/PublisherMetadata/ProviderMetadata.cs | Updates EVT handle extraction to match new EvtVariant field naming. |
| src/EventLogExpert.Eventing/PublisherMetadata/MessageTableReader.cs | New bounds-checked message-table reader (count + per-id extraction). |
| src/EventLogExpert.Eventing/PublisherMetadata/LegacyMessageFileSource.cs | New lazy message source over provider DLL message tables. |
| src/EventLogExpert.Eventing/PublisherMetadata/EventMessageProvider.cs | Migrates legacy message loading to the new table reader + lazy source path. |
| src/EventLogExpert.Eventing/Interop/NativeMethods.Evt.cs | Switches to pointer-indexed EvtVariant reads, adds EvtQueryWithFlags, and optimizes system-property extraction. |
| src/EventLogExpert.Eventing/Interop/NativeMethods.cs | Adds SizeofResource P/Invoke used by message table reader. |
| src/EventLogExpert.Eventing/Interop/EvtVariant.cs | Updates EvtVariant to explicit, blittable layout with typed union fields and stable offsets. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jschick04
force-pushed
the
jschick/log-load-perf
branch
from
e1f3270 to
38011a3
Compare
jschick04
force-pushed
the
jschick/log-load-perf
branch
from
38011a3 to
1984478
Compare
jschick04
force-pushed
the
jschick/log-load-perf
branch
from
1984478 to
a820254
Compare
jschick04
force-pushed
the
jschick/log-load-perf
branch
from
a820254 to
6b354a2
Compare
jschick04
force-pushed
the
jschick/log-load-perf
branch
from
6b354a2 to
3d39cd3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Overhauls log loading so the newest events render almost immediately and message resolution uses far less memory and time.
EventLogReadergains reverse-direction reading and a direction-agnostic newest bookmark (IEventLogReader), so a freshly opened log paints its first screenful from the newest events instead of reading the whole log first.PrioritySemaphore,ResolutionPriority,EventLogReaderFactory).EVT_VARIANTlayout), value-type system properties read without boxing, batches of 256, handles released on a failed read.ILazyMessageSource,CompactMessageStore,LegacyMessageFileSource,MessageTableReader) instead of materializing every provider's full table - removing a large regression where modern EventSource providers each loaded a 65,536-entry legacy table to serve ~1 lookup.Measured (5 largest live logs, vs last stable release)
First paint 169-369x faster; read throughput +5-10% with ~45% less read allocation/event; provider-metadata memory back to stable-release level (was ~4.5x regressed); ~46% less per-event memory; resolve on par to ~5% faster.
Tests
Adds reverse-read, priority-semaphore, message-table-reader, and event-variant/
GetEventRecordmarshalling tests. Unit suites pass (1986 local); resolution parity verified byte-for-byte over 69,353 exported events.