Repository navigation
Speed up startup and map rendering - #218
Merged
Slazanger merged 39 commits intoNov 10, 2026
Merged
Conversation
JumpRangeCache.dat held every system within 10 LY of every other as 35 MB of XML. Loading it took ~750 ms at startup, and InitNavigation then recomputed each of its 1,027,811 distances with decimal maths and DecimalEx.Sqrt (~780 ms), so the cache only saved the pair search. Build the table in InitNavigation instead: bucket the systems a capital can jump into (not highsec, Pochven or Zarzakh) into 10 LY cubes and measure each system against the 27 cubes around it, in double. That takes ~40-60 ms and gives the identical table (same destinations for all 5,202 systems, ranges within 1.4e-14 LY); 200 seeded capital routes are unchanged. Storing RangeLY as double also makes capital routing about 2.3x faster. Drop JumpRangeCache.dat, its generation in CreateFromScratch and Navigation.CreateStaticNavigationCache.
The MainWindow constructor called UpdateMetaliminalStorms(), which fetches the EVE-Scout storm tracker with a blocking HtmlWeb.Load on the UI thread. The startup window waited for that round trip, and with HtmlWeb's default 100 s timeout a slow or unreachable site could hold it for up to 100 s. UpdateMetaliminalStormsAsync() does the download and the strong/weak area calculation on a worker thread, then swaps the list and raises StormsUpdateEvent on the UI thread, as the Thera/Turnur updates do; the maps read MetaliminalStorms while drawing. Failures are logged.
IsSystemOnMap scanned every map system comparing names, although MapSystems is keyed by system name (all 6,306 entries in MapLayout.dat match, and CreateFromScratch is the only writer). It has 26 call sites, including the 2-second map refresh and the zKillboard feed filter. A null name still returns false.
The feed's CollectionView filter copied the whole kill stream, filtered it by region and took the newest 30 for every item it tested, so one Refresh() cost O(kills^2 x region systems): 70-190 ms on the UI thread with 250-750 kills in the default 30 minute window. Every new kill triggers a refresh on the next 1-second tick. Work out the visible kills once and reuse them until the stream changes (kills enter at the front and expire from the back, so its count, first or last kill changes) or the region or region filter changes. A refresh now costs about 0.02 ms. Replaying 20,000 random feed events (943,384 filter evaluations) gives the same result as the old filter every time.
A console app in tools/PerfBench (not part of SMT.sln) that references EVEData and times the startup data path, the jump-range table, log tailing, intel matching, the zKillboard filter and the universe gate list. It also checks behaviour: nav-check compares the jump table and 200 seeded capital routes with a baseline recorded before the runtime jump table (tools/PerfBench/baseline/routes.txt), zkb-check replays random feed activity against the old filter, and fsw-quiet reproduces FileSystemWatcher missing a quiet log file's new line. It never runs the EveManager constructor, so it doesn't touch %AppData%\SMT; scratch files go to %TEMP%\SmtPerfBench.
Once the main window has rendered, SMT writes one line to its log with the time from process start to the first frame and how long each startup phase took (window, settings, EVE data, log watchers, navigation, region map, universe map, ...). SMT.exe --perftest[=results.json] drives the running app through a fixed script after the first frame: region switches, full and partial region map redraws, system clicks through the real click handler, a data overlay toggle, an unrelated setting change, and 20 s idle with the region map in view. It writes the timings, redraw counts, CPU, allocation and GC figures as JSON and closes. Such a run changes the region and toggles on purpose, so it saves nothing on exit except the characters, whose ESI tokens may have been refreshed; it also skips the new-version check so no dialog can stop it. ReDrawMap now counts full and partial redraws and their time.
SMT watched Chatlogs and Gamelogs with FileSystemWatchers. EVE keeps its logs open, and NTFS updates such a file's directory entry only when another process opens it, so a background loop opened every recent file every 1.5 s to make the watchers fire. On each event SMT reopened the file, re-read it from the start and skipped the lines it had seen. That had three problems: - The loop picked files by their directory-listed last-write time, the same stale metadata, so a channel quiet for over 10 minutes stopped being opened and its next line raised no event (reproduced). - Intel arrived up to ~1.5 s late (median 0.8 s in a replay). - Every change re-read the whole file: 3 ms and 8.6 MB of garbage per event for a 4 MB game log, growing with the session. LogFolderMonitor (LogTail.cs) replaces the watchers and the loop: a name-only scan finds session files once a second, and every 250 ms each followed file's size is read from an open handle and only the appended bytes are decoded, as complete lines. Files older than the last downtime (26 h) are ignored. In a replay of 8 intel channels and 4 game logs, latency drops to a median of 140 ms (max 250 ms) and CPU to a third. Also: - New lines reach the UI once per poll; the intel and game log lists insert them oldest first, and every new intel line is checked for alerts. - A Local log seen for the first time sets the character's current system from its latest "Channel changed" line instead of clearing it. - Intel channel filters match the file name, not the whole path. - A game log line without "(type)" no longer loops forever. - Read delays (read time minus the line's EVE timestamp) are logged every 5 minutes and at shutdown with the new AppLog.Metrics, which writes to the log without replacing the status bar text; the startup timing line uses it too.
stat-check shows which size queries see a held-open file grow (a path query and a handle do, directory enumeration doesn't). log-check runs LogFolderMonitor through 14 cases with EVE-like writers: skipped existing content, partial lines, a quiet spell, new and still-empty files, long lines, truncation and old sessions. log-latency replays the same traffic through the old watcher/touch-loop/re-read approach and through LogFolderMonitor and reports latency, CPU and allocations.
Measured with --perftest, the region map was fully rebuilt (~90 ms each in Delve) twice per region switch, 2.5 times per data overlay toggle and 1.5 times for a setting it doesn't even read, and clicked systems were redrawn twice: - SelectRegion set MapConf.DefaultRegion, and MainWindow redrew the region map on any MapConfig change before SelectRegion redrew it. - The NPC/pod/ship kill and jump toggles turn each other off, and every setter raised PropertyChanged without checking for a change, each one triggering a full redraw. - SelectSystem set the dropdown first, whose SelectionChanged selected the system again and redrew. RegionControl.RequestRedraw(full) now posts one redraw before the next frame: requests in between coalesce, a redraw that happens meanwhile satisfies them, and a hidden map redraws when it is shown again. The settings handler only asks for it for settings the region map reads; unknown settings still redraw. The setters return when nothing changes (RegionControl, the UniverseControl toggles and MainWindow's ActiveCharacter, whose event makes every open overlay redraw), and SelectSystem sets SelectedSystem before the dropdown and ignores the dropdown's own change. The client-sync timer switches characters only when the focused client belongs to someone else. SelectRegion and MapObjectChanged no longer nest Dispatcher.Invoke on the UI thread. Full redraws per action are now 1 per region switch, 1 per data toggle and 0 for unrelated settings; a click redraws once.
The --perftest run now opens an overlay for a test character in 1DQ1-A (not ESI-linked, so never saved), refreshes it 30 times and reports the refresh time, how many ToolTipOpening handlers one system shape carries afterwards and what opening its tooltip costs. Overlay gets an internal SuppressPlacementSaving switch so the test overlay doesn't store its window placement. Each measured action also reports the time spent inside ReDrawMap(true) itself, apart from the layout pass and rendering that follow.
DrawSystemToOverlay runs for every system on every overlay refresh (each jump of any character, and every second with client sync before the previous commit), on shapes that come from pools and are reused. It added two ToolTipOpening handlers and a ToolTipClosing handler each time and never removed them: after 32 refreshes one system shape carried 64 ToolTipOpening handlers, and opening its tooltip ran every copy, each scanning all systems. It also built a new ContextMenu, a new security brush and the tooltip text for every system on every refresh. Subscribe each handler once (remove before adding), create the ToolTip object once and fill it when it opens, build a system's ContextMenu only when the system or the character's ESI link changes, and share frozen security brushes. UpdateRouteData, which runs every second, now skips redrawing an unchanged route, and returns old route lines to their pools with their dash animation stopped instead of dropping them. Measured with --perftest: 2 ToolTipOpening handlers per system after 32 refreshes instead of 64 (and no longer growing), and an overlay refresh of 110 systems in 10-13 ms instead of 18-21 ms.
After the overlay test the run measures idle time three more ways: the region map with no system selected (no highlight animation), a region with jump bridges from the profile's list (also reporting how many shapes are animated), and the universe map in view.
Each idle measurement also reports how many shapes on the region canvas have a running animation (on a property or on their render transform).
Two animations cost most of the region map's idle CPU: - The selected system's highlight was rebuilt by every 2 s partial redraw, which restarted its single 6 s turn, so it never stopped turning. It now stays on the map until the selection changes or a full redraw clears the canvas, and turns once as intended. - Each jump bridge and faction warfare line ran its own endless dash animation at 20 frames a second, although the dashes only creep 2 units a second (0.1 a frame). They now share one frozen timeline per kind at 5 frames a second, which looks the same. Measured with --perftest while idle: with 11 jump bridge lines in view (Etherium Reach) CPU fell from 100-125 ms/s to 22-46 ms/s and allocations from 3.1-3.5 to 0.8 MB/s; with a system selected in Delve the median fell from ~30 to ~17 ms/s.
A --perftest run no longer restores the saved window placement: the main window opens in the middle of the first secondary monitor (the primary if there is none) at 1280x900 pixels, the same on every run, and the test overlay at 500x500 next to it (Overlay gets an internal PlacementOverride applied after its own placement). The run also probes the universe map's right-click lookup for every system when the build has one. PerfMetrics.MarkStartup ignores marks after the first frame, so code that also runs later can mark its startup steps.
Building the universe map was the slowest step of startup: 2.3-3.1 s of a 4.6-4.9 s start to the first frame. Startup marks inside it showed three loops of 0.6-0.8 s each (6,872 gate lines, 5,202 system shapes, 5,202 region blobs): every DrawLine/DrawEllipse took ~0.1 ms because the pens were never frozen, so WPF tracked the same unfrozen pen as a dependent resource on every call. - Freeze the pens and brushes the universe map draws with, and create the jump range pens once instead of one per draw call. - Draw all systems into one visual instead of one visual per system; a right click finds the system under the pointer geometrically (FindSystemAt; every system checked by --perftest). - Draw all system names into one visual, and lay them out when they are first shown (they're hidden until zoomed in) or when the UI goes idle. - Draw the ESI data layer into one visual instead of 5,202 mostly empty ones. - De-duplicate the gate list with a HashSet instead of a linear scan per jump (60 ms to ~4 ms, same 6,872 gates). - Startup marks for the universe map's steps. The universe map step now takes ~16 ms, and the first frame comes ~2.4 s sooner (4.2 s to 1.7 s with tiered compilation off).
EveManager.StartLoadingStaticData() reads MapLayout.dat, Systems.dat and ShipTypes.dat (~15 MB of XML) on a worker thread; MainWindow starts it first thing, so it overlaps InitializeComponent, and LoadFromDisk takes the result. The intel alert player (NAudio output plus the mp3 reader) is opened on first use instead of in the constructor. The "EVE data" startup step drops from ~310 ms to ~30 ms (warm) and the 40 ms audio step goes; the first frame comes ~350 ms sooner.
The hover handler's body moves into UpdateSystemPopup(shape, show), so the perftest can run it without a real mouse: for 20 systems it opens the popup with its highlights, then leaves. It times both, records the popup's laid-out size per system (to compare content between builds), and times the intel row highlight on its own, with the intel list filled to its 250-row limit by test rows that are removed afterwards.
Hovering a system took ~20 ms on the UI thread. Splitting it showed 0.8 ms of SMT's own work (labels, brushes, the jump bridge scan) and ~19 ms in Popup.IsOpen = true: WPF creates the popup's window on every open (~9 ms, it is destroyed on close) and lays out its content, where each Label's template adds four elements. The popup's rows are now plain TextBlocks with one frozen text brush, and the unused Label made per character in the system is gone. With --perftest (20 systems, 2 runs per build, tiered compilation off) a hover enters in 18.3 ms instead of 19.3-20.3 ms and allocates 0.68 MB instead of 0.79 MB; the popup lays out to the same size for every system. The intel row highlight costs 0.1 ms and is left as it is.
Navigation.GetSystemsXJumpsFrom (danger zones at 1-6 jumps, 5 by default, on every location change; storms at 1 and 3) followed every path of up to X jumps without a visited set, and after each step rescanned its whole result list with List.Contains. It ran off the UI thread, but held the character's update lock while it did. It is now a breadth-first search with HashSets. PerfBench danger-check compares it with the old walk: 0 differences for all 5,202 start systems at 1-5 jumps and 1,398 at 6 (the old walk is slow there), with the old walk's median and worst case going from 0.29 ms / 76 ms to 0.007 ms at 5 jumps, and from 1.8 ms / 600 ms to 0.014 ms / 0.11 ms at 6.
The region map, the universe map and the route panel each copied and sorted all 5,202 systems for their system pickers at startup (2-5 ms each). EveManager.SystemsByName sorts them once, with the same comparison, and the five pickers share it.
Every visual below the canvas, templates included: what layout and rendering visit on a full redraw.
After the full redraws, each system name block is rendered on its own and its pixels hashed, and the whole region map canvas is saved as a PNG next to the results, so two builds' rendering can be compared name by name and by eye.
Only ~25 ms of a full region map redraw (~100-190 ms) is SMT's own code; the rest is WPF laying out the elements. Every system name was a Grid > StackPanel > Label, and the Label's template adds a Border, a ContentPresenter, an AccessText and its TextBlock. - System names (and the ice belt / blue star icons) are TextBlocks, aligned by layout as the Label's content was, so they render on the same pixels. - The ~20 brushes the static layer shares are frozen; the per-system security brush is only made when it is used, and the unused Polygon per system is gone. With --perftest in Delve (6 alternating runs per build, tiered compilation off): 1,128 -> 732 visuals after a full redraw; full redraw median 164 -> 98 ms, region switch 167 -> 105 ms, data overlay toggle 197 -> 127 ms; 14.4 -> 10.2 MB allocated per full redraw. All 110 system names render pixel-identical to before, and so does the whole map apart from live data that changed between runs.
SHA-1 raised CA5350 in a clean build. The hash only compares renders.
LogFolderMonitor starts reading the logs it finds at startup from their end, and a Local log only listed its character once a new line arrived. EVE writes nothing to Local while a character sits in a quiet system (and buffers what it does write), so the character wasn't listed: the active character couldn't be picked and the maps couldn't follow it. Before the log follower, the loop that opened the files made the FileSystemWatcher fire, which listed the character from the log header. Now every Local log found lists its character straight away, in the system of its last "Channel changed to Local" line. Per character only the newest session counts, and only if it started after the last daily downtime (11:00 UTC), which logs everyone out: an older session would put the character back where it was hours ago, or list one that is offline. In a perftest with quiet Local logs, SMT now knows the logged-in character and its system 5 s after startup; before, the list stayed empty until Local got a line.
How many characters SMT has 5 s after startup, and how many of them it knows a system for. They come from the Local logs (or ESI), so a log follower that stops finding them shows up here.
The status bar shows the last message logged. When zKillboard dropped the connection, the feed logged the error, retried after 10 s and carried on, but logged nothing more. Since the performance lines no longer reach the status bar, the red error then stayed for hours while the feed worked: on 2026-11-09 two dropped connections at 19:25 and 19:37 still showed at 21:44. After an error the feed now logs "Reconnected to the live feed." once a poll succeeds again, which turns the status bar green and records in SMT.log how long the feed was down.
The system pickers (region map, universe map and the three in the route planners) are DotNetKit auto-complete boxes, which narrow their suggestions by setting a filter on their items. Controls that are given the same list share one collection view, so that filter applied to all of them: typing in one picker cleared the selection of another, which then removed the filter from under the box being typed in. The three route pickers have always shared a list. Since "Sort the system list once for all system pickers" the two map pickers shared it with them. EveManager.GetSystemsByName() still sorts once, and hands every caller its own copy of the sorted list.
LogFolderMonitor.Poll let an exception from one file end the poll, and the monitor loop logged it and tried again 250 ms later. A followed file that kept failing with anything but an IOException (access denied, say), or lines the handler threw on, meant the files after it in the folder were never read, the game logs weren't read at all when it was a chat log, and SMT.log got the stack trace four times a second. Poll now reads every file whatever happens to the others, and reports a failing file once, until it has worked again. The monitor loop reads the chat logs and the game logs in separate steps and logs a failure that keeps repeating once a minute at most. A scan that fails no longer runs again on the next poll, and a new file that can't be opened yet is tried again by the next scan. A log whose header couldn't be read was still marked as dealt with, so its character was never listed (Local) or its lines were dropped for the rest of the session (game log). Both are now tried again: a Local log on the next poll, a game log with its next lines.
Fields added in front of LoadFromDisk, Overlay.UpdateRouteData and Overlay.UpdateSystemTooltip had landed between each method and its summary, so the summaries described the fields.
Eleven comments described the code they replaced or quoted what it
measured ("a linear scan per jump made this 60 ms instead of ~4"). That
is in the commit messages; next to the code it describes something a
reader can't see, and the numbers go stale. Each now gives only the
reason the code is the way it is.
CleanLine compared against a U+FEFF typed into the source as the character itself, which editors and diffs show as an empty char literal. Same character, written as an escape.
The CI workflow already has a "dotnet test" step, and the solution had no tests for it to run. EVEData.Tests gives it twelve, each for a behaviour of the code this branch rewrites that a user would notice if it broke: - The log reader (LogFolderMonitor and LogFileTail), on log files written the way the EVE client writes them: held open and appended to, UTF-16 with a mark on every line for chat, UTF-8 for game logs. What a log held before SMT started is not reported again, half a line waits for its line break, a login while SMT runs is read from after the header (also when the file is caught still empty), years of old logs are not opened, a log that fails is reported once and does not stop the others, and the Local lines from before SMT started can be read back to place a character. The monitor takes the time from its caller, so the tests move the clock themselves and never wait. - The capital jump table, against every pair of systems in the shipped Systems.dat measured one by one. - The systems within a number of gate jumps (danger zones, storm areas), against a plain walk over the gates. Navigation.GetJumpableSystems is internal and exists for the jump table test. xunit is a test-only dependency.
PerfBench grew with the research. Most of its modes ran a copy of code
this branch replaces next to the new code, tried ideas that were not
adopted, or probed how NTFS reports the size of an open file. Once the
branch is merged, those have nothing left to compare. The checks among
them are tests now (EVEData.Tests).
What is left measures SMT as it is, and needs no recorded baseline:
PerfBench bench [seconds]
The startup data (the .dat files and the navigation tables), the
systems within 1 to 6 gate jumps, and the log reader on logs
written like the EVE client's: what a poll costs that finds
nothing, how long a line takes from being written to being
handled, and what looking for new files costs in a folder with
20,000 old logs.
PerfBench compare <before> <after>
Sets the results of "SMT.exe --perftest" from two builds side by
side: two result files, or two folders with several runs each.
Values that repeat exactly need no comment; values whose spread
over the runs doesn't overlap are flagged; timings from single
runs only count beyond the usual quarter.
With compare, --perftest answers "did this change make SMT slower, or
redraw more?" for any two builds, not only for this branch against
master. PerfBench is in SMT.sln, so CI builds it.
The perftest found RegionControl.UpdateSystemPopup, Overlay.RefreshCurrentView, Overlay.systemData and UniverseControl.FindSystemAt by name through reflection. Renaming one of them would only have shown the next time someone ran a perftest: the hover and universe lookup steps were skipped without a word when their method wasn't found, and the overlay step failed the run. They are internal now and called directly, so a rename is a compile error. What is still found by reflection belongs to WPF (a shape's handler list, a click count, a tooltip event). If the handler list can't be read the value is now left out of the results, where PerfBench compare shows it as missing, and no longer reported as zero handlers. The results also say whether the JIT's tiered compilation was on, since timings taken with it off don't compare with timings taken with it on, and building the results can no longer fail before the error handling starts.
SystemNamePixelHashes rendered every name block through a VisualBrush with a frame at the block's own origin. A VisualBrush draws a visual where it sits in its parent, so for every block not next to the canvas's top-left corner the frame showed nothing: in Delve, 106 of the 110 hashes were the hash of a blank image, and compared equal whatever the names looked like. The frame is now placed at the block's position on the canvas, and the results say how many of the renders show anything (systemNamesDrawn), so a capture that goes blank again shows as a changed count. "Draw region map system names as TextBlocks" said that all 110 names render on the same pixels as the Labels did, on the strength of those hashes. Checked again with the capture working, against a build of the commit before it: all 110 name renders are identical, and the two map images differ in one place, the marker of a character that the older build did not list yet.
Two --perftest runs of the same build, compared with one run a side, came out with 15 timings marked lower or higher. Startup phases of a few milliseconds, the first read of the data files, idle CPU and GC counts differ by a third to a half between runs, so the rule that single runs count beyond a quarter was too loose. Counts repeated exactly between the two runs (redraws per action, elements and visuals on the map, the name renders, handlers), so they are still compared from any number of runs. Timings, CPU, allocation and GC counts get a verdict only from four runs a side and are listed without one below that: with n runs a side, two builds that perform the same still show spreads that don't overlap 2 in C(2n, n) times, which is a third of the time for 2 runs, a tenth for 3 and 3% for 4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As mentioned on Discord, this is the whole performance work in one PR. As discussed on Discord, the whole performance work in one PR.
Release builds on a Ryzen 7 7800X3D, two runs each. Also: an intel line is handled a median of 140 ms after it is written instead of 816 ms, and idle CPU with jump bridges in view drops from 100–125 to 22–46 ms per second. The commit messages have the details.
It also fixes missed intel: on master, the next line of a channel that had been quiet for over 10 minutes was never read.
Behaviour changes
JumpRangeCache.dat(35 MB) is gone; the table is built at startup.Added
EVEData.Tests: 12 tests for the log reader and navigation, run by the existingdotnet teststep. xunit is a test-only dependency.SMT.exe --perftestandtools/PerfBenchto measure a build and compare two.Reviewing
The commits are small, and each message carries its own measurements. These change the most behaviour and are the ones to read closely:
JumpRangeCache.dat; capital routes are planned over the new table.PerfBench and the perftest mode were added early and trimmed at the end, so read those in the final diff.