Parallelise in chunks for extent calc and vertex counting + cache PROJ transform per GLB export worker#89
Merged
Merged
Conversation
Ylannl
changed the title
Ylannl
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extent calculation now uses cjindex ref pages, splits each page into 2_048-feature chunks, and processes those chunks with Rayon using thread-local CityIndex readers. Grid indexing now does the same: it no longer serially decodes a full page before parallel processing; decoding plus selected_geometry_stats plus vertex counting now happen per parallel chunk.
The final grid integration is still serial because it assigns stable feature ids and mutates the dense grid. That part should be much smaller than decoding/stat/ counting, and keeping it serial preserves output order.
This gives me a 30% speedup with 8 threads for the ams-up data.
Next was an issue with PROJ that was reinitialising the CRS transform for the GLB writer for every tile. This was fixed by doing this once per worker and then caching and reusing it.
That gave another speedup of ~60-70%.
A bit more details in ADR 009. Overal this brought down the processing time from 1m59 to 27s.