Repository navigation
Refresh the saved sources weekly, downloading only what changed - #3
Merged
Merged
Conversation
Cities, regions, countries and UNESCO sites are now built from saved copies in data/reference/ that npm run refresh keeps current: one conditional request per source (GeoNames once a month), a download only when the source changed, and a new saved copy only when the part used here changed. A source that fails or shrinks by more than 5% keeps its saved copy. Ids never change: dropped cities are kept and UNESCO sites keep their ids by site number. The built files are unchanged. all-the-cities and world-countries are no longer needed, nor the city-admin1 script.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before: cities, regions, countries and UNESCO sites came from copies frozen on the day they were vendored (npm packages and gzipped files), and only a manual run could update them.
After: a weekly workflow (Sunday, the day before the build) looks for a newer version of each source and commits a new saved copy only when the part used here changed. The built files are unchanged by this PR.
How it stays polite and safe:
data/reference/sources.lock.jsonrecords what was seen.npm run checkand a build) before it is committed; a commit from the workflow doesn't start a build, the Monday one picks it up.data/reference/heritage-ids.jsonby site number, even when renamed.Also: all-the-cities, world-countries and the city-admin1 script are gone. Cities now come from GeoNames' cities1000 directly, so the first refresh will add the cities listed since the old package's snapshot. The first countries refresh will rename CG "Congo" in
names.json(mledoze's current name);article-names.jsonkeeps "Republic of the Congo", the article title.Note: the workflow pushes to main with the built-in token, so Actions needs write permission in the repository settings if it is restricted.