Repository navigation
perf(sync): mirror skills concurrently and keep the offline modes offline - #77
Merged
Merged
Conversation
kieronlanning
enabled auto-merge (squash)
November 16, 2026 13:24
…line Follow-up to the concurrent documentation fetch: syncSkills fetched one file at a time, per skill, for 108 skills (~160 requests). It now plans the work first - which ref, cached or not, which files - fetches the flattened list eight at a time, and only then writes, marks and reports, in plan order, so the mirror and the log are identical to the serial result. cache and fixture promised a sync that touches no network but listed every repository tree anyway before requiring the skills cache. They now take the skill set from the cache (cachedSkills): the per-skill .ref markers say what was fetched and at which ref, and a per-project marker records the ref the project was last synced for. That marker is what tells an empty skill set apart from one that was never fetched, so an offline sync neither makes a request nor goes quiet when the cache is cold. auto no longer lists a tree for a project whose declared skills are all cached and that asks for no discovery - there is nothing to look up. listRepositoryFiles is memoized per repository and ref for the process, so a sync stops listing the same tree twice (once for the documentation channels, once for the skills mirror). Measured on one machine: a live sync goes from 140.0s to 74.6s (210.6s before the first concurrent-fetch change), an auto sync with a warm cache to 7.1s, and a cache-mode sync to 1s. Unit tests cover the cached skill set (ref filtering, stray files, an absent cache directory) and the project ref marker.
kieronlanning
force-pushed
the
chore/sync-followups
branch
from
November 16, 2026 13:27
1349a3e to
ce36d69
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Four follow-ups to #76, all the same shape: work that was serial, repeated, or unnecessary.
What changed
1. The skills mirror is fetched concurrently.
syncSkillsfetched one file at a time, per skill - 108 skills, ~160 requests, each waiting for the last. It now plans first (which ref, cached or not, which files), fetches the flattened list eight at a time, then writes, marks and reports in plan order, so the mirrored tree and the log are identical to the serial result.2.
cacheandfixtureare offline again. They promise a sync that touches no network (data-sync.ts), but the skills mirror listed every repository tree before requiring the cache - so the documented offline loop (DATA_MODE=fixture just build) made ~13 REST calls. Those modes now take the skill set from the cache (cachedSkills): the.refmarker beside each cached skill names the ref it was fetched from, and a declared skill that was never fetched still fails loudly instead of vanishing.3.
autostops looking up what it cannot need. A project whose declared skills are all cached and that asks for no discovery (skills:list rather thanskills: true) no longer lists its tree: nothing to fetch, nothing to discover.4. One tree listing per repository, not two.
listRepositoryFilesis memoized perrepository@reffor the process, because a sync lists most trees twice - once for the documentation channels, once for the skills mirror. A failed listing is not cached, so a transient error is retried rather than remembered.The 3rd item on my follow-up list is a documentation change, not a code change. The PR gate and the
maindeploy run the same chain deliberately;skills/site-validation-loopnow says so, and says that removing the second run is a decision about what the deployment no longer has to prove. For the record,release.ymlis already path-filtered topackage.json, somaindoes not run a second full pipeline on a content push.Evidence
A/B of
DATA_MODE=live bun run data:sync, same machine, same work, same 108 skills:-65% overall, -47% on top of #76.
Verified: 452 unit tests pass (2 new for the cached skill set),
astro check0 errors / 0 warnings, oxlint and oxfmt clean, andjust validategreen in 44s - which includescheck-generatedandtest-distagainst the skills mirror and documentation this code produced.Not done, and why
main). Sharing one run between them would mean publishing on the strength of a PR-time validation, on a different runner, against different live data. That is a policy decision, not a refactor, so it is written down rather than assumed.skills: truemeans "publish every skill underskills/", so inauto/livethe repository tree is the source of truth for which skills exist and must stay listed. Only the cache-only modes can answer without it.