Repository navigation
perf: batch source hashing so close takes seconds, not minutes (v0.20.0) - #15
Merged
Merged
Conversation
The source snapshot started one to three processes per file, and reading a delivered commit started two or three more per file, one of them git. A shipped close with a verification recipe also walked the source twice. On Git Bash, where starting a process is slow, closing a repo with a few thousand files took minutes. The worktree files are now hashed with one xargs-driven sha256sum. The commit is checked out once into a temp dir through a temp index, which runs the same filters, and is hashed the same way. close.sh reuses the digest the source binding has just taken. Either batch falls back to the per-file read on any failure. The verdicts do not change: the digests match v0.19.0 exactly on three real repos. Selftest section 10 checks that the worktree, one-checkout and blob-by-blob reads agree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.20.0 — batched source hashing: close in seconds, not minutes
close.sh <slug> shippedcould take minutes on Windows (Git Bash) in a repowith a few thousand files. The source snapshot started a process or three per
file, the commit read (a delivered
Sourcesha) started two or three more perfile, one of them
git, and a shipped close with a verification recipe walkedthe whole source twice. Git Bash starts processes slowly, so the cost was
mostly process startup. The verdicts are the same. Only the number of
processes changes.
Changes
sdlc_sha256_paths: hashes many files with onesha256sum(orshasum) run throughxargs -0, one digest per path inorder. The openssl fallback still hashes one file at a time.
sdlc_source_entries): files are classified asbefore, then hashed in one batch. If the batch fails (an unreadable file, or
a file gone since it was listed), the files are hashed one at a time, so the
FAIL:line still names the file.sdlc_tree_entries): the commit is checked out once into atemp dir through a temp index, so the repository's index and worktree are
untouched.
checkout-indexwrites the bytes a checkout would, through thesame filters, and one batch hashes them. Symlinks, submodules and quoted
names still go blob by blob. On any failure (names that clash when case is
ignored, a name the filesystem refuses), the whole commit is read blob by
blob as before.
tools/handoff.shuses this read too.ok, the digestit just computed is reused by the verification check (
SDLC_SOURCE_DIGEST_NOW).The variable is cleared each time
_common.shloads, so a value from theenvironment is never trusted.
Measured on macOS (bash 3.2) with the same digests before and after:
Most of the remaining commit-read time comes from endpoint antivirus scanning
the freshly written temp files on first read. A second read of the same files
takes 0.5 s. A shipped close over a commit
Sourcein the 3,700-file repo wentfrom about 130 s to about 12 s. The approve, status, verify and handoff
commands share these helpers, so they get faster too.
Validation
bash gates/selftest.sh→SELFTEST PASSon macOS; CI runs ubuntu, macosand windows. The new section 10 checks that the worktree snapshot, the
one-checkout commit read, and the blob-by-blob read of the same commit give
identical entries. The fixture has spaces, a leading
-, a Hangul name, anexecutable, a symlink, an
eol=crlffilter and.sdlc/. It also checks theone-at-a-time fallback, a case clash that has to take the blob read, and
that an inherited
SDLC_SOURCE_DIGEST_NOWis ignored.byte-identical to v0.19.0 for both the worktree and the
HEADreads.