Conversation
newren
force-pushed
the
en/pack-aggregate
branch
3 times, most recently
from
September 13, 2026 06:05
b1b0dcf to
d11f6d7
Compare
When considering delta compression, try_delta() skips pairs of
non-delta objects from the same pack, assuming an earlier delta search
already rejected them. Packs written by fast-import, bulk check-in,
or pack-objects --window=0 need not satisfy that assumption, as Jeff
King discussed:
https://lore.kernel.org/git/20231009202149.GA3281325@coredump.intra.peff.net/
Recognize a .baddeltas sidecar to disable this shortcut for marked
packs. Existing deltas remain reusable.
Assisted-by: Claude Opus 4.7
Signed-off-by: Elijah Newren <newren@gmail.com>
index-pack creates .keep and .promisor markers without overwriting existing files. Extract this writer into pack-write.c so other pack producers can reuse it, leaving filename selection and output reporting in index-pack. Signed-off-by: Elijah Newren <newren@gmail.com>
Packs written without a full delta search need a way to request reconsideration during later repacks. Add --mark-bad-deltas to write an empty .baddeltas marker beside each output pack. This option is incompatible with --stdout. Publish the marker before the index so readers discover the pack with its marker already in place. Assisted-by: Claude Opus 4.7 Signed-off-by: Elijah Newren <newren@gmail.com>
With --stdin-packs, the revision walk supplies namehash hints for
delta selection and, with --stdin-packs=follow, finds reachable
objects outside the listed packs.
Skip the walk when --stdin-packs=follow is not in use and delta search
is disabled by --window=0 or --depth=0.
Skipping the walk alone does not eliminate all the unnecessary work:
seeding via add_pending_oid() still parses commits and looks up
objects. These lookups can map every pack index in the object store,
which in some cases can even exhaust the per-process mapping limit.
Skip the commit seeding as well.
On a repository with about 35 million objects, packing 14,174 objects
from 316 packs in a store of 1,427 packs gave:
variant time max RSS
----------------------------- ------- --------
seed and walk (before) 176s 8.2 GiB
skip walk, still seed 0.38s 154 MiB
skip walk and seeding (after) 0.31s 66 MiB
In the worst case, skipping seeding also reduced distinct pack-index
mappings from 1,427 to 316. The output pack was byte-identical in all
three runs.
Assisted-by: Claude Opus 4.7
Signed-off-by: Elijah Newren <newren@gmail.com>
When combining packs with --stdin-packs --window=0, pack-objects reuses existing deltas without searching for new ones. However, if an object appears in multiple included packs, it uses the first copy it finds, even if that copy stores the object in full and another pack stores it as a delta. Add --prefer-reused-deltas to prefer a copy stored as a delta over one stored in full. It has no effect with --no-reuse-delta. On a roughly 37-million-object repository, first-copy selection lost 25,310 existing deltas while combining 7,253 input packs. Adding --prefer-reused-deltas retained 20,547 of them (81%). Assisted-by: Claude Opus 4.8 & GPT-5.6 Sol Signed-off-by: Elijah Newren <newren@gmail.com>
When combining packs, one pack can store A as a delta against B while another stores B as a delta against A. --prefer-reused-deltas can select both representations, creating a two-object delta cycle. break_delta_chains() repairs this by dropping one delta. However, the object whose delta is dropped then needs to be recompressed, rather than reusing a copy stored in full. Reject a candidate delta when its base is already selected as a delta against the same object. This preserves the selected full copy and avoids the recompression, while still retaining one delta for the pair. Limit this check to two-object cycles so each candidate requires only inspecting its base's selected representation, not walking a delta chain. Longer cycles remain handled by break_delta_chains(). This is a bounded optimization of delta selection, not a replacement for the general cycle-breaking pass. In a synthetic 1,000-pack workload with 50 reciprocal pairs of 1 MiB objects, this avoided 50 MiB of recompression and reduced median elapsed time from 1.86s to 0.44s, without changing the delta count or output size. A separate workload with 10,000 candidate deltas but no cycles showed no measurable slowdown. These timings used a warm disk cache. Assisted-by: Claude Opus 4.8 Signed-off-by: Elijah Newren <newren@gmail.com>
Repositories that have gone without maintenance can accumulate large
numbers of loose objects and small packfiles. Unrelated Git commands
then pay the cost of scanning the resulting sprawl in `objects/` and
`objects/pack/`, even when those commands would otherwise be quick.
Introduce a new plumbing builtin, `git pack-aggregate`, as a cheap
cleanup pass. With `--once`, it performs two steps:
1. Bundle local loose objects into new packs with `pack-objects
--window=0 --mark-bad-deltas --delta-base-offset
--no-write-bitmap-index`, then unlink the loose copies.
2. Aggregate local packs, excluding packs in the
multi-pack-index and packs carrying `.keep`, `.promisor`,
`.mtimes`, or `.bitmap` sidecars, then unlink the source packs.
Both steps skip delta search and reuse packed representations where
possible; loose objects still need compression. Either step can
produce multiple packs under `pack.packSizeLimit`. Each output carries
a `.baddeltas` marker so a later thorough repack knows to reconsider its
intra-pack deltas.
If packing loose objects produces just one pack, allow it to participate
in the subsequent pack aggregation. If it produces multiple packs,
likely due to `pack.packSizeLimit`, skip those new packs instead.
Make this a separate builtin rather than another `git repack` mode. Its
contract is deliberately narrow and cheap, unlike repack's range of
optimization modes. A later integration will run both processes at
once; distinct command names make their roles clear in process listings
and traces, while keeping the aggregator's looping and coordination
options out of repack's already broad option and configuration surface.
Use `pipe_command()` and run-command's child-cleanup machinery to avoid
I/O deadlocks and orphaned `pack-objects` processes.
Delete only the consumed pack's `.pack`, `.idx`, `.rev`, and
`.baddeltas` files after rechecking its protective sidecars.
`unlink_pack_path()` is not suitable here: its non-force mode protects
only `.keep` and otherwise also removes `.bitmap`, `.promisor`, and
`.mtimes` sidecars that cause aggregation to leave a pack alone.
Like `git repack`, `git pack-aggregate` takes no locks of its own. Do
not run it concurrently with other pack-rewriting maintenance.
Assisted-by: Claude Opus 4.7 & GPT-5.6 Sol
Signed-off-by: Elijah Newren <newren@gmail.com>
A .baddeltas pack can sit above the geometric split indefinitely, so pack-objects never gets a chance to reconsider its delta layout. Include marked packs in the rollup even when their size would otherwise keep them out. If their inclusion breaks the estimated progression, include one more eligible pack, without consuming smaller candidates unnecessarily. Keep the retained region sorted for `pack_geometry_preferred_pack()`. `.keep`-marked packs are still excluded from the rollup by the existing `pack_kept_objects` / `remove_redundant_packs` paths, so a user who really wants to pin a `.baddeltas` pack in place has that escape hatch. Promisor packs are intentionally not touched: the aggregator skips them, and rolling a promisor pack up into a normal pack would silently strip its "objects may live remotely" semantics. Add tests for forced demotion, geometric balancing, and preservation of unaffected packs. Assisted-by: Claude Opus 4.7 & GPT-5.6 Sol Signed-off-by: Elijah Newren <newren@gmail.com>
`git pack-aggregate` can select packs with millions of objects outside the multi-pack-index, even in otherwise well-maintained repositories. Enumerating those objects and rebuilding their indexes costs work for little reduction in pack count. Add a `--max-objects=<n>` cap (and matching `pack.aggregateMaxObjects` config, default 100000, 0 to disable) to bound that per-object work. Leave packs above the cap untouched. Estimate the object count from the size of the pack's `.idx` file, reusing the `lstat()` already needed to confirm it exists. This avoids opening and mapping every candidate index just to read its object count, saving work and avoiding exhaustion of `vm.max_map_count` when there are many packs. Assisted-by: Claude Opus 4.8 & GPT-5.6 Sol Signed-off-by: Elijah Newren <newren@gmail.com>
With pack.packSizeLimit=1m, aggregating 700-KiB single-blob packs just rewrites them without reducing their number. Skip packs larger than half the effective output limit, using their already-known sizes. Add --max-input-pack-size and pack.aggregateMaxInputPackSize to allow a lower threshold, or an independent byte cap when output size is unlimited. Assisted-by: GPT-6 Astra Signed-off-by: Elijah Newren <newren@gmail.com>
pack-aggregate currently feeds every eligible loose object to a single
pack-objects process. That is acceptable for the normal case, but an
extremely neglected repository can have millions of loose objects.
Such a repository remains burdened by the large number of loose-object
files until that invocation finishes, and pack-objects must hold
bookkeeping for the entire backlog at once.
Add --max-loose-objects=<n> and pack.aggregateMaxLooseObjects to limit
how many loose objects each pack-objects invocation processes. Each
batch can produce multiple packs under pack.packSizeLimit. Install all
its output packs before unlinking its loose copies and starting the
next batch, allowing the repository to improve incrementally and
preserving completed work if the process is interrupted.
Create a temporary marker in the object directory and use its filesystem
mtime as the cycle cutoff. Rescans ignore objects that are not older
than that marker, so concurrent writers cannot keep a cycle running
indefinitely. Using a marker on the same filesystem also avoids
comparing file mtimes against a potentially different host clock.
When loose-object rollup produces multiple packs, leave all of them
out of the pack-aggregation phase of the same cycle. This avoids
immediately copying the entire recovered loose-object backlog a second
time. A single pack produced from loose objects retains the existing
behavior and can be combined with other small packs immediately.
Default the limit to 100000. On a synthetic repository containing ten
million small loose blobs, the following limits produced:
limit first pack total time peak RSS
100000 12.4s 1520.0s 26MB
500000 64.1s 1530.8s 115MB
1000000 132.6s 1537.0s 226MB
The total time was effectively unchanged, while 100000 made the first
improvement visible much sooner and used substantially less memory. A
value of 0 processes all eligible loose objects in one batch.
Assisted-by: GPT-5.6 Sol
Signed-off-by: Elijah Newren <newren@gmail.com>
pack-aggregate feeds all aggregatable packs to a single pack-objects invocation. On a normally-maintained repository that is fine, but repositories can accumulate an extreme number of packs -- tens or even hundreds of thousands -- before maintenance catches up. Processing that many packs at once increases the memory needed for pack data and indexes and can exhaust the per-process limit on memory mappings. Add a --max-packs=<n> option (and a pack.aggregateMaxPacks config) that caps the number of input packs per pack-objects invocation. When more than <n> candidates are present, split them into approximately balanced batches by pack count, each handled by a separate invocation. The final batch may be smaller. For example, 12000 packs with --max-packs=5000 are processed as three batches of 4000 -- not 5000, 5000, and 2000. Split on whole-pack boundaries. Because on-disk packs are always self-contained (git never stores thin packs), a delta and its base stay together in the same input batch. Each invocation can reuse existing deltas where possible, though pack.packSizeLimit can split its output into multiple packs and require expanding deltas whose bases land in another output pack. Default --max-packs to 10000; use a value of 0 to process all aggregatable packs in one batch. Assisted-by: Claude Opus 4.8 & GPT-5.6 Sol Signed-off-by: Elijah Newren <newren@gmail.com>
Repositories that have gone without maintenance can accumulate enough loose objects and small packs to slow down every Git command that needs to inspect the object store. A thorough repack will eventually repair that state, but it must first operate while the repository is still at its least healthy. Teach `git repack` to optionally run `git pack-aggregate --once` before it inspects the existing packs and loose objects. This cheap preliminary pass quickly reduces the number of files the main repack must contend with, leaving the thorough repack to optimize deltas and pack layout. Add `--aggregate-once` and `--no-aggregate-once`, together with the matching `repack.aggregateOnce` configuration variable. Keep the behavior disabled by default. Run the aggregation pass before repack snapshots refs for a MIDX bitmap, reads the existing MIDX, or computes geometric pack selection. If the aggregation fails, abort before starting the main pack-objects process. Since aggregation mutates the repository, validate incompatible `--geometric` and `-a`/`-A` options, and `--filter-to` without `--filter`, before invoking it. Invalid commands therefore remain side-effect free. Like `git repack` and `git pack-aggregate` themselves, this integration takes no locks. Serialization with other pack-rewriting maintenance remains the caller's responsibility. Signed-off-by: Elijah Newren <newren@gmail.com>
Let pack-objects publish the local packs and loose objects visible before object enumeration, using `--emit-input-packs` and `--emit-input-loose`. A concurrent maintenance process can exclude these inputs without racing to enumerate them before pack-objects. Include all local packs, not just those selected for rewriting. A geometric repack may include retained packs in its replacement MIDX; a concurrent aggregator must leave them alone too, or the MIDX could reference a pack it retired. The loose-object snapshot reduces duplicate packing: deleting a loose copy does not prevent pack-objects from writing another packed copy. Write each file through `<file>.tmp` and rename it into place, so readers never see a partial list. These options support the planned background repack integration and remain undocumented. Assisted-by: Claude Opus 4.7 & GPT-5.6 Sol Signed-off-by: Elijah Newren <newren@gmail.com>
The `--once` mode provides a quick manual cleanup pass, but a long-running repack leaves time for new loose objects and small push packs to accumulate while it works. Repeated aggregation can keep that new material under control without interfering with the larger repack, provided the two processes operate on disjoint inputs. Add a `--loop` mode that runs aggregation passes repeatedly, sleeping `--interval` seconds between them. Add exclusion files for packs and loose objects so a concurrent `pack-objects` can declare the inputs it owns; the aggregator skips those inputs on every pass. Add `--parent-pipe-fd` so a parent process can keep the aggregator alive by holding a pipe open. Poll the pipe during the interval so the aggregator exits promptly when its parent goes away. Handle termination signals by recording a stop request so the current cycle can unwind and the loop can exit at a safe boundary. An in-flight `pack-objects` is already marked `clean_on_exit`, so run-command's standard signal cleanup forwards the signal to it before chaining to the loop's handler. This stops the child promptly while allowing the loop to unwind in an orderly way. Assisted-by: Claude Opus 4.7/4.8 & GPT-5.6 Sol Signed-off-by: Elijah Newren <newren@gmail.com>
Both `git repack` and `git pack-aggregate` stage a new pack as ".tmp-<pid>-pack-<hash>.*" and rename it to "pack-<hash>.*" only once it is fully written (the .idx last). Before running these commands concurrently, prevent them from selecting each other's staging files. The object-store scan treats any "*.idx" as a pack regardless of prefix, while exclusion lists and ".keep" markers use the final "pack-<hash>" name. Without a name check, pack-aggregate can delete a temporary pack owned by another process, or geometric repack can feed it to pack-objects just before its owner renames it. Require a canonical "pack-<hash>.pack" basename where packs are selected, using a shared helper in collect_pack_candidates() and the geometry rollup. Keep the object-store scan unchanged: `git repack -ad` legitimately enumerates stale ".tmp-*" packs in order to delete them. Signed-off-by: Elijah Newren <newren@gmail.com>
On a busy server hosting a large repository, `git repack` routinely takes long enough for thousands of packs and tens of thousands of loose objects to accumulate before it finishes. Every subsequent Git operation pays the price: each pack lookup walks the longer list, and each loose-object miss has to consult more sources. Wire `git repack` to optionally drive `git pack-aggregate` as a background subprocess, controlled by `repack.aggregateLoop` or the equivalent `--aggregate-loop` and `--no-aggregate-loop` command-line options. Keep the behavior disabled by default and independent of the one-shot aggregation pass. Have the main pack-objects write its input snapshots to a temporary directory under `objects/`. Once both files exist, launch aggregation with those exclusion lists. Protect new repack outputs with temporary `.keep` files before their indexes become visible. The aggregator already skips packs in the current MIDX. The exclusions and temporary markers also protect every pack in the replacement MIDX's explicit include list, so aggregation cannot retire a pack the repack is about to reference. Keep the aggregator running through cruft packing, pack installation, and the MIDX write. Stop it before redundant-pack cleanup, then remove only the `.keep` files this repack created. The aggregator detects an unexpected repack exit through a parent pipe. Mark unrelated pipe ends close-on-exec so inherited writers cannot suppress EOF and leave either child waiting. `git repack` itself takes no locks, and neither does the aggregator we spawn; serialization across maintenance runs is the caller's responsibility (typically `git gc` or a server-side maintenance driver). This matches the prior behavior of `git repack`. Assisted-by: Claude Opus 4.7 & GPT-5.6 Sol Signed-off-by: Elijah Newren <newren@gmail.com>
A pack-aggregate cycle that finds nothing to do leaves no on-disk artifact, so process lifetime and repository state are insufficient to count cleanup cycles. Add trace2 regions around each pack-aggregate cycle and repack's redundant-pack removal so trace consumers can account for these operations, including no-op aggregation cycles. Signed-off-by: Elijah Newren <newren@gmail.com>
newren
force-pushed
the
en/pack-aggregate
branch
from
September 23, 2026 18:01
d11f6d7 to
1d72451
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[This should be split into about 3 series before sending to the mailing list...]
Busy or neglected repositories can accumulate enough loose objects and
small push packs to slow every operation that scans the object store. A
full repack eventually repairs that state, but it must operate while the
repository is at its least healthy and can run long enough for more
loose objects and packs to accumulate behind it.
This series introduces the git pack-aggregate plumbing command. It
quickly bundles loose objects and rolls up small packs by copying their
existing representations, without delta search or recompression. Its
output is marked with a new .baddeltas sidecar so a later thorough
repack knows to reconsider the inherited delta layout.
Aggregation can run once as a preliminary cleanup or loop alongside a
long-running repack. The concurrent mode coordinates the two processes
with pack/loose-object exclusion snapshots and temporary .keep
markers, preventing either process from consuming inputs owned by the
other. Limits on loose objects, input packs, and pack object counts keep
individual aggregation steps bounded and allow severely degraded
repositories to improve incrementally.
The series also teaches geometric repacks to roll up .baddeltas
outputs, adds pack-objects plumbing to preserve reusable deltas and
publish input snapshots, excludes protected and in-flight packs, and
adds trace2 regions for cleanup cycles. All repack integration is
opt-in.