connected: add incremental connectivity check via rev-list

The connectivity check uses rev-list to find commits reachable
from the incoming tips but not from any local ref and then walks
their object closure.  Commit traversal stops at the connectivity
boundary, but trees and blobs reachable from that boundary still
need to be walked so they can be marked uninteresting, allocating
a struct object for each one.  On repositories where the boundary
commits have large trees, the connectivity check for small
incoming changes visits and tracks more objects than needed.

Add an alternative connectivity check that verifies incoming
commits incrementally against their parents.

Instead of traversing the full boundary closure, the new check
compares each new commit's tree with its parent trees.  Already
trusted entries are skipped, changed subtrees are descended into
recursively, and blobs are checked for existence.  This approach
thus avoids descending into untouched subtrees.

For example, consider a commit that changes one file under lib/
and also moves an unchanged subtree from src/ to dev/:

    Parent tree              New tree
    +-- src/   (aaa)         +-- dev/   (aaa)
    +-- lib/   (bbb)         +-- lib/   (ccc)
         +-- foo.c (ddd)          +-- foo.c (ddd)
         +-- bar.c (eee)          +-- bar.c (fff)

The verifier first scans the new root and collects aaa and ccc as
work items.  It then scans the parent root, publishing aaa and bbb
into the trusted sets and recording bbb as the comparison base for
ccc.

When the work list is revisited, aaa is now trusted and skipped
even though it appears at a different path.  The verifier descends
into ccc using bbb as its parent base.  Scanning bbb similarly
makes ddd and eee trusted, leaving only the new fff blob to be
checked for existence.

Thus neither the moved subtree nor any other unchanged subtree is
recursively explored; only the changed lib/ subtree is descended
into, and only the new bar.c blob needs an existence check.  The
root trees still need to be read and scanned as comparison bases.

New commits are processed with ancestors before descendants.
Parents outside the incoming commit set are on the already-connected
side of the boundary and provide the initial trusted bases.  Once
an incoming commit has been verified, its tree can in turn be used
as a trusted base for descendant commits.

The verifier distinguishes trusted trees from expanded trees.  A
trusted tree can be accepted without further verification.  An
expanded tree has additionally published its direct non-gitlink
entries into the trusted sets.  Expanded parent trees therefore
need not be read again for blob-only work, but may still be reread
when recursive verification needs same-path parent subtrees.

The implementation adds a --verify-trees-incremental flag to
rev-list, following the same pattern as --exclude-promisor-objects:
a pre-setup_revisions() scan sets the flag, the
post-setup_revisions() option loop skips it, and the main
traversal short-circuits into the incremental verifier after
collecting commits from the revision walk.

Because the incremental block eagerly consumes all commits via
get_revision(), the subsequent mark_edges_uninteresting() and
traverse_commit_list_filtered() calls naturally find no commit
work to do and only process any non-commit tips (trees, blobs)
left in revs.pending.

For partial clones, missing promisor objects (trees and blobs
promised by a promisor remote) are silently accepted during
verification instead of triggering errors.  The revision walk
receives --exclude-promisor-objects so promisor commits do not
enter the verification set.

Gate the new algorithm behind transfer.connectivityCheck=incremental,
keeping full as the default.  Fall back to the full algorithm for
deepening fetches to keep this commit easy to reason about, though
incremental mode could potentially handle them as well.

p5412 results (median of 3), scaling one dimension at a time.
Each modified file is in a different directory, with
directories chosen round-robin.

Scaling tree size (10 commits, 10 files/commit):

    files    full  incr.  full/incr
      5K    0.01s  0.01s    1.1x
     50K    0.04s  0.01s    2.2x
    200K    0.14s  0.02s    5.7x
    800K    0.60s  0.04s   14.8x

The full mode must traverse the boundary tree closure, which
grows with overall tree size.  Incremental still scans the
root trees, but avoids descending into unchanged subtrees,
so it grows much more slowly with repository size.

Scaling commit count (200K files, 10 files/commit):

    commits    full  incr.  full/incr
          1    0.14s  0.01s    7.5x
         10    0.15s  0.02s    6.8x
        100    0.16s  0.07s    2.2x
        500    0.29s  0.29s    0.9x
       3000    1.26s  1.67s    0.7x
       5000    1.95s  2.71s    0.7x
      10000    3.74s  6.02s    0.6x

With many commits the per-commit overhead of scanning both
the new and parent root trees accumulates and incremental
becomes slower.  Breakeven is around 500 commits and the
ratio stabilizes near 0.6x for this fixture.

Scaling files per commit (200K files, 10 commits):

    files/commit    full  incr.  full/incr
               1    0.15s  0.02s    7.5x
              10    0.15s  0.02s    6.8x
             100    0.15s  0.04s    3.1x
             500    0.26s  0.16s    1.6x
            1000    0.36s  0.35s    1.0x
            2000    0.66s  0.79s    0.8x

Breakeven is around 1000 files/commit.  At 2000 files/commit
(every directory touched), incremental is about 1.2x slower.

For small repositories both modes are fast enough that the
difference is difficult to measure reliably.

Signed-off-by: Kristofer Karlsson <krka@spotify.com>

Author: spkrka

Documentation/config/transfer.adoc

@@ -1,3 +1,23 @@
+transfer.connectivityCheck::
+	Choose which algorithm to use for the connectivity check
+	performed during object transfer operations such as
+	linkgit:git-fetch[1] and linkgit:git-receive-pack[1].
+	The connectivity check verifies that all objects reachable
+	from the incoming tips are available locally or, in a partial
+	clone, promised by a promisor remote.
+	The variants are as follows:
++
+--
+`full` (default);;
+	Walk the full object closure of the boundary commits.
+`incremental`;;
+	Verify incoming commits by diffing their trees against parent
+	trees, recursively descending only into entries that differ.
+	The largest benefits occur when incoming commits change a
+	small fraction of a large tree closure.
+	Falls back to `full` for deepening fetches.
+--
+
 transfer.credentialsInUrl::
 	A configured URL can contain plaintext credentials in the form
 	`<protocol>://<user>:<password>@<domain>/<path>`. You may want

Documentation/rev-list-options.adoc

@@ -1089,6 +1089,12 @@ we cannot get their Object ID though, an error will be raised.
 	stronger than `--missing=allow-promisor` because it limits the
 	traversal, rather than just silencing errors about missing
 	objects.
+
+`--verify-trees-incremental`::
+	(For internal use only.)  Verify tree connectivity
+	incrementally by comparing each commit's tree against its
+	parent trees.  Used by `check_connected()` when
+	`transfer.connectivityCheck` is set to `incremental`.
 endif::git-rev-list[]
 
 `--no-walk[=(sorted|unsorted)]`::

Makefile

@@ -1357,6 +1357,7 @@ LIB_OBJS += trailer.o
 LIB_OBJS += transport-helper.o
 LIB_OBJS += transport.o
 LIB_OBJS += tree-diff.o
+LIB_OBJS += tree-verify.o
 LIB_OBJS += tree-walk.o
 LIB_OBJS += tree.o
 LIB_OBJS += unpack-trees.o

builtin/rev-list.c

@@ -28,6 +28,7 @@
 #include "commit-reach.h"
 #include "quote.h"
 #include "strbuf.h"
+#include "tree-verify.h"
 
 struct rev_list_info {
 	struct rev_info *revs;
@@ -706,6 +707,7 @@ int cmd_rev_list(int argc,
 	int bisect_find_all = 0;
 	int use_bitmap_index = 0;
 	int filter_provided_objects = 0;
+	int verify_trees_incremental = 0;
 	const char *show_progress = NULL;
 	int ret = 0;
 
@@ -748,6 +750,8 @@ int cmd_rev_list(int argc,
 		if (!strcmp(arg, "--exclude-promisor-objects")) {
 			repo->fetch_if_missing = 0;
 			revs.exclude_promisor_objects = 1;
+		} else if (!strcmp(arg, "--verify-trees-incremental")) {
+			verify_trees_incremental = 1;
 		} else if (skip_prefix(arg, "--missing=", &arg)) {
 			parse_missing_action_value(repo, arg);
 		} else if (!strcmp(arg, "-z")) {
@@ -822,6 +826,8 @@ int cmd_rev_list(int argc,
 
 		if (!strcmp(arg, "--exclude-promisor-objects"))
 			continue; /* already handled above */
+		if (!strcmp(arg, "--verify-trees-incremental"))
+			continue; /* already handled above */
 		if (skip_prefix(arg, "--missing=", &arg))
 			continue; /* already handled above */
 
@@ -935,6 +941,18 @@ int cmd_rev_list(int argc,
 
 	prepare_maximal_independent(&revs);
 
+	if (verify_trees_incremental) {
+		struct commit *commit;
+		struct commit_list *new_commits = NULL;
+
+		while ((commit = get_revision(&revs)) != NULL)
+			commit_list_insert(commit, &new_commits);
+
+		verify_commits_incremental(repo, &new_commits,
+					   revs.exclude_promisor_objects);
+		commit_list_free(new_commits);
+	}
+
 	if (revs.tree_objects)
 		mark_edges_uninteresting(&revs, show_edge, 0);
 

connected.c

@@ -1,6 +1,7 @@
 #define USE_THE_REPOSITORY_VARIABLE
 
 #include "git-compat-util.h"
+#include "config.h"
 #include "gettext.h"
 #include "hex.h"
 #include "odb.h"
@@ -67,6 +68,26 @@ static int check_connected_promisor(oid_iterate_fn fn,
 	return 1;
 }
 
+static int incremental_check_applicable(struct check_connected_options *opt)
+{
+	const char *algorithm = NULL;
+
+	if (repo_config_get_string_tmp(the_repository,
+				       "transfer.connectivitycheck",
+				       &algorithm))
+		return 0;
+	if (!strcasecmp(algorithm, "full"))
+		return 0;
+	if (strcasecmp(algorithm, "incremental"))
+		die(_("unknown transfer.connectivityCheck algorithm '%s'"),
+		    algorithm);
+
+	if (opt->is_deepening_fetch)
+		return 0;
+
+	return 1;
+}
+
 /*
  * If we feed all the commits we want to verify to this command
  *
@@ -133,6 +154,9 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
 	if (opt->progress)
 		strvec_pushf(&rev_list.args, "--progress=%s",
 			     _("Checking connectivity"));
+	if (incremental_check_applicable(opt))
+		strvec_push(&rev_list.args,
+			    "--verify-trees-incremental");
 
 	rev_list.git_cmd = 1;
 	if (opt->env)

meson.build

@@ -562,6 +562,7 @@ libgit_sources = [
   'transport-helper.c',
   'transport.c',
   'tree-diff.c',
+  'tree-verify.c',
   'tree-walk.c',
   'tree.c',
   'unpack-trees.c',

t/meson.build

@@ -652,6 +652,7 @@ integration_tests = [
   't5409-colorize-remote-messages.sh',
   't5410-receive-pack.sh',
   't5411-proc-receive-hook.sh',
+  't5412-connectivity-check.sh',
   't5500-fetch-pack.sh',
   't5501-fetch-push-alternates.sh',
   't5502-quickfetch.sh',

t/perf/generate-repo-p5412-connectivity-check.perl

@@ -0,0 +1,44 @@
+#!/usr/bin/perl
+#
+# Generate a fast-import stream for p5412 connectivity check benchmarks.
+#
+# Usage: generate-repo-p5412-connectivity-check.perl
+#            <dirs> <files_per_dir> <commits> [<hot_dirs>] [<files_per_commit>]
+#
+# Creates one initial commit with dirs*files_per_dir files, then
+# <commits> additional commits each modifying <files_per_commit>
+# files in directories chosen round-robin from 1..<hot_dirs>.
+
+use strict;
+use warnings;
+
+my ($nd, $nf, $nc, $hot, $fpc) = @ARGV;
+$hot = $nd if !$hot || $hot > $nd;
+$fpc = 1   if !$fpc;
+
+sub data {
+	printf "data %d\n%s\n", length($_[0]), $_[0];
+}
+
+# Initial tree: one commit with nd*nf files.
+printf "commit refs/heads/main\n";
+printf "committer perf <perf\@test.com> now\n";
+data("initial");
+for my $d (1..$nd) {
+	for my $f (1..$nf) {
+		printf "M 100644 inline d-%04d/f-%03d\n", $d, $f;
+		data(sprintf "%03d%03d", $d, $f);
+	}
+}
+
+# Subsequent commits (auto-chained by fast-import).
+for my $i (1..$nc) {
+	printf "commit refs/heads/main\n";
+	printf "committer perf <perf\@test.com> now\n";
+	data(sprintf "change-%03d", $i);
+	for my $j (0..$fpc-1) {
+		my $d = (($i + $j) % $hot) + 1;
+		printf "M 100644 inline d-%04d/f-001\n", $d;
+		data(sprintf "c%d-%d", $i, $j);
+	}
+}

t/perf/p5412-connectivity-check.sh

@@ -0,0 +1,87 @@
+#!/bin/sh
+
+test_description='performance of connectivity check modes
+
+Compare the default and incremental rev-list connectivity modes
+directly, avoiding pack transfer noise.
+
+Each repository has a flat tree of many directories with 100 files
+in each.  Three axes are scaled independently: tree size, commit
+count, and files changed per commit.'
+
+. ./perf-lib.sh
+
+test_perf_fresh_repo
+
+generate="$TEST_DIRECTORY/perf/generate-repo-p5412-connectivity-check.perl"
+
+# $1=dirs  $2=files_per_dir  $3=commits  $4=hot_dirs (optional, default=all)
+# $5=files_per_commit (optional, default=1)
+test_perf_conn () {
+	local nd="$1" nf="$2" nc="$3" hot="${4:-$1}" fpc="${5:-1}"
+	local total=$(($nd * $nf))
+	local name="repo-${nd}d-${nf}f-${nc}c-${hot}h-${fpc}fpc"
+	local label="${total} files, ${nc} commits"
+	if test "$hot" -lt "$nd"
+	then
+		label="$label (${hot} hot dirs)"
+	fi
+	if test "$fpc" -gt 1
+	then
+		label="$label (${fpc} files/commit)"
+	fi
+
+	test_expect_success "setup $label" '
+		git init '"$name"' &&
+		"$PERL_PATH" '"$generate"' '"$nd"' '"$nf"' '"$nc"' '"$hot"' '"$fpc"' |
+		git -C '"$name"' fast-import --date-format=now --quiet &&
+		(
+			cd '"$name"' &&
+			git rev-parse main~'"$nc"' >../'"${name}"'_old &&
+			git rev-parse main >../'"${name}"'_new &&
+			git update-ref refs/heads/main \
+				$(cat ../'"${name}"'_old) &&
+			git repack -ad &&
+			git config gc.auto 0
+		)
+	'
+
+	test_perf "$label (full)" '
+		cat '"${name}"'_new |
+		git -C '"$name"' rev-list \
+			--objects --stdin --not --all --quiet \
+			--exclude-promisor-objects
+	'
+
+	test_perf "$label (incremental)" '
+		cat '"${name}"'_new |
+		git -C '"$name"' rev-list --verify-trees-incremental \
+			--objects --stdin --not --all --quiet \
+			--exclude-promisor-objects
+	'
+}
+
+# Scaling tree size (10 commits, 10 files/commit).
+test_perf_conn   50 100 10   50 10
+test_perf_conn  500 100 10  500 10
+test_perf_conn 2000 100 10 2000 10
+test_perf_conn 8000 100 10 8000 10
+
+# Scaling commit count (200K files, 10 files/commit).
+test_perf_conn 2000 100    1 2000 10
+test_perf_conn 2000 100   10 2000 10
+test_perf_conn 2000 100  100 2000 10
+test_perf_conn 2000 100   500 2000 10
+test_perf_conn 2000 100  3000 2000 10
+test_perf_conn 2000 100  5000 2000 10
+test_perf_conn 2000 100 10000 2000 10
+
+# Scaling files per commit (200K files, 10 commits).
+test_perf_conn 2000 100 10 2000   1
+test_perf_conn 2000 100 10 2000  10
+test_perf_conn 2000 100 10 2000  100
+test_perf_conn 2000 100 10 2000  500
+test_perf_conn 2000 100 10 2000 1000
+test_perf_conn 2000 100 10 2000 2000
+
+test_done