feat(mcp): opt-in async index_repository with status polling (#2144) - #2357
Merged
Merged
Conversation
index_repository blocks until the whole index completes. MCP clients with a per-call deadline (Copilot for IntelliJ) give up on a large repository; their cancel drops the daemon job's last subscriber, the daemon cancels the worker, and every retry starts over, so the index never finishes. Add an opt-in async mode and a status query to the same tool; the default synchronous behaviour is unchanged: - async: true starts the project's index job in the daemon, or joins the one already running for it, and returns at once with its state. The application holds one subscriber reference for an async job until its terminal publish, so the request returning, the client cancelling or the starting session closing no longer cancels it. Only daemon shutdown (or the final session of a non-permanent generation) ends it. Async requests never queue behind the physical job limit; a full daemon answers busy. async/status are stripped from the worker args, so an async request coalesces with an identical synchronous one. - status: true reports the running or last job for the project (queued/running/cancelling/succeeded/failed/cancelled, started_at, finished_at, error summary) from a small per-project record kept in the daemon's job registry after the job itself is reaped. An unknown project is an error; a project indexed before this daemon generation reports idle. No freshness key is added: index_status keeps the #1561 freshness object. - async + status together, non-boolean values, and async with cross-repo-intelligence are refused. In-process servers (index worker, embedders) refuse both modes: nothing there outlives the call. - On a temporary (non-permanent) daemon, async is refused when the request arrives on the one-shot `cli` tool channel and that session is the daemon's only live session: the daemon would stop and cancel the job the moment the command exits. The error points at `daemon start` (or an open MCP session). A long-lived MCP session stays a valid host even when it is the only one, which is the IDE case this exists for; status stays allowed everywhere. The daemon lifecycle is unchanged. - New allocations go through the memory core (cbm_alloc/cbm_calloc/ cbm_mem_strdup/cbm_free); blocks returned by non-core APIs are released with safe_free, so no file's raw allocator count grows. - A cut-short synchronous call now offers the async alternative: in the cancellation text the daemon or frontend delivers (including the -32800 reply to a cancelled index_repository), and, because a timed-out client never reads that reply, as a notice on the next index_repository or status call for the project. - The tool description and schema, the CLI help and the README explain the flags, the deadline problem and the polling pattern. No server-side timeout or budget is introduced. Tests (deterministic, held fake worker, bounded observation waits): async returns before completion and status observes running then succeeded; an async job survives its session closing while a sync job in the same shape is still cancelled; validation errors; the cancellation reply and the next-call notice carry the advice; a lone one-shot client of a temporary daemon is refused async while status and an MCP session on the same daemon are allowed; in-process refusal; schema; notice helper. Refs #2031 Fixes #2144 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in async mode and a status query to
index_repository, so indexes that outlive an MCP client's per-call deadline (e.g. Copilot for IntelliJ) still complete.async: truestarts (or joins) the project's index job in the daemon and returns immediately. The job is decoupled from the request and its session: a client cancel, deadline or disconnect no longer cancels it. It never queues behind the job limit (busy is reported with a retry hint).status: true(samerepo_path/name) reports queued / running / cancelling / succeeded / failed / cancelled,started_at,finished_atand an error summary, from a per-project record in the daemon's job registry.index_statusis untouched and no freshness key is added.clicall that is the only client of a temporary daemon (the daemon would stop and cancel the job when the command exits); the error points tocodebase-memory-mcp daemon startor an open MCP session. An IDE/MCP session on a temporary daemon is a valid host.statusis never refused. The daemon lifecycle is unchanged.noticeon the next index/status call for that project. No new server-side timeouts or budgets.--helpand README document the polling pattern and the host requirement. Default synchronous behaviour and its cancel-on-last-subscriber semantics are unchanged.mem_core;make lint-cipasses (memory-core: none grew).Tests (deterministic, held fake worker, no sleeps or wall-time assertions): async returns before completion; status polls running → succeeded; an async job survives its session closing; a sync job of the same shape is still cancelled; validation errors; the cut-short hint in the cancel reply and on the next call; refusal for a lone one-shot client of a temporary daemon while an MCP session on the same daemon is accepted. Each is RED before the change and RED on revert.
Proof: a 57 s index of dotnet/runtime
src/libraries(~20.9k .cs files) started by a call that returned in 1.44 s, polled tosucceeded; 595,714 nodes / 3,570,617 edges persisted.Related: #2031 (live progress in
index_status), not touched here and not conflicting.Fixes #2144