Non-git silent on staleness

[chaking22]

Version

codebase-memory-mcp 0.10.8

Platform

Windows (x64)

Install channel

GitHub release archive / install.sh / install.ps1

Binary variant

standard

What happened, and what did you expect?

I understand non-git repos aren't currently watched, just throwing this in there to be helpful:
A project indexed from a directory that is not a git repository is never refreshed again. It stays frozen at its first index for the life of the database, and nothing in the API indicates this.

and poll_project() has:

/* Skip non-git projects */
if (!s->is_git) {
    return;
}

The problem is not that these projects go stale. It is that going stale is invisible, and a stale graph produces confident wrong answers rather than obviously missing ones. A search_graph for a function that exists on disk returns:

total: 0
hint: "No nodes match this pattern. Check spelling or try a broader regex."

That is indistinguishable from a true negative, so a caller - especially an LLM agent - concludes the symbol does not exist and acts on it. Meanwhile index_status reports "status": "ready" with no last-indexed timestamp and no watched/unwatched flag.

Expected: either non-git roots get some refresh path (an mtime scan fallback), or - much cheaper - the API exposes the coverage, so a caller can decide whether to trust the answer and run an index or whatnot as needed.

Suggested fix:

1. Surface it. Add a watched: true|false field and a last-indexed timestamp to index_status. Callers can then tell "no results" from "no results, and this project has not been checked since Tuesday." This is small and fixes the dangerous part — the silence.
2. Fallback scan. For non-git roots, poll a stat-walk signature (path:size:mtime folded into a hash) on the same adaptive interval already used for git roots. git_dirty_signature() in watcher.c already does essentially this for dirty paths; the non-git case needs the same idea applied to the whole tree.

Additional Context:

Found while building an out-of-process freshness gate to work around this. Happy to test a patch — I have reproductions for both the git and non-git paths.

Reproduction

1. Create a plain directory that is not a git repository:

   mkdir /tmp/nogit && cd /tmp/nogit
   cat > app.py <<'EOF'
   def alpha():
       return 1
   
   def beta():
       return alpha() + 1
   EOF

   (No git init - that is the whole point.)

2. Index it:

   codebase-memory-mcp cli index_repository --repo-path /tmp/nogit

3. Confirm the graph is populated:

   codebase-memory-mcp cli search_graph --project <name> --name-pattern '.*alpha.*'
   # -> 1 result

4. Append a function to the file:

   printf '\ndef gamma_uniquename():\n    return beta() * 2\n' >> app.py

5. Wait as long as you like — minutes, hours, a server restart — then:

   codebase-memory-mcp cli search_graph --project <name> --name-pattern '.*gamma.*'

Actual: total: 0, with the "check spelling" hint. Permanently — no poll interval ever covers this project.

Expected: the new function is found, or the response indicates the index may be stale.

Doing the same thing in a git init-ed directory behaves correctly: the
watcher notices and reindexes.

Logs

Diagnostics trajectory (memory / performance / leak issues)

Project scale (if relevant)

No response

Confirmations

• I searched existing issues and this is not a duplicate.
• My reproduction uses shareable code (a dummy snippet or a public OSS repository), not proprietary code.

[DeusData]

Thank you, @chaking22, for such a careful report. The repro, the pointer at poll_project(), and especially your point that silent staleness is the dangerous part made this easy to act on. We're doing both of your suggestions:

1. The staleness signal will go into index_status through the freshness object being built in feat(mcp): report truthful index freshness from checkout evidence #1561, so callers can tell "no results" from "no results, and this index may be old". We're keeping one freshness surface rather than adding a second key.
2. The fallback scan is up as feat(watcher): opt-in polling of non-git project roots (#1948) #2333, as an opt-in: codebase-memory-mcp config set watch_non_git true, then codebase-memory-mcp daemon stop so the next session starts a daemon that reads it. Non-git roots are then polled on the same adaptive interval with a stat-walk signature (the indexer's own ignore rules, hashing path, size and mtime), very close to what you proposed. It's opt-in because it walks the whole tree on every poll. Your reproduction picks up the new function within one poll interval.

We'd gladly take you up on your offer to test the patch, on Windows especially.

[chaking22]

Happy to be of some help - I took you up on the offer and ran some tests via windows sandbox. Found 1 issue with non-ASCII project roots (fairly serious, see below), passed everything else.

Env:

• Windows 10 Enterprise 24H2 build 26100.9550
• MSYS2 CLANG64, clang version 22.1.8 (x86_64-w64-windows-gnu)
• Branch head e65a4e7
• binary reports codebase-memory-mcp dev
• built with recipe from _build.yml - 165s clean build

Actions:

• cli index_repository for each test folder
• opened one stdio MCP session per project
• cwd was project root
• for watch_non_git, set true in config, results in watcher.non_git mode=tree_poll
• Re-indexes were counted from watcher.changed lines in cbm-daemon.log
• Searches used cli search_graph

Tests and Results:

#	Case	Result
1	#1948 repro, key off: non-git folder, append 'gamma_uniquename'	PASS - watcher.baseline strategy=none, no reindex, search total: 0 (old behaviour, as documented)
2	Same, key on	PASS - exactly 1 reindex on the first poll after start; edit detected in 2.2 s, symbol found at 3.5 s
3	Same as 2, root 'C:\Users\…\work\c3-tëst-ñame-日本`'	FAIL - see below
4	Same-size in-place edit (alpha→alphb, byte count unchanged)	PASS - detected 6.6 s, found 7.8 s (mtime alone changed the signature)
5	Edit only a file listed in the folder's '.gitignore'	PASS - no reindex
6	Runaway guard: 5 quiet intervals after reindexes (2 projects)	PASS - 0 further reindexes
7	Non-git subfolder of a dirty git repo, indexed as its own project	PASS - 'watcher.nested_non_git' logged; editing the parent's tracked file: 0 reindexes; editing the subfolder: reindex, found at 5.6 s
8	Git project, key off (regression check)	PASS - strategy=git, detected 4.4 s, found 7.9 s

# 3: non-ASCII root is treated as missing
Indexing the folder works and watcher.watch logs the correct path, however polling shows:

level=warn 
msg=watcher.root_missing 
project=C-Users-WDAGUtilityAccount-work-c3-tc3abst-c3b1ame-e697a5e69cac 
path=C:\Users\WDAGUtilityAccount\work\c3-tëst-ñame-日本 polls=N

No watcher.baseline or watcher.changed and the appended symbol is never found. Believe that poll_project() calls root_status() before anything else and root_status() uses stat(root_path, &st) on the UTF-8 path. With MinGW that goes through the ANSI code page, fails with ENOENT, and returns 'ROOT_MISSING'.

To add to that, it looks like root_missing feeds root pruning which deletes the project. Presumably this affects git and non-git projects. Tested this by setting grace period to 0 and it was removed after 3 polls ~26s.