Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -738,10 +738,12 @@ codebase-memory-mcp config set auto_watch false # don't register backgr
codebase-memory-mcp config set watcher_enabled false # stop the watcher thread entirely (default: true)
codebase-memory-mcp config set index_max_files 250000 # optional per-index source-file limit
codebase-memory-mcp config set index_max_source_mb 16384 # optional per-index source-size limit
codebase-memory-mcp config set index_max_rss_mb 8192 # optional worker-tree charged-memory limit
codebase-memory-mcp config set index_max_duration_seconds 3600 # optional per-request duration
codebase-memory-mcp config reset auto_index # reset to default
```

The two `index_max_*` settings default to `off`. Exceeding one fails the complete
The four `index_max_*` settings default to `off`. Exceeding one fails the complete
index attempt rather than publishing a partial graph; an existing serving index
is preserved. See [Index resource limits](docs/INDEX_RESOURCE_LIMITS.md).

Expand Down
10 changes: 7 additions & 3 deletions docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@ Current keys:
| `watcher_enabled` | `true` | Master switch for the background watcher subsystem. Set `false` to stop the watcher from starting at all — no poll thread and no project registration. Reindex manually with `index_repository` when disabled. |
| `index_max_files` | `off` | Optional maximum number of accepted source files in one discovery run. |
| `index_max_source_mb` | `off` | Optional maximum accepted source size in MiB in one discovery run. |
| `index_max_rss_mb` | `off` | Optional maximum charged memory in MiB for the complete contained index-worker process tree (`64..1048576`). On macOS this is phys_footprint, not resident size. |
| `index_max_duration_seconds` | `off` | Optional maximum wall-clock duration in seconds for the whole index request, including crash/hang recovery (`1..86400`). |

> **`watcher_enabled` vs `auto_watch`.** `watcher_enabled` controls whether the
> watcher *subsystem* starts at all (the background poll thread). `auto_watch` is
Expand Down Expand Up @@ -118,11 +120,13 @@ Current keys:
> `auto_index` still runs, and `index_repository` stays available for manual
> reindexing.

The two `index_max_*` settings are independent and disabled by default. They
The four `index_max_*` settings are independent and disabled by default. They
apply to explicit indexing, automatic indexing, and watcher re-indexing, but not
to `cross-repo-intelligence`, which does not scan repository source files.
Equality is allowed; exceeding either setting fails the complete index request
and preserves any previously serving database. See
Equality is allowed; exceeding any setting fails the complete index request and
preserves any previously serving database. Worker RSS covers descendants and is
not the same as the internal `CBM_MEM_BUDGET_MB` allocation budget. Total
duration is independent of the existing 15-minute no-log-progress timeout. See
[Index resource limits](INDEX_RESOURCE_LIMITS.md) for counting, validation, and
error-response details.

Expand Down
54 changes: 50 additions & 4 deletions docs/INDEX_RESOURCE_LIMITS.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Index resource limits

Index resource limits are optional operator controls for repositories whose
discovery breadth is not known in advance. They are disabled by default so
existing large-repository workloads retain their current behavior.
discovery breadth or worker runtime is not known in advance. They are disabled
by default so existing large-repository workloads retain their current behavior.

## Discovery settings

Expand All @@ -23,6 +23,45 @@ Values use base-10 integers. MiB means 1,048,576 bytes. Empty values, zero,
negative values, suffixes, trailing characters, and values outside the stated
ranges are rejected without changing the stored value.

## Worker settings

| Key | Default | Accepted value | Protects |
|---|---:|---:|---|
| `index_max_rss_mb` | `off` | `off` or `64..1048576` | Charged memory of the complete worker process tree |
| `index_max_duration_seconds` | `off` | `off` or `1..86400` | Wall-clock duration of the whole index request |

```bash
codebase-memory-mcp config set index_max_rss_mb 8192
codebase-memory-mcp config set index_max_duration_seconds 3600
```

RSS is the charged memory of the contained worker and every descendant, not
the worker's allocation budget and not peak memory. On macOS that quantity is
`phys_footprint` (the same number `cbm_mem_charged()` enforces), not
`resident_size`, which still counts pages the allocator has already handed
back. Linux and Windows use RSS / working set. This hard watchdog is
separate from the internal `CBM_MEM_BUDGET_MB` soft budget. The supervisor
samples the tree at most once every 250 milliseconds so the watchdog does not
turn full process-table enumeration into a busy loop.

Duration is per request: the clock starts at the first worker spawn and is
not reset when crash/hang recovery starts a later attempt. Continuous log
progress does not reset it. It is independent of the existing 15-minute quiet
timeout, which still identifies a worker that stops making progress. A
duration limit shorter than that quiet timeout kills a hung worker before hang
quarantine can name the in-flight file, so the next attempt may hang on the
same file.

Equality is allowed. The first RSS or elapsed-duration observation above its
limit starts the existing graceful-to-force process-tree shutdown. CBM reports
terminal only after the tree is quiescent or a bounded containment failure is
explicitly surfaced. Resource termination is not retried and does not
quarantine a source file.

If RSS is enabled and three consecutive probes cannot obtain any trustworthy
tree measurement while the root worker is still running, CBM fails closed with
`code=resource_probe_failed`.

## Counting and failure semantics

`index_max_files` counts a file only after it passes directory pruning, ignore
Expand Down Expand Up @@ -55,6 +94,12 @@ The previous database remains available because publication occurs only after a
complete discovery and successful staged build. If no previous database exists,
`serving_index_preserved` is false.

Worker limit failures use the same shape with `stage=worker`,
`resource=rss_bytes` and `unit=bytes`, or `resource=duration_ms` and
`unit=milliseconds`. RSS measurement failures use `code=resource_probe_failed`
and omit `observed`, `limit`, and `unit` because no trustworthy observation was
available.

## Trust and compatibility

Limits are read from the CLI-managed `_config.db`; they are not MCP request
Expand All @@ -64,5 +109,6 @@ parent policy.

These settings do not replace or increase `auto_index_limit`, change the 512 MiB
single-file cap, alter workspace-root authorization, or affect
`cross-repo-intelligence`. With both settings `off`, discovery follows the
existing unbounded path.
`cross-repo-intelligence`. With all settings `off`, discovery follows the
existing path and the supervisor performs no periodic RSS probe or total-duration
termination.
3 changes: 2 additions & 1 deletion scripts/test-runtime.sh
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,8 @@ _cbm_test_runtime_daemon() {
# cbm_mcp_index_policy_add_to_args: every key in cbm_index_policy_key_at, and
# nothing else.
cbm_test_index_worker_policy_json() {
printf '%s' '"_cbm_index_policy":{"index_max_files":"off","index_max_source_mb":"off"}'
printf '%s' '"_cbm_index_policy":{"index_max_files":"off","index_max_source_mb":"off"'
printf '%s' ',"index_max_rss_mb":"off","index_max_duration_seconds":"off"}'
}

cbm_test_runtime_cleanup() {
Expand Down
11 changes: 9 additions & 2 deletions src/cli/cli.c
Original file line number Diff line number Diff line change
Expand Up @@ -7449,6 +7449,9 @@ static const config_key_def_t CONFIG_KEYS[] = {
{CBM_CONFIG_UI_PORT, "9749", "Port for the graph UI listener when enabled"},
{CBM_INDEX_CONFIG_MAX_FILES, "off", "Max accepted source files per index, or off"},
{CBM_INDEX_CONFIG_MAX_SOURCE_MB, "off", "Max accepted source MiB per index, or off"},
{CBM_INDEX_CONFIG_MAX_RSS_MB, "off", "Max worker process-tree charged memory MiB, or off"},
{CBM_INDEX_CONFIG_MAX_DURATION_SECONDS, "off",
"Max wall-clock seconds for the whole index request, or off"},
};

/* #1558: ui_enabled and ui_port were reachable ONLY by hand-editing
Expand All @@ -7475,8 +7478,12 @@ static bool config_key_is_ui(const char *key) {
}

static bool config_key_is_index_policy(const char *key) {
return key && (strcmp(key, CBM_INDEX_CONFIG_MAX_FILES) == 0 ||
strcmp(key, CBM_INDEX_CONFIG_MAX_SOURCE_MB) == 0);
for (size_t index = 0; key && index < cbm_index_policy_key_count(); index++) {
if (strcmp(key, cbm_index_policy_key_at(index)) == 0) {
return true;
}
}
return false;
}

static int config_index_policy_write(cbm_config_t *config, const char *key, const char *value) {
Expand Down
30 changes: 25 additions & 5 deletions src/daemon/application.c
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ struct cbm_daemon_application_job {
bool cancelled;
bool cancel_requested;
bool supervision_failed;
uint64_t request_started_ms;
cbm_daemon_application_job_t *next;
};

Expand Down Expand Up @@ -303,11 +304,21 @@ static void application_project_lock_release_fully(cbm_project_lock_lease_t **le
static int application_worker_start_default(void *context, const char *args_json,
size_t memory_budget_bytes, const char *marker_file,
const char *quarantine_file,
uint64_t duration_origin_ms,
cbm_daemon_application_worker_t *worker_out) {
(void)context;
cbm_index_resource_policy_t resource_policy;
char error[CBM_SZ_256] = {0};
if (!cbm_mcp_index_policy_from_internal_args(args_json, &resource_policy, error,
sizeof(error))) {
cbm_log_error("daemon.index.policy", "error", error);
*worker_out = NULL;
return -1;
}
cbm_index_worker_handle_t *worker = NULL;
int result = cbm_index_worker_start(args_json, memory_budget_bytes, false, marker_file,
quarantine_file, &worker);
int result = cbm_index_worker_start_with_policy(args_json, memory_budget_bytes,
&resource_policy, false, marker_file,
quarantine_file, duration_origin_ms, &worker);
*worker_out = worker;
return result;
}
Expand Down Expand Up @@ -1115,10 +1126,13 @@ static application_attempt_status_t application_job_run_attempt(cbm_daemon_appli
}

cbm_daemon_application_worker_t worker = NULL;
if (job->request_started_ms == 0) {
job->request_started_ms = cbm_index_worker_now_ms();
}
application_tmp_lock();
int start_result =
application->worker_ops.start(application->worker_ops.context, job->args_json,
memory_budget_bytes, marker_path, quarantine_path, &worker);
int start_result = application->worker_ops.start(
application->worker_ops.context, job->args_json, memory_budget_bytes, marker_path,
quarantine_path, job->request_started_ms, &worker);
application_tmp_unlock();
if (start_result != 0 || !worker) {
return application_job_cancel_requested(job) ? APPLICATION_ATTEMPT_CANCELLED
Expand Down Expand Up @@ -1284,6 +1298,12 @@ static application_attempt_decision_t application_consume_attempt(
application_attempt_free(attempt);
return APPLICATION_ATTEMPT_DECISION_SUCCESS;
}
if (disposition == CBM_MCP_SUPERVISED_RESULT_RESOURCE_FAILURE) {
execution->response =
cbm_mcp_index_worker_resource_response(job->args_json, &attempt->result);
application_attempt_free(attempt);
return APPLICATION_ATTEMPT_DECISION_STOP;
}
if (disposition == CBM_MCP_SUPERVISED_RESULT_UNSAFE_TERMINAL) {
execution->unsafe_terminal = true;
execution->supervision_failed =
Expand Down
2 changes: 1 addition & 1 deletion src/daemon/application.h
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ typedef void *cbm_daemon_application_update_worker_t;
typedef struct {
void *context;
int (*start)(void *context, const char *args_json, size_t memory_budget_bytes,
const char *marker_file, const char *quarantine_file,
const char *marker_file, const char *quarantine_file, uint64_t duration_origin_ms,
cbm_daemon_application_worker_t *worker_out);
cbm_index_worker_poll_t (*poll)(void *context, cbm_daemon_application_worker_t worker,
const cbm_index_worker_result_t **result_out);
Expand Down
Loading
Loading