diff --git a/CHANGELOG.md b/CHANGELOG.md index de9de4fc..75d22172 100755 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,40 +7,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +### Changed + ### Fixed -- **`aim-sot consult` no longer returns stale registry entries** — consult read the - derived memory cache (5b) first and returned it whenever non-empty, with no binding - to the committed `.sot/registry.yaml`, so after any registry edit (or when another - project's rows shared the `group_id`) it shadowed the file with stale or cross-state - data. Consult now uses the cache only when the drift cache's `registry_sha` matches - the committed file's SHA, otherwise it falls back to the committed file; it remains - strictly read-only. Also: `verify --proposal` now flags a proposal that lacks an - `entries` key instead of silently verifying it as empty; `detect-propose run` prunes - drift-cache records for components no longer in the registry (parity with reindex); - and the SKILL.md 5b-cache field name is corrected to `type=sot_entry` to match the - engine. (`aim_sot_consult.py`, `aim_sot_verify.py`, `aim_sot_detect_propose.py`, `SKILL.md`) +## [2.6.0] - 2026-06-16 -- **BUG-302: tier-2 fallback marker now shows `remaining=` alongside `tokens=` and `budget=`** - — the marker previously printed `tokens= budget=`, which read as a false - contradiction when `tokens < budget` but `tokens > budget − tokens_used`. Adding - `remaining=budget−tokens_used` makes the correct reject self-evident. - (`context_injection_tier2.py` marker block) +### Upgrade Instructions -- **`aim-sot` is now discoverable as a skill in installed projects** — the skill's - engine lives under `_ai-memory/skills/aim-sot/`, which the installer copied as - canonical files but never surfaced to `.claude/skills/`, so Claude Code could not - index it even though its session/drift hooks were registered. `deploy_ai_memory_skills` - now generates a thin discovery shim in `.claude/skills/` for `aim-*` skills that live - under `_ai-memory/skills/` without a full copy. It runs on every install (matching - aim-sot's always-on SOT hooks), never clobbers a skill that already has a full copy, - and is idempotent on re-install; the `aim-*` prefix deliberately excludes the - oversight-internal `parzival-save-*` skills. The `AI_MEMORY_SOT_HOOKS` opt-out is now - documented as a commented line in `docker/.env.example`. - (`scripts/install.sh`, `docker/.env.example`) +- **Existing installs**: re-run `./scripts/install.sh ` to pick up the `aim-sot` skill surface + agent-guidance files (both deploy on every install). +- **Embedding image (TD-626)**: run `~/.ai-memory/scripts/stack.sh restart` to recreate services and pull the prebuilt GHCR embedding image (`ghcr.io/hidden-history/ai-memory-embedding`). `scripts/install.sh` brings services up with `--no-recreate`, so a running `embedding` container keeps its previous locally-built image until a restart. Fresh installs pull the image automatically. +- **aim-sot hooks**: the SOT digest/drift hooks auto-register on install for all supported CLIs (default-on); set `AI_MEMORY_SOT_HOOKS=off` in `docker/.env` before install to opt out. +- **Maintainer (one-time)**: set the `ai-memory-embedding` GHCR package to **Public** (repo → Packages → Package settings → Change visibility) so anonymous `scripts/install.sh` pulls succeed without authentication; until then those pulls fall back to a local source build. ### Added +- **`aim-sot` Source-of-Truth subsystem** — new `aim-sot` skill that tracks where the + canonical truth lives for each boundary of the user's own project. A committed + `.sot/registry.yaml` (in the user's own repo) is the registry of record; the skill + ships the schema, templates, and a three-mode engine: + - **consult** — read-only query over the registry (served from the derived memory + cache, falling back to the committed file); answers "where does X live / who owns it" + for any component. + - **detect-propose** — hybrid auto-discover → propose: scans for candidate components, + computes actual state (SHA-256 of each `sot_location` file), and emits a **proposed + patch** on drift or new candidates. **Never writes the registry** — the propose-only + guarantee is unconditional; every registry change goes through the human-review + + verify gate. The baseline SHA is held (not advanced) when drift is detected, so the + proposal re-fires until a human confirms the change. Cold-start `drift_status` is + `unverified`, not `clean`. + - **verify** — 16-check gate (Schema · Referential · Completeness · Content) returning + PASS / CONDITIONAL / FAIL. K1 (content-hash check) reports CONDITIONAL — not a + silent pass — when no baseline exists. Verdicts distinguish checks that ran and + produced a result from no-op/inert checks and skipped-no-baseline checks. + + Two runtime caches support the feature: a per-install drift cache + (`~/.ai-memory/drift-state/sot_drift_{project_id}.json`, machine-local, never + committed) and a derived memory cache (Qdrant `conventions` collection, + `memory_type=sot_entry`, rebuildable from the committed registry at any time via + `detect-propose reindex`). The session-start digest hook and drift Stop hook are + **auto-registered on install for all supported CLIs** (Claude Code, Codex, Cursor, + Gemini) — default on; set `AI_MEMORY_SOT_HOOKS=off` in `docker/.env` to skip + registration. The engine also runs standalone (`detect-propose run`) as the default + no-hook path. All trigger paths are fail-open (exit 0 on any error). + + **Data safety**: SOT `owner` and `added_by` fields carry GitHub handles that must + survive the shared PII scanner intact. For `sot_entry` writes only, the scanner's + `HANDLE` (GitHub @-handle) redaction is exempted so ownership attribution is + preserved faithfully in the derived cache; all other masking — secrets, email + addresses, IP addresses, SSNs — is unchanged for SOT entries and for every other + memory type. + - **Per-source budget ledger in `select_results_greedy`** — `meta["per_source"]` now carries a per-collection breakdown of `requested_tokens`, `loaded_tokens`, and `dropped` counts by reason (`budget_exceeded`, `score_gap`, `freshness_block`, @@ -51,16 +70,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `per_source`. Observe-only: selection output is unchanged. (`injection.py`, `context_injection_tier2.py`) -### Changed - -- **Agent-guidance files now refresh on every install** — existing installations - pick up updated guidance automatically by re-running - `./scripts/install.sh `. No special flag is needed; the guidance - file for each configured CLI is deployed on every install, not only on the - initial setup. - -### Added - - **Sanctum content-drift detection (`aim-content-drift`)** — a new skill that compares an operator's scaffolded sanctum files (BOND, CAPABILITIES, CREED, INDEX, LORE, MEMORY, PERSONA, PULSE) against the reference templates and surfaces @@ -131,36 +140,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `INSTALL-GUIDE-POV.md` gains an `add-project` / shared-stack section documenting the `INSTALL_MODE=add-project` flow: when it triggers, what the installer skips vs. runs, prerequisites, per-project configuration, and verification steps. -- **`aim-sot` Source-of-Truth subsystem** — new `aim-sot` skill that tracks where the - canonical truth lives for each boundary of the user's own project. A committed - `.sot/registry.yaml` (in the user's own repo) is the registry of record; the skill - ships the schema, templates, and a three-mode engine: - - **consult** — read-only query over the registry (served from the derived memory - cache, falling back to the committed file); answers "where does X live / who owns it" - for any component. - - **detect-propose** — hybrid auto-discover → propose: scans for candidate components, - computes actual state (SHA-256 of each `sot_location` file), and emits a **proposed - patch** on drift or new candidates. **Never writes the registry** — the propose-only - guarantee is unconditional; every registry change goes through the human-review + - verify gate. The baseline SHA is held (not advanced) when drift is detected, so the - proposal re-fires until a human confirms the change. Cold-start `drift_status` is - `unverified`, not `clean`. - - **verify** — 16-check gate (Schema · Referential · Completeness · Content) returning - PASS / CONDITIONAL / FAIL. K1 (content-hash check) reports CONDITIONAL — not a - silent pass — when no baseline exists. Verdicts distinguish checks that ran and - produced a result from no-op/inert checks and skipped-no-baseline checks. - - Two runtime caches support the feature: a per-install drift cache - (`~/.ai-memory/drift-state/sot_drift_{project_id}.json`, machine-local, never - committed) and a derived memory cache (Qdrant `conventions` collection, - `memory_type=sot_entry`, rebuildable from the committed registry at any time via - `detect-propose reindex`). The Claude `Stop` hook and multi-CLI adapters (Codex, - Cursor, Gemini) ship with the install but are **not auto-registered** — opt-in only - (see `docs/AIM-SOT.md`). The engine also runs standalone (`detect-propose run`) - as the default no-hook path. All trigger paths are fail-open (exit 0 on any error). ### Changed +- **Agent-guidance files now refresh on every install** — existing installations + pick up updated guidance automatically by re-running + `./scripts/install.sh `. No special flag is needed; the guidance + file for each configured CLI is deployed on every install, not only on the + initial setup. + - **Sanctum templates** — aligned the eight scaffolded sanctum files (`CREED`, `PERSONA`, `BOND`, `LORE`, `MEMORY`, `CAPABILITIES`, `INDEX`, `PULSE`) to a per-file memory-type content model: each file states its memory type, and content @@ -204,6 +192,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **`aim-sot consult` no longer returns stale registry entries** — consult read the + derived memory cache (5b) first and returned it whenever non-empty, with no binding + to the committed `.sot/registry.yaml`, so after any registry edit (or when another + project's rows shared the `group_id`) it shadowed the file with stale or cross-state + data. Consult now uses the cache only when every row's stamped `registry_sha` matches + the committed file's SHA **and** the row count matches the committed entry count; a + SHA mismatch or count shortfall falls back to the committed file. Consult remains + strictly read-only. Also: `verify --proposal` now flags a proposal that lacks an + `entries` key instead of silently verifying it as empty; `detect-propose run` prunes + drift-cache records for components no longer in the registry (parity with reindex); + and the SKILL.md 5b-cache field name is corrected to `type=sot_entry` to match the + engine. (`aim_sot_consult.py`, `aim_sot_verify.py`, `aim_sot_detect_propose.py`, `SKILL.md`) + +- **BUG-302: tier-2 fallback marker now shows `remaining=` alongside `tokens=` and `budget=`** + — the marker previously printed `tokens= budget=`, which read as a false + contradiction when `tokens < budget` but `tokens > budget − tokens_used`. Adding + `remaining=budget−tokens_used` makes the correct reject self-evident. + (`context_injection_tier2.py` marker block) + +- **`aim-sot` is now discoverable as a skill in installed projects** — the skill's + engine lives under `_ai-memory/skills/aim-sot/`, which the installer copied as + canonical files but never surfaced to `.claude/skills/`, so Claude Code could not + index it even though its session/drift hooks were registered. `deploy_ai_memory_skills` + now generates a thin discovery shim in `.claude/skills/` for `aim-*` skills that live + under `_ai-memory/skills/` without a full copy. It runs on every install (matching + aim-sot's always-on SOT hooks), never clobbers a skill that already has a full copy, + and is idempotent on re-install; the `aim-*` prefix deliberately excludes the + oversight-internal `parzival-save-*` skills. The `AI_MEMORY_SOT_HOOKS` opt-out is now + documented as a commented line in `docker/.env.example`. + (`scripts/install.sh`, `docker/.env.example`) + - **aim-sot verify** — `verify run --registry ` no longer crashes on a non-conforming (flat) registry path. A registry outside `/.sot/` makes the project root resolve to `None`, which previously reached the path checks diff --git a/README.md b/README.md index 2c4689bc..1c434bc5 100755 --- a/README.md +++ b/README.md @@ -146,6 +146,7 @@ Memory retrieval fires automatically on: See [docs/HOOKS.md](docs/HOOKS.md) for complete hook documentation and the full trigger keyword reference. See [docs/AI_MEMORY_ARCHITECTURE.md](docs/AI_MEMORY_ARCHITECTURE.md) for the full system architecture reference. +See [docs/AIM-SOT.md](docs/AIM-SOT.md) for the Source-of-Truth subsystem — registry schema, consult / detect-propose / verify engine, and automatic drift detection hooks. --- diff --git a/docs/AIM-SOT.md b/docs/AIM-SOT.md index fd003849..053e038e 100644 --- a/docs/AIM-SOT.md +++ b/docs/AIM-SOT.md @@ -10,7 +10,7 @@ The registry lives in **your own repository** as a committed `.sot/registry.yaml |------|----------|----------| | **Registry of record** | `.sot/registry.yaml` | Your repo (committed, diff-reviewable) | | **Skill** | `aim-sot` (consult · detect-propose · verify) | `_ai-memory/skills/aim-sot/` | -| **Triggers (opt-in)** | Claude `Stop` hook; Codex, Cursor, Gemini adapters | Shipped unregistered — see [Opt-In](#enabling-automatic-drift-detection) | +| **Triggers (default-on)** | Claude session-start digest + `Stop` hook; Codex, Cursor, Gemini adapters | Auto-registered on install; opt out with `AI_MEMORY_SOT_HOOKS=off` — see [Automatic Drift Detection](#automatic-drift-detection) | | **Per-install drift cache (5a)** | `~/.ai-memory/drift-state/sot_drift_{project_id}.json` | Runtime; never committed | | **Derived memory cache (5b)** | `conventions` collection, `memory_type=sot_entry` | Qdrant; rebuildable | @@ -77,7 +77,7 @@ bash "${AI_MEMORY_INSTALL_DIR:-$HOME/.ai-memory}/scripts/memory/run-with-env.sh" ### consult — Orient before acting -Read-only query over your committed registry, served from the derived memory cache (5b) with a fallback to the committed file. Useful for agents that need to locate a component's owner or canonical location before editing it. +Read-only query over your committed registry. Results are served from the derived memory cache (5b) only when every cached row's stamped `registry_sha` matches the committed registry's SHA **and** the row count matches the committed entry count — otherwise the engine falls back to the committed file directly. This ensures consult never returns stale, partial, or cross-state entries. Useful for agents that need to locate a component's owner or canonical location before editing it. ```bash bash "${AI_MEMORY_INSTALL_DIR:-$HOME/.ai-memory}/scripts/memory/run-with-env.sh" \ @@ -190,11 +190,11 @@ This is an unconditional structural guarantee: the engine, the Stop hook, and al All trigger paths are **fail-open**: any error is logged to stderr and the adapter exits 0, so a transient failure never blocks a Claude Code (or Codex/Cursor/Gemini) session. -## Enabling Automatic Drift Detection +## Automatic Drift Detection -The Stop hook and all CLI adapter hooks ship **unregistered**. The installer does not modify your `settings.json`, `.cursor/hooks.json`, `.codex/hooks.json`, or `.gemini/settings.json`. Opt in manually per CLI. +The per-session-start SOT digest hook and the end-of-session drift hook are **registered automatically on install** for every configured IDE (Claude Code, Codex, Cursor, Gemini). To opt out, set `AI_MEMORY_SOT_HOOKS=off` (case-insensitive; only `off` disables — `false`/`0`/`no` leave hooks on) before running `install.sh`. Note: `AI_MEMORY_SOT_HOOKS=off` prevents registration on install but does not remove hooks already written by a prior install; to disable an existing install, remove the SOT hook entries from the relevant config file manually. -The engine also runs standalone with no hook required — use this as the default no-hook path, or schedule it as a cron job: +The engine also runs standalone with no hook required — use this as an alternative, or schedule it as a cron job: ```bash bash "${AI_MEMORY_INSTALL_DIR:-$HOME/.ai-memory}/scripts/memory/run-with-env.sh" \ @@ -202,9 +202,13 @@ bash "${AI_MEMORY_INSTALL_DIR:-$HOME/.ai-memory}/scripts/memory/run-with-env.sh" run ``` -### Claude Code — Stop hook +### Claude Code -Add the following entry to your project's `.claude/settings.json` under `hooks.Stop`: +The installer registers two SOT hooks in your project's `.claude/settings.json`: + +**Session-start digest** (under `hooks.SessionStart`, matcher `resume|compact`): invokes `sot_digest_session_start.py` and injects a compact SOT registry digest as ambient context on session resume or compact. + +**Stop drift hook** (under `hooks.Stop`): invokes `sot_drift_stop.py` at every session end and prints a one-line drift summary to stderr when drift or new candidates are detected. The installed entry: ```json { @@ -225,11 +229,9 @@ Add the following entry to your project's `.claude/settings.json` under `hooks.S } ``` -Once registered, the hook fires at every Claude Code session end and prints a one-line drift summary to stderr when drift or new candidates are detected. - ### Codex — Stop hook -Add under `hooks.Stop` in `.codex/hooks.json`: +The installer registers a session-start digest hook and a Stop drift hook in `.codex/hooks.json`. The Stop entry: ```json { @@ -251,7 +253,7 @@ Add under `hooks.Stop` in `.codex/hooks.json`: ### Cursor — stop hook -Add under `hooks.stop` in `.cursor/hooks.json`: +The installer registers a session-start digest hook and a stop drift hook in `.cursor/hooks.json`. The stop entry: ```json { @@ -269,7 +271,7 @@ Add under `hooks.stop` in `.cursor/hooks.json`: ### Gemini CLI — AfterAgent hook -Add under `hooks.AfterAgent` in `.gemini/settings.json`: +The installer registers a session-start digest hook and an AfterAgent drift hook in `.gemini/settings.json`. The AfterAgent entry: ```json { @@ -308,7 +310,9 @@ The cache is keyed by `project_id` (resolved from the `AI_MEMORY_PROJECT_ID` env After each approved apply, the engine reindexes the committed registry into this cache so `aim-sot consult` (and any agent using `aim-search`) can retrieve SOT entries via semantic search. -**Indexed fields**: `id`, `description`, `sot_location`, `owner`, `provenance_note`, `status`. +**Indexed fields**: `id`, `description`, `sot_location`, `owner`, `added_by`, `provenance_note`, `status`. + +**Owner fidelity**: `owner` and `added_by` handles are stored exactly as authored in the registry. The shared security scanner's HANDLE-pattern filter is exempted for SOT entries so that ownership handles (e.g. `@platform-team`) are preserved faithfully — only PII and secrets screening applies. **Not indexed here**: machine drift state (`last_verified_at`, `last_verified_sha`, `drift_status`) — that lives in 5a. diff --git a/pyproject.toml b/pyproject.toml index 615448d9..206080be 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta" [project] name = "ai-memory" -version = "2.5.0" +version = "2.6.0" description = "AI Memory Module - Vector-based memory system for AI agents" readme = "README.md" requires-python = ">=3.10" diff --git a/scripts/install.sh b/scripts/install.sh index 6ef69971..af8df001 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -109,7 +109,7 @@ EMBEDDING_PORT="${AI_MEMORY_EMBEDDING_PORT:-28080}" MONITORING_PORT="${AI_MEMORY_MONITORING_PORT:-28000}" STREAMLIT_PORT="${AI_MEMORY_STREAMLIT_PORT:-28501}" CONTAINER_PREFIX="${AI_MEMORY_CONTAINER_PREFIX:-ai-memory}" -INSTALLER_VERSION="2.5.0" +INSTALLER_VERSION="2.6.0" # Logging functions log_info() { diff --git a/src/memory/__version__.py b/src/memory/__version__.py index 81cea22b..ff89f25e 100755 --- a/src/memory/__version__.py +++ b/src/memory/__version__.py @@ -4,7 +4,7 @@ Follows PEP 440 and semantic versioning principles. """ -__version__ = "2.5.0" +__version__ = "2.6.0" __version_info__ = tuple(int(part) for part in __version__.split(".")) # Version history: diff --git a/version.txt b/version.txt index 437459cd..e70b4523 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -2.5.0 +2.6.0