Skip to content

overview.md

Latest commit

 

History

History
161 lines (124 loc) · 8.84 KB

overview.md

File metadata and controls

161 lines (124 loc) · 8.84 KB
Raw

Storage Backends — Overview

AIWG persists artifacts (memory pages, knowledge-base entries, activity log, reflections, provenance records, research corpus, sandbox identities) through a pluggable storage adapter system. By default everything lives on the local filesystem under .aiwg/. With one config file you can route any subsystem to:

  • Obsidian vault — your AIWG memory becomes searchable via the graph
  • Logseq graph — pages flow into your daily journal workflow
  • Fortemi — legacy MCP semantic-memory storage adapter (Rust + Postgres + pgvector); deprecated for index/search routing
  • A different filesystem location — heavy artifacts on a secondary drive
  • Any future backend (S3, WebDAV, Notion, AnythingLLM — tracked, not yet shipped)

Each subsystem is configured independently. You can keep the activity log on local disk for compliance while routing memory pages into Obsidian and the research corpus to a network share.

Fortemi Integration Planes

Do not conflate storage configuration with AIWG's two file-based Fortemi integrations:

Plane Configuration / command Semantics
Static index aiwg index sync, aiwg index query Rebuildable local AIWG v2 search and graph cache consumed through Fortemi Core
Portable shard conversion aiwg index export --format fortemi-shard Explicit v2 index-to-shard conversion; server compatibility is profile-scoped and requires published-package plus real server-import evidence
Live MCP persistence "type": "fortemi" in .aiwg/storage.config Reads and writes configured subsystems through a running Fortemi service

Success in one plane is not evidence for another. In particular, aiwg storage test exercises live MCP persistence only; it does not validate the static index contract or a Knowledge Shard.

Quick start

If you do nothing, AIWG works exactly like before — every subsystem persists under .aiwg/.

For private per-project memory that should not be committed, AIWG can resolve the memory subsystem from a user-level project registry under ~/.aiwg/projects/ when no project-local .aiwg directory is present. See docs/user-level-project-memory.md.

To change behavior, create .aiwg/storage.config:

{
  "$schema": "https://aiwg.io/schemas/storage.config.v1.json",
  "version": "1",
  "roots": {
    "research": "/mnt/archive/aiwg-research"
  },
  "backends": {
    "memory":      { "type": "obsidian", "vault": "~/vaults/main", "folder": "AIWG/memory" },
    "kb":          { "type": "logseq",   "graph": "~/.logseq/graphs/work" },
    "activity_log":{ "type": "fs" }
  }
}

Then verify:

aiwg storage show           # effective config + resolved paths
aiwg storage list-backends  # which adapters are READY vs STUB
aiwg storage test memory    # round-trip write/read/list/delete probe

Subsystems

AIWG defines eight subsystems. Each one can be configured independently in storage.config.

Subsystem Default location What lives here
memory .aiwg/memory/ Consumer-framework semantic memory (pages, lessons)
reflections .aiwg/reflections/ Agent-loop reflections, ralph-reflect output
kb .aiwg/kb/ Knowledge-base entity/concept pages
activity_log .aiwg/activity.log Cross-framework chronological event log
provenance .aiwg/provenance/ W3C PROV records
research .aiwg/research/ Research corpus — papers, summaries, citations
media .aiwg/media/ Media artifacts (curator framework)
sandbox_identity ~/.config/aiwg/sandbox-agents.json Persistent agent-instance identities (host-level)

Configuration shape

.aiwg/storage.config is JSON, validated against https://aiwg.io/schemas/storage.config.v1.json.

{
  "version": "1",                         // required; v1 is the only currently-supported version
  "roots": {                              // optional path overrides for fs-backed subsystems
    "research": "/mnt/archive/aiwg-research"
  },
  "backends": {                           // optional per-subsystem backend selection
    "memory":      { "type": "obsidian", "vault": "~/vaults/main" },
    "kb":          { "type": "logseq",   "graph": "~/.logseq/graphs/work" },
    "activity_log":{ "type": "fs" }
  },
  "fallback": "cache_and_warn"            // optional: cache_and_warn (default) | block
}
  • roots redirects the physical filesystem location of an fs-backed subsystem. No effect on non-fs backends.
  • backends picks the adapter type per subsystem. Subsystems not listed default to fs.
  • fallback controls behavior when an external backend is unreachable. cache_and_warn (default) queues writes under .aiwg/.storage-cache/ and continues; block refuses the write.

Every backend's required fields are documented in docs/storage/backends/<type>.md.

CLI surface

Command Purpose
aiwg storage show Print the effective config and resolved physical paths per subsystem
aiwg storage list-backends Inventory of compiled-in adapters with READY/STUB status
aiwg storage test <subsystem> Round-trip write/read/list/delete probe — proves connectivity
aiwg storage migrate <subsystem> --from … --to … Copy entries from one backend to another. See docs/storage/migration.md
aiwg doctor Validates .aiwg/storage.config (schema + credential walk + reachability probes)

Per-subsystem CLIs route through the configured adapter:

Command Subsystem
aiwg memory memory
aiwg reflections reflections
aiwg kb kb
aiwg activity-log activity_log
aiwg provenance provenance
aiwg research-store research

Each one supports path / list / get / put / delete / append-log (where applicable). See per-skill documentation for usage.

Security

The schema actively forbids credential-named properties at every nesting depth:

token, password, secret, apiKey, api_key,
accessKey, accessKeyId, secretAccessKey

Loading a storage.config containing any of these throws an error pointing at the offending path. Tokens, passwords, and API keys must come from environment variables or your OS keychain, never from storage.config.

See docs/storage/security.md for the full security model — credential handling, path traversal rejection, atomic writes, doctor validation.

What's implemented vs deferred

Backend Status Notes
fs READY Default; local filesystem
obsidian READY Direct fs writes against vault; refuses .obsidian/ config dir
logseq READY Direct fs writes against graph; YAML→property:: transform
fortemi READY (alpha, legacy for search) Routes persistence through Fortemi MCP tools; index/search uses Fortemi Core by default; use --backend local for legacy fallback
notion STUB (#959) Planned — REST + external_id upsert
anythingllm STUB (#960) Planned — multipart upload + cache mirror
s3 STUB (#962) Planned — phase 3 (bulk artifacts)
webdav STUB (#963) Planned — phase 3 (Nextcloud/ownCloud)

aiwg storage list-backends shows the live status. Declaring a STUB backend in storage.config produces a clear error citing the tracking issue.

Further reading

  • docs/storage/security.md — credential handling, path safety, doctor validation
  • docs/storage/migration.md — aiwg storage migrate walkthrough
  • docs/storage/backends/<type>.md — per-backend setup, env vars, caveats
  • docs/fortemi-core-prebuilt-indices.md — Fortemi Core prebuilt framework index packaged with npm releases
  • .aiwg/architecture/storage-design.md — full design (adapter interface, subsystem registry, phasing)
  • .aiwg/architecture/adr-configurable-storage-backends.md — decision record
  • .aiwg/architecture/schemas/storage.config.v1.json — published JSON Schema