feat(cli): cross-platform portability, opt-in model fetch, 512-token embedding window#53
Merged
Merged
Conversation
5a1d0fd to
9274362
Compare
…embedding window Portability (Windows / macOS / Linux): - HOME -> dirs::home_dir() everywhere (shipping + tests); $HOME is unset on Windows, which silently disabled dense + the lexicon store there. - Per-user caches via dirs::cache_dir() (%LOCALAPPDATA% / ~/Library/Caches / ~/.cache) with a non-destructive legacy ~/.cache fallback (prefer_native). - java_resolver infers packages via Path::components() instead of matching "/java/" in the stringified path (the old Windows branch matched a literal that never occurs). Tooling: - Opt-in `aden model fetch` (model-fetch feature): streamed, size-capped, sha256-verified, https_only download of the bge model; compiled out by default (network-free core). Dense / retrieval: - MAX_SEQ 128 -> 512 so full prose sections embed (CLS pooling, single forward). A Pro Git A/B showed a modest prose-retrieval lift over 128; a whole-doc mean-pool variant trialed on top was net-negative and reverted (only the cap ships). - query provenance: via_edge_types, inferred (embedding-derived edges only), node confidence. - Confidence-gated ask: pull budget toward base on ambiguous routing, with an honest note. - Index/embedding versions bumped so stale vectors re-embed. Tests: - cargo test --workspace and --features dense,model-fetch green; clippy clean (no new warnings). - Unit tests: prefer_native, infer_java_package (5 cases), model hashing; mcp_flag_parity exempts the `model` command under model-fetch. Signed-off-by: RioPlay (Ernest Hamblen) <rioplay@rioplay.dev>
9274362 to
e881ce5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cross-platform portability fixes, an opt-in
aden model fetch, and a small set ofadditive/feature-gated index + provenance changes — including raising the dense embedding
window to 512 tokens, which a Pro Git A/B shows modestly improves prose retrieval. Everything
new is gated or additive, so the default build and behavior are unchanged. Two features that
an earlier revision of this branch carried were tested and removed (see Notes): prose↔prose
SimilarToedges (produced no traversable edges) and whole-doc mean-pooling (net-negative forretrieval).
Portability (Windows / macOS / Linux)
std::env::var("HOME")→dirs::home_dir()(shipping + tests);$HOMEis unset onWindows, which silently disabled dense + the lexicon store there.
dirs::cache_dir()(%LOCALAPPDATA%/~/Library/Caches/~/.cache)with a non-destructive legacy
~/.cachefallback (prefer_native).java_resolverinfers packages viaPath::components()instead of matching/java//\java\in the stringified path (the old Windows branch matched a literal that never occurs).Tooling
aden model fetch(opt-inmodel-fetchfeature) streams the bge model to disk through abounded reader with incremental sha256 verification (checksum gates before the atomic
rename),
https_only. Compiled out by default; the core stays network-free.Dense embedding window: MAX_SEQ 128 → 512 (validated)
The old 128-token cap truncated long prose to its first ~80 words. 512 lets a full section
embed in one forward (CLS pooling).
EMBED_PARAM_VERSION/index version bumped so stalevectors re-embed.
Benchmark (Pro Git, 20 labeled prose queries, hybrid):
Modest but consistent lift (R@5 +0.05, R@10 +0.10, MRR +0.025) over 128; dense ≫ BM25 on
prose throughout. N=20 so individual deltas are within noise; the direction is consistent.
Additive provenance (additive JSON; no default-behavior change)
queryoutput carriesvia_edge_types,inferred(embedding-derived edges only, viaEdgeType::is_inferred), and nodeconfidence.ask: on ambiguous routing, pull the budget back toward base and print anhonest note (now distinguishes a genuine near-tie from a deliberate overview pick).
Testing
cargo test --workspacegreen;cargo clippy --workspaceand--features dense,model-fetchclean (no new warnings);
aden ci-check .passes.prefer_native,infer_java_package(5 cases), model hashing.mcp_flag_parityexempts the
modelcommand under--features model-fetch.Notes — what was tested and removed
net-negative on every metric (e.g. MRR 0.385 with pooling vs 0.435 without, at 512) —
mean-pooling a long chapter dilutes the answering section. Reverted; only the cap ships.
SimilarToedges: produced zero traversable edges on a real corpus(derived over the search-index anchor namespace, which doesn't match the graph's, so they
dangle as orphans), and used an absolute cosine gate rather than the mutual-kNN + whitening
the design specifies. Removed; returns as a sequenced follow-up (index-split → whitening →
mutual-kNN), gated on the eval harness +
.aden/storesign-off. Thesimilar_pairsprimitive (with tests) is retained for that work.
License checklist (please confirm)
cargo denyCI greendirs(MIT OR Apache-2.0),sha2(MIT OR Apache-2.0, already default viaaden-paths) — AGPL-compatible
Maintainer (BDFL) change: the commit carries the DCO
Signed-off-by; the contributor CLAacknowledgement does not apply to the Maintainer (CLA Section 1).