Recall a mention by its name vector so a short form or another script meets its entity - #877
Conversation
9e39183 to
294f1be
Compare
|
The design is right for cut 2 of 0041 decision 3: channel 2 proposes a pair with its cosine and decides nothing, the literal channel keeps its ground, a family mismatch is a skip and an unknown family is a question. The store module follows 0035's two rules and 0070's same-base keys, and the extraction path degrades to literal recall on every failure rather than failing the document. One thing blocks the merge and one is a question. Blocking: the migration number collided again. Question, not blocking: Everything else read cleanly: the |
43fafec to
4dd6d6d
Compare
… meets its entity Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: WaylandYang <the-das302@spoko.pl>
4dd6d6d to
b9dccec
Compare
|
Both taken, in b9dccec (rebased onto #886's single-binary test layout):
Also in this revision: the E2E pass (five rounds against the real server, described in the body) found that |
… are still computed Only the chat client is absent when a chunk is the contract itself; the embedding model that #877 uses for name vectors comes from the same settings row and applies to pushed names as much as extracted ones. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
… meets its entity (#877) Signed-off-by: Wayland Yang <wayland0916@gmail.com> Co-authored-by: Wayland Yang <wayland0916@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
… are still computed Only the chat client is absent when a chunk is the contract itself; the embedding model that #877 uses for name vectors comes from the same settings row and applies to pushed names as much as extracted ones. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
* Let a source push statements in the open contract instead of prose a model must read back
A statements source takes the extraction contract itself (e/s/n) on
POST /sources/{id}/statements with the api push's identity, versions and
tombstones; the payload is one chunk and extraction parses it instead of
prompting, so a pushed statement is an open statement and reaches the typed
graph only through alignment. Record 0054.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
* Keep the embed settings on the pushed-statements path so name vectors are still computed
Only the chat client is absent when a chunk is the contract itself; the
embedding model that #877 uses for name vectors comes from the same settings
row and applies to pushed names as much as extracted ones.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
* Locate a pushed statement's time words in the payload itself and refuse an unlisted subject at the door
`locate_time` begins with `quote?`, and the door requires a pushed
statement's quote to be null, so every `when` / `ended` on a pushed
statement fell into the TIME_NOT_IN_QUOTE branch: the fact landed with
only doc_time and a drop signal, and no time mention was ever recorded.
On the pushed path the chunk is the payload, so the words are looked up
in the chunk directly. The DB test now asserts the mention lands; it
fails without this change.
The door now resolves each statement's subject and each name's entity
against `e`, folding whitespace and case as `name_key` does, and answers
422 instead of letting extraction drop the item silently as UNKNOWN_REF.
An object not in `e` still lands as a literal value, as it does for a
model's reply. The route carries a body limit of four payloads so the
2 MiB default is no longer buffered before the token check; the
documented 64 KiB refusal stays a 422 with its message.
Tests added: tombstone marks and a new push revives, a tombstone for an
unknown identity is a no-op, the missing Authorization header, the
64 KiB limit over HTTP, the 200-item limit, an `n` entity not in `e`,
a non-null `n` quote, and the folded subject that must pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
* Let a statements source's push token be viewed and rotated like an api source's
Found by running the server end to end: the token came back once at
creation, then GET /token and POST /rotate-token answered 404 because
both still gated on kind == "api". One predicate, has_push_token, now
names the two kinds that carry a push token, and creation, view and
rotation use it. A test signs in as the editor, reads the token back,
rotates it, and checks that the old token is refused and the new one
pushes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
---------
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
0041 decision 3 names three recall channels; today only the first exists (a literal match of the mention's name against name facts). This lands the second: the mention's name vector against the name vectors of the base, so 海探1 finds 海洋探测器1号 and a name written in another script finds its entity (#709) without either being spelled the same.
What it does, and deliberately does not do
name_vector|<cosine>pair for the batch adjudicator, which already judges pairs from two profiles and precedent. Evidence-based deciding is 0041 cut 3 and is not touched. So this change can only add questions, not silent merges.type_familyis known and differs from the mention's is skipped. An unknown family blocks nothing: when unsure, ask.Pieces
migrations/0073_a_name_has_a_vector.sql—name_vectors(fact_id PK, kb_id, entity_id, embedding vector), one row per name fact; composite same-KB foreign keys and thekb_ownership_is_not_reassignedtrigger per 0070. A side table rather than a column onfacts(everySELECT * … Factwould have to change) or onentities(an entity has as many vectors as it has names).utopia_store::name_vectors—pending,set(registers the HNSW build withvector_index, newTarget::NameVectors),nearest(0035's two rules: casts on both sides of<=>,vector_dimspredicate; outer re-sort qualified to the CTE).resolution::resolve_mentiongainsname_vector: Option<&[f32]>; the old body isresolve_by_nameunchanged. All callers updated; mapping paths passNone.extraction_open::run_openembeds each chunk's named entities in one batch before resolving them (failure → warn, chunk falls back to literal recall), and after the document tops up name facts that have no vector yet (embed_pending_names, batched, failure → warn, never fails the document).name_vectorreason in both packs;design/identity.mdand 0041's status line updated.Verified
Against pgvector/pgvector:pg16 on a fresh database migrated through 0073:
cargo test -p utopia-store: 333 passed, 0 failed (4 new ina_similar_name_is_proposed_not_merged.rs: near name → new entity + adjudicator pair; other family → not proposed; far / no vector → nothing; literal hit → not re-proposed)cargo test -p utopia-server: 384 passed, 0 failedcargo clippy --workspace --all-targets -D warningsclean;cargo fmt --checkcleantsc,pnpm guard(99 files),pnpm test132 passedEnd-to-end, five rounds
Real server from this branch (port 1518) against a fresh Postgres, a deterministic OpenAI-compatible stub standing in for the models (character-hash embeddings, canned open-extraction replies, a batch adjudicator that answers from a per-round alias list), one new knowledge base per round, documents pushed through
/kbs/{id}/ingestand jobs left to run:name_vectorsThe first E2E pass found a bug the unit tests could not:
name_vectors::pendingordered byfacts.created_at, a column that does not exist, so the end-of-document top-up failed on every document (logged, swallowed) and channel 2 never fired. Fixed to order byid(uuid v7) in the second commit, with apending → set → pendinground-trip test. Every server warning was clean on the final pass.The harness (stub + driver) is deliberately outside the repository, per the project's rule on harnesses; it is available on request.
Not verified here, and needs a maintainer run
scripts/bench/identity.mjshas not been run: this machine has no embedding endpoint configured (llm_settings.embed_modelis null), and the bench needs one.TOP_K = 8andSIM_FLOOR = 0.60are provisional values with their reasoning in the comments; the bench should set them. Retiringrecall_keysand the generic-suffix lists once channel 2 finds what they found is cut 4, not this PR.Two small costs worth knowing: names resolved in a chunk are embedded again by the end-of-document top-up (the resolver does not learn the name fact's id; one extra embed call per document), and
nearestfetches4·krows inside the CTE to leave room for the status filters outside it.Migration number: 0073. This started as 0072 and collided with #878, which took 0072 on
devwhile this was open, exactly the case CONTRIBUTING warns about: git merges the two files without complaint and sqlx refuses to run. Renumbered and mergeddevback in; the版本号不得重复check and_sqlx_migrationson a fresh database (72 rows, max 73) both confirm. 0071 is still held by #861.utopia-cli'sCURRENT_SCHEMA_VERSIONis pinned by test to the count of files inmigrations/, so it is 72 here (70 + #878's 0072 + this 0073); its comment now describes the gap rule without quoting a number that moves.Full
cargo test --workspaceagainst Postgres, the way thebackendjob now runs it: 1042 passed, 0 failed.🤖 Generated with Claude Code