feat(observe): read how often a fact was restated across superseding keys - #165
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…keys `recordRevisions` reports how often one record key was rewritten. This profile does not correct a fact by rewriting its key: it writes a new observation whose `supersedes` names the older one. So every key in a correction chain is typically written once and reports a revision count of zero, while the fact behind them was restated repeatedly. The revision count was exact and cheap and simply could not see this, which the store specification already said and this closes. `ohObservationSupersessionV1` follows `supersedes` from one observation toward the oldest record. It asks a store only for `get`, so a snapshot or change-feed reader answers it without local SQL, and it never writes. The honesty condition is the same one the revision read needed. `resolved` is true only when the walk ends at a record superseding nothing, and only then are `depth` and `origin` exact. A cycle, the 8192-link bound, or an absent record reports `resolved: false`, names the missing key, and leaves `depth` a floor and `origin` the oldest key that could be read. A partial answer must not be shaped like a complete one; that was the defect independent review found in the feed reader, and it is the same trap here. Loop detection, the chain bound, and the treatment of a damaged chain as unusable rather than repairable follow `reachesExcluded`, which already walks these links for the supersession policy. No meaning is attached. A fact restated many times may be contested, progressively refined, or simply discussed often, and the profile does not distinguish those; whether a depth warrants review stays the application's decision. Additive and read-only: no new record kind, identifier grammar, migration, cost surface, or change to the observation value. Verified against a chain built by the production ingestion path rather than by hand — the existing supersession test's three linked observations read depth 2 with the oldest key as origin, while each key's own revision count is zero. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ur claims
Independent review found the walk gating only on the observation *value*, so a
record stored at an `edition:obs-` key under another kind was counted as a link
and the result was stamped `resolved: true` — the one value documented as exact.
Confirmed against a real store: a `kind: "entity"` record in the middle of a
three-link chain read `depth: 2, resolved: true` while every other reader in the
file rejected it. The walk now goes through `parseOhObservationRecordV1`, as
`applySupersessionPolicyV1` already does, and reports the record as damage.
Fixing that exposed two of my own test fakes as ill-formed: they carried empty
dependencies, which is not a well-formed observation record. They now list the
receipt, the cited source and the superseded key, sorted, so the damaged-chain
cases exercise real records instead of ones the stricter gate would reject
anyway. A case for the foreign-kind record is added.
Four claims were wrong and are corrected rather than softened:
- The read port does not admit "a snapshot or libSQL reader". `OhStoreV1` is
async and has no `get`; a snapshot is a record array. The type states the
requirement, and today only `OhSqliteStore` and the SDK store satisfy it.
- The two signals are not independent. `applySupersessionPolicyV1` re-puts a
record when its link changes, so a key linked by that path reads one revision.
Only the `observeOhV1({ supersession: true })` path leaves each key at zero.
- `resolved: true` does not mean the chain is complete. A link comes from a
facet-and-speaker match over a bounded lookup, so a fact whose facet changed
between sessions starts a fresh chain and resolves at a lower depth. `depth`
describes the link graph, not the number of restatements, and the naming now
says so.
- The bound is 8192 records read, so the longest resolvable chain is 8191 links.
Also documented: `depth` counts one link past the last readable record when
unresolved, `missing === key` is the case where nothing was read, and a cycle
closing beyond the bound reports `truncated` rather than `loop`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
186e069 to
29bf338
Compare
|
Independent review found one real defect and four wrong claims of mine. All fixed in The defect is worth naming precisely. The walk gated on the observation value and never on That fix then rejected two of my own test fakes, which carried empty Four claims corrected rather than softened. The read port does not admit a snapshot or libSQL reader — The third correction changed what this feature claims. It no longer reports how often a fact was restated; it reports how far the recorded link chain reaches. Those differ whenever the linker's heuristic missed, and the earlier framing would have had a consumer read a heuristic's output as a count of events. That is the same failure this whole line of work keeps finding, now three times over: a number that is wrong in a way nothing downstream can detect. The partial-feed read in #164, the unjudged lexical proxy scored against a release threshold, and this. Each was individually plausible and each produced an answer shaped exactly like a correct one. Merging with the limitation on the record. |
…urated #165 shipped with a gap stated in its own merge note: `resolved: true` reports that the walk reached a record superseding nothing, not that the store's link graph is complete. A link is chosen from a keyword lookup bounded at 100 candidates, and when that lookup saturates the true predecessor may never have been examined, so the new record starts a fresh chain. The read then returned `resolved: true` with a short depth and no signal at all. The store already records this. Every observation's receipt lists the session's observations whose candidate lookup hit the bound, so the read now checks the receipt of each record it walks and reports `candidatesTruncated`. Receipts are memoised, so a chain drawn from one session reads its receipt once. What the flag does and does not promise is stated in both the doc comment and the specification. True means a record on the walk had its lookup saturate, so the chain may be short by an unknown amount and `depth` is not a floor on the links that exist. False means no record on the walk was affected. It is not a promise that the link graph is right: a facet that changed between sessions starts a fresh chain and leaves no trace for any reader to find, which is a limit of the linker rather than something this read can recover. Tested with a receipt that names one of two linked observations, so the flag is true reading from the named record and false reading from the other. Adding the field without a case that drives it true would repeat the defect review found in hraness/sponge#303, where new behaviour passed every gate while nothing pinned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch was successfully deployed
1 participant
Completes the pair.
#164shippedrecordRevisions, which reports how often one record key was rewritten, and documented a limitation it could not fix: this profile does not correct a fact by rewriting its key.applySupersessionPolicyV1writes a new observation whosesupersedesnames the older one.The consequence is that the signal shipped in
#164is blind to exactly the churn this profile produces. Every key in a correction chain is typically written once, so each reportsrevisions: 0, while the fact behind them was restated repeatedly. The two numbers measure different things and neither subsumes the other.What this adds
ohObservationSupersessionV1(store, key)followssupersedestoward the oldest record and reportsdepth,origin,resolved,loop,missing,truncated.It asks a store for
getand nothing else, through a narrowOhObservationReadStoreV1port, so a snapshot or change-feed reader answers it without local SQL and the read cannot commit.The honesty condition, again
resolvedis true only when the walk ends at a record superseding nothing, and only then aredepthandoriginexact. A cycle, the 8192-link bound, or an absent record reportsresolved: false, names the missing key, and leavesdeptha floor withoriginas merely the oldest readable key.This is deliberate and it is the lesson from
#164's review: the defect found there was a reader whose partial answer was shaped identically to a complete one, so no consumer could discount it. A supersession walk has three ways to stop early, so the same trap was available here and is closed by construction rather than by documentation.Loop detection, the chain bound, and treating a damaged chain as unusable rather than repairable follow
reachesExcluded, which already walks these links for the supersession policy. No second convention.No meaning attached
A fact restated many times may be contested, progressively refined, or simply discussed often, and this does not distinguish them. Whether a depth warrants review stays the application's decision, matching the split
#164established andAGENTS.mdrequires.Worth stating because the temptation is real: a naive reading would rank high-depth facts as low-confidence. The measurements behind this work argue against exactly that move — a model's coarse self-reported confidence did not predict its own errors (1.21x and 1.03x lift, intervals spanning 1.0), and a third party's published
highlabel failed to predict even its own revision (8.8% vs 9.7%, n=14,560). Churn is worth reporting because it needs no external answer key; turning it into a trust score is the step that has repeatedly failed to measure.Verification
Tested against a chain built by the production ingestion path, not by hand: the existing supersession test's three linked observations read
depth: 2with the oldest key asorigin, and each key's own revision count is zero — which is the whole point. Damaged chains are covered by a narrow fake store exercising an absent link, an absent start key, a cycle, the chain bound, and three invalid keys.bun run typecheckclean;bun test ./src/observe.test.ts ./src/store.test.ts ./src/sqlite/store.test.ts72 pass, 0 fail;check:cost-surfaces21 surfaces;check:effectpasses;site/tests/source.test.ts15 pass;bun run buildbyte-reproducible across two runs with no untracked files;spec/v1/observation.mdand itssite/publicmirror confirmed identical bydiff.Additive and read-only. No new record kind, identifier grammar, migration, cost surface, or change to the observation value or the
OhStoreV1port.🤖 Generated with Claude Code
Second pass: independent review
One real defect and four wrong claims, fixed in the pushed head.
The walk gated only on the observation value, never on
record.kind. A record stored at anedition:obs-key under another kind was counted as a chain link and the result stampedresolved: true— the one field documented as exact. Reproduced against a real store: akind: "entity"record mid-chain readdepth: 2, resolved: truewhile every other reader in the file rejected it. It now goes throughparseOhObservationRecordV1, asapplySupersessionPolicyV1already does, and reports the record as damage.That fix immediately rejected two of my own test fakes, which carried empty
dependenciesand were not well-formed observation records. They now list the receipt, the cited source and the superseded key, sorted.Four claims were wrong and are corrected, not softened:
OhStoreV1is async with noget; a snapshot is a record array. OnlyOhSqliteStoreand the SDK store qualify, so the port widens nothing yet.revisions: 0"observeOhV1({ supersession: true })path.applySupersessionPolicyV1re-puts a record when its link changes, so keys linked that way read1. The signals overlap.resolved: trueimplies a complete chainBecause of the third correction, this no longer claims to count restatements.
depthdescribes the link graph the store holds, and the doc and spec now say that directly. Also documented:depthcounts one link past the last readable record when unresolved,missing === keyis the case where nothing was read, and a cycle closing beyond the bound reportstruncatedrather thanloop.The reviewer confirmed by dumping every table before and after eight calls including throwing ones that the read writes nothing, found no bypass of key validation across 48 generated keys plus 13 hostile ones, and confirmed the spec mirror byte-identical.
Rebased onto current
main;dist/rebuilt after the rebase.typecheckclean,observe/store/sqlite-store73 pass,check:cost-surfaces21 surfaces,check:effectpasses,site/tests/source.test.ts15 pass.