Publish matched framework pilot: Oh within noise of Supermemory on LongMemEval-S - #191
Merged
Merged
Conversation
…ngMemEval-S The first three-arm matched comparison closed all 60 planned LongMemEval-S questions per arm: Supermemory 75.00%, Oh default retrieval 71.67% and BM25 68.33% conservative success with every cell kept in the denominator. The paired Oh-Supermemory difference of -3.33 points sits inside a 95% within-type bootstrap interval of -13.33 to +6.67, so the honest public result is parity-grade evidence, not a superiority claim. Adds the aggregate-only result JSON with freeze/plan/result digests, the result document with per-type outcomes and full cost accounting, the adapters addendum recording the executed session-level Supermemory profile and its earlier unit-level diagnostic, the homepage matched chart with protocol qualifications, the benchmark:framework-pilot-result cost surface and publication tests that keep the artifact aggregate-only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The aggregates-only check listed /Users/ as a bare string, which the public-surface scanner reads as an absolute macOS user path in the test source itself. Match the established regex-literal idiom instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
benchmarks/FRAMEWORK_PILOT_RESULT_V1.mdrecords per-type outcomes, the six failed/unattempted local cells, the four zero-candidate Supermemory searches, readiness timing (p50 61.5 min), and cost accounting ($43.34 Supermemory credits, $2.28 reader+judge).benchmarks/results/memory-framework-pilot-v1.jsonis aggregate-only: digests, counts, rates and limits; no questions, answers, raw responses, namespaces or private paths.benchmarks/FRAMEWORK_PILOT_ADAPTERS_V1.mdgains the executed session-level profile note, including the failed unit-level diagnostic it replaced.benchmark:framework-pilot-resultincosts.json; the new publication test pins the 180-cell matrix, per-arm dispositions, paired comparisons, caps and the aggregates-only rule.Test plan
bun test tests/memory-benchmark-framework-pilot-publication.test.ts— 3 passbun test site/tests/home.test.tsx— 8 passbun run check:cost-surfaces— 38 surfaces passbun run typecheck(root) andbun run typecheck(site) — cleanbun run test:benchmarks— 1775 pass; 7 environmental failures identical on an unrelated checkout (5 process-custody kills inmemory-benchmark-claude-subscription, 2 load timeouts inmemory-benchmark-evolution-v9); none touch this diffcheckon the governed head🤖 Generated with Claude Code