Skip to content

docs: add experimental roadmap and evidence context to vision#33

Closed
abrichr wants to merge 2 commits into
mainfrom
chore/reframe-positioning
Closed

docs: add experimental roadmap and evidence context to vision#33
abrichr wants to merge 2 commits into
mainfrom
chore/reframe-positioning

Conversation

@abrichr

@abrichr abrichr commented Mar 2, 2026

Copy link
Copy Markdown
Member

Summary

  • Add 2×2 experimental matrix (retrieval × fine-tuning) to Core Thesis in vision.md
  • Add evidence context to benchmark table: note it's an internal synthetic benchmark that validates the pipeline, not real-world performance

Why

Honest scoping of claims. The benchmark results are real but should be framed as "pipeline validated" not "production-proven." Links to openadapt-evals for ongoing WAA/OSWorld evaluation.

🤖 Generated with Claude Code

abrichr and others added 2 commits March 1, 2026 23:42
- Add 2x2 experimental matrix (retrieval × fine-tuning) to Core Thesis
- Add evidence context to benchmark table: note it's an internal synthetic
  benchmark (~3 UI elements) that validates the pipeline, not real-world
  performance. Link to openadapt-evals for ongoing WAA/OSWorld evaluation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Was showing 33-47% range which conflated preliminary (n=3) and full
(n=45) results. The validated number is 46.7%.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@abrichr

abrichr commented Mar 4, 2026

Copy link
Copy Markdown
Member Author

Closing — these changes are already on main.

The 2x2 experimental matrix and evidence context additions to vision.md were incorporated into main via PR #34 (commit 339e5d3). The branch chore/reframe-positioning has diverged (2 ahead, 7 behind main), but the actual diff (experimental roadmap table, benchmark context note) is already present in the current main version of docs/vision.md.

Content review notes (for the record):

  • The 2x2 matrix correctly references 46.7% -> 100% first-action accuracy
  • The "internal synthetic login benchmark" framing is honest and appropriate
  • The note about WAA/OSWorld evaluation being ongoing is accurate
  • Consistent with "Trajectory-Conditioned Disambiguation" framing used in the meta-package README
  • No overclaiming — well-calibrated positioning

All 4 CI checks passed (ubuntu/macos, Python 3.10/3.11).

No action needed — content is already merged.

@abrichr abrichr closed this Mar 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant