Skip to content

docs: add experimental roadmap and evidence context to vision #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
abrichr wants to merge 2 commits into from
Closed

docs: add experimental roadmap and evidence context to vision #33

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1aa4d46
docs: add experimental roadmap and evidence context to vision
abrichr Mar 2, 2026
c04ac10
fix: use 46.7% consistently in 2x2 matrix
abrichr Mar 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/vision.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,15 @@ because:
2. Specialization compounds: the model learns *your* apps, *your* patterns
3. The workflow structure is known — the model just navigates it

**The experimental roadmap** (see [research thesis](research_thesis.md)):

| | No Retrieval | With Retrieval |
|---|---|---|
| **No Fine-tuning** | 46.7% (baseline) | **100%** (validated) |
| **Fine-tuning** | Standard SFT (baseline) | **Demo-conditioned FT** (unique value) |

Phase 2 (retrieval-only) is validated. Phase 3 (demo-conditioned fine-tuning — training models to *use* demonstrations they haven't seen) is the core planned work.

## Architecture

OpenAdapt treats workflows as **state machines**, not pixel sequences:
Expand All @@ -37,12 +46,16 @@ abstract states in visual reality.

## Why Specialization Wins

Results on an internal synthetic login benchmark (~3 UI elements, ~20-30 training examples):

| Model | Action Accuracy | Click Hit Rate |
|-------|-----------------|----------------|
| Qwen 2B Fine-tuned | 42.9% | 100% |
| Claude Sonnet 4.5 | 11.2% | 0% |
| GPT-5.1 | 23.2% | 66.7% |

> **Note**: These results validate that the training pipeline works and that specialization provides signal. They do not yet represent real-world performance — evaluation on standard benchmarks (WAA, OSWorld) is ongoing via [openadapt-evals](https://github.com/OpenAdaptAI/openadapt-evals).

General-purpose models must infer workflow structure from scratch on every query.
OpenAdapt agents *know* the structure — they just navigate it.

Expand Down