Skip to content

Commit d3754df

Browse files
authored
Merge branch 'main' into fix/dockerfile-dev-venv-volume
2 parents 356894f + 5db4858 commit d3754df

167 files changed

Lines changed: 14507 additions & 3319 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.env.example

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,14 @@ GITHUB_TOKEN=
7878
# OpenAlex email for faster rate limits (paper search works without it)
7979
# OPENALEX_EMAIL=you@example.com
8080

81+
# ═══════════════════════════════════════════════════════════
82+
# PROJECT WORKSPACES
83+
# ═══════════════════════════════════════════════════════════
84+
85+
# Where project workspaces are stored (bind mount in production)
86+
# Docker Compose dev uses a named volume; production uses this path.
87+
# OPENMLR_WORKSPACES_PATH=./.workspaces
88+
8189
# ═══════════════════════════════════════════════════════════
8290
# SANDBOX / CODE EXECUTION
8391
# ═══════════════════════════════════════════════════════════

.github/workflows/docs.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ on:
55
branches: [main]
66
paths:
77
- "site/**"
8+
- "CHANGELOG.md"
89
- ".github/workflows/docs.yml"
910
workflow_dispatch:
1011

@@ -31,6 +32,11 @@ jobs:
3132
run: npm ci
3233
working-directory: site
3334

35+
- name: Sync changelog into docs
36+
run: |
37+
printf '%s\n' '---' 'title: Changelog - OpenMLR' 'description: OpenMLR version history and release notes.' '---' '' > site/docs/changelog.md
38+
cat CHANGELOG.md >> site/docs/changelog.md
39+
3440
- name: Build docs
3541
run: npx vitepress build docs
3642
working-directory: site

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ Thumbs.db
1010
*.swo
1111
*~
1212

13+
.pnpm-store/
14+
1315
# Environment
1416
.env
1517
.env.local
@@ -45,6 +47,7 @@ backend/openmlr/db/migrations/versions/*.py
4547
site/node_modules/
4648
site/docs/.vitepress/dist/
4749
site/docs/.vitepress/cache/
50+
site/docs/changelog.md
4851

4952
# Misc
5053
tmp/

.pre-commit-config.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ repos:
2020
hooks:
2121
- id: eslint
2222
name: eslint (frontend)
23-
entry: bash -c 'cd frontend && npx eslint --max-warnings 50 "$@"' --
23+
entry: bash -c 'args=(); for f; do args+=("${f#frontend/}"); done; cd frontend && npx eslint --max-warnings 50 "${args[@]}"' --
2424
language: system
2525
files: ^frontend/src/.*\.(ts|tsx)$
2626
types: [file]

CHANGELOG.md

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
# Changelog
2+
3+
## v0.5.0
4+
5+
Project-scoped conversations, unified file workspace, Monaco code viewer, TODO approval flow, comprehensive agent guidance, and test infrastructure improvements.
6+
7+
### Project-Scoped Conversations
8+
- **Mandatory projects** -- Every conversation must belong to a project; the "All Conversations" concept has been removed entirely
9+
- **Onboarding project creation** -- 3-step onboarding flow: Providers > Model > Create First Project. Users name their project and understand the concept from the start
10+
- **Orphan cleanup** -- Existing conversations without a project are automatically deleted when the conversation list is loaded
11+
- **Project selector** -- No more null state; always shows a real project. Dropdown removed the "All" option
12+
13+
### Unified File Workspace
14+
- **Resources materialized as files** -- PLAN.md, completion reports, and paper drafts are now written as real files to the project workspace (`.project-meta/plans/`, `.project-meta/reports/`, `papers/`) and appear in the Files panel
15+
- **Resources panel removed** -- The separate Resources section in the right panel has been replaced by the unified FileTree. Tasks tab renamed to Todos and displayed alongside Files as stacked collapsible panels (no more tab buttons)
16+
- **workspace_files_changed SSE event** -- FileTree auto-refreshes when resources are written to the workspace
17+
- **File badges** -- Pin icon for PLAN.md, clipboard icon for reports, book icon for paper drafts in the FileTree
18+
19+
### Monaco Code Viewer
20+
- **Agent/Editor tabs** -- Main content area now has two tabs: Agent (chat) and Editor (read-only code viewer)
21+
- **Monaco Editor integration** -- VS Code engine (`@monaco-editor/react`) with full syntax highlighting, minimap, line numbers, bracket pair colorization, and word wrap. Supports 15+ languages (Python, TypeScript, JSON, Markdown, YAML, LaTeX, shell, etc.)
22+
- **Multi-tab file viewer** -- Open multiple files as tabs in the Editor, close individually. Closing the last tab switches back to the Agent tab
23+
- **FileTree integration** -- Clicking a file in the Files panel opens it in the Editor tab; inline preview removed
24+
25+
### Agent Improvements
26+
- **Message persistence fix** -- Assistant text that precedes tool calls is now persisted to the database via an `assistant_message` event, surviving page refreshes
27+
- **Plan-mode research budget** -- Warns after 5+ research tool calls in Plan mode; guides the agent to save comprehensive research for Execute mode tasks
28+
- **TODO approval in Execute mode** -- Creating or adding tasks in Execute mode requires user approval via a dedicated review UI with current vs. proposed diff view and inline task editing
29+
- **Blank section enforcement** -- Writing tool warns about incomplete sections after each write; `get_draft` appends a WARNING listing all `[Not yet written]` placeholders
30+
31+
### Workspace Targeting
32+
- **Project workspace auto-targeting** -- `read`, `write`, `edit`, and `bash` tools automatically target the active project workspace directory. Files created by the agent appear in the Files panel
33+
- **Workspace context wiring** -- Both inline and Celery worker sessions resolve the project workspace and set context for workspace tools (knowledge graph, notes) and local tools (file operations)
34+
35+
### Comprehensive Agent Guidance
36+
- **System prompt v7** -- Full rewrite with a Tool Selection Decision Tree, Workspace Structure reference, expanded Code Execution constraints, and strengthened Plan/Execute mode rules
37+
- **Expanded tool descriptions** -- `bash` (Docker constraints, common patterns), `read` (offset/limit usage), `write` (workspace targeting, anti-pattern for papers), `edit` (find-and-replace semantics), `research` (available tools, iteration limits), `writing` (full workflow, placeholder enforcement), `plan_tool` (enforcement rules, workspace file locations)
38+
- **Research sub-agent prompt** -- Now lists available tools, constraints (60 iterations, ~190k token budget), stop conditions, error handling guidance, and clarifies that the sub-agent cannot write to the workspace
39+
40+
### Testing & Infrastructure
41+
- **Pytest hang fix** -- Removed deprecated `event_loop` session fixture, added `_dispose_engine_at_exit` for async engine cleanup, set `asyncio_mode = "auto"` in pytest config. Full backend suite now exits cleanly in ~50s
42+
- **47 new backend tests** -- Message persistence, research budget, project workspace targeting, incomplete section warnings, orphan cleanup, workspace path resolution, TODO approval Redis helpers, report persistence
43+
- **22 new frontend tests** -- EditorPanel (12 tests), FileTree (10 tests)
44+
- **ProjectSelector tests** -- 12 tests verifying no-null project selection
45+
- **TodoReviewDrawer tests** -- 10 tests for the approval UI
46+
- **Total: 858 backend + 226 frontend = 1,084 tests**
47+
48+
### Bug Fixes
49+
- Fixed silent exception swallowing in `_wire_persistence` (now logs errors via `logger.exception`)
50+
- Fixed inline-mode SSE events missing `conversation_uuid` (prevented cross-conversation event leakage)
51+
- Fixed `pnpm-lock.yaml` not including `@monaco-editor/react` for Docker builds
52+
- Fixed `test_list_conversations` and `test_conversations_isolated_by_user` for project-scoped queries
53+
- Fixed `test_conversations.py` integration tests to create conversations with a project
54+
55+
---
56+
57+
## v0.4.0
58+
59+
Projects, workspaces, interactive terminal, multi-provider model picker, security hardening, and centralized version management.
60+
61+
### Projects & Workspaces
62+
- **Project management** — Create, rename, archive, and manage projects via the sidebar or Settings. Each project gets a persistent workspace directory with standard subdirectories (`code/`, `data/`, `papers/`, `research/`, etc.)
63+
- **Default project** — Every user gets an auto-created "All Conversations" project; conversations can be moved between projects
64+
- **File browser** — Interactive file tree panel for browsing project workspace files, with lazy-loading directories, file-type icons, inline text preview, and file sizes on hover
65+
- **File operations API** — Browse, read, write, delete, and upload files within project workspaces via REST endpoints
66+
- **Knowledge graph** — Per-project persistent knowledge graph (backed by `networkx`) for cross-conversation memory. Supports 10 entity types (paper, concept, method, dataset, etc.) and 13 relationship types (cites, implements, extends, etc.). Context is injected into new conversations automatically
67+
- **Workspace persistence** — Saves search results, research notes, parsed papers, tool failure logs, compute probes, experiment logs, and cross-conversation state to the project workspace
68+
- **Workspace agent tools** — New `workspace` tool with 8 operations: `status`, `search`, `note`, `knowledge_add`, `knowledge_relate`, `knowledge_query`, `knowledge_summary`, and `recent_failures`
69+
- **Project manage modal** — Bulk project management UI with inline rename, conversation counts, and delete with confirmation
70+
71+
### Interactive Terminal
72+
- **WebSocket PTY terminal** — Full interactive bash shell in the browser via xterm.js, connected to a real server-side PTY process scoped to the project workspace
73+
- **Environment scrubbing** — Terminal sessions receive a minimal allowlisted environment; server secrets (API keys, DATABASE_URL, JWT_SECRET_KEY, etc.) are never exposed
74+
- **Terminal UI** — Maximize/minimize toggle, connection status indicator, manual reconnect, 5,000-line scrollback, JetBrains Mono font
75+
76+
### Model Picker & Providers
77+
- **Multi-provider model catalog** — Browse models from OpenAI, Anthropic, OpenRouter, OpenCode Go, Ollama, and LM Studio in a unified picker. Live model list fetched from `models.dev` with hardcoded fallbacks
78+
- **Custom providers** — Register arbitrary OpenAI-compatible, Anthropic SDK, OpenRouter, or LiteLLM endpoints with custom base URLs and API keys. Fetch and cache model lists from custom endpoints
79+
- **Model picker UI** — Two-tab modal with search/filter, provider logos, recently used models (up to 5), and a custom model ID input for arbitrary model strings
80+
- **Provider settings** — Tabbed settings page (Models, Search, Papers, Compute, Others) with per-provider status indicators, inline API key management, and an "Add Custom Provider" modal
81+
- **Automatic SDK routing** — Models are routed to the correct SDK (OpenAI or Anthropic) based on provider and model name. Custom providers specify their SDK type explicitly
82+
- **Recently used models** — Last 5 used models tracked per-user and shown at the top of the picker
83+
84+
### Security
85+
- **Path traversal prevention** — All file operations use `Path.relative_to()` containment checking; symlinks pointing outside the workspace are blocked for both reads and deletes
86+
- **Resource limits** — File uploads capped at 100 MB, writes at 10 MB, reads truncated at 500 KB; knowledge graph limited to 10,000 nodes / 50,000 edges; terminal input capped at 4 KB per message
87+
- **Process isolation** — Terminal shells spawned with `start_new_session=True`, `close_fds=True`, `--norc --noprofile`; zombie prevention via SIGTERM/SIGKILL escalation with proper `waitpid` reaping
88+
- **Protected directories** — Top-level workspace directories (`code`, `data`, `papers`, etc.) cannot be deleted via the API
89+
- **Config allowlist**`POST /api/config` only accepts a hardcoded list of 10 environment variable names
90+
91+
### Version Management
92+
- **Single source of truth** — New `VERSION` file at repo root; all version references derive from it
93+
- **Automatic version bumping**`make version-patch`, `make version-minor`, `make version-major` commands bump and propagate across the entire monorepo
94+
- **Explicit version setting**`make version-set V=X.Y.Z` for arbitrary version changes
95+
- **Version display** — Current version shown in the webapp sidebar footer and docs site footer
96+
97+
### Bug Fixes
98+
- Fixed `test_list_projects_api` failing due to auto-created default project not being accounted for
99+
- Fixed `KeyManager` path to use `/app/.keys` instead of `/.keys`
100+
- Fixed dev entrypoint to always sync dependencies on startup
101+
- Fixed eslint pre-commit hook path resolution for new files
102+
- Added missing `python-multipart` dependency for file upload endpoint
103+
104+
### Internal
105+
- 65 new tests for projects, workspaces, knowledge graph, and workspace tools (740 backend tests, 186 frontend tests total)
106+
- Unified version strings: removed hardcoded versions from `app.py`, `health.py`, `test_app.py`
107+
- Changelog moved to repo root; synced into docs site at build time with VitePress frontmatter
108+
- Pydantic request/response models for all API endpoints (`backend/openmlr/models.py`)
109+
110+
---
111+
112+
## v0.3.0
113+
114+
Compute environments, UI improvements, and bug fixes.
115+
116+
### Compute Environments
117+
- **Multi-backend compute** — Execute code on local Docker, SSH remotes, or Modal cloud
118+
- **SSH key management** — Generate Ed25519/RSA keys, upload existing keys via Settings > Compute
119+
- **Compute probing** — Detect OS, GPUs (with CUDA version), Python versions, and disk space
120+
- **Compute selection** — Switch between configured compute nodes mid-conversation
121+
- **Connection pooling** — SSH connections are reused across tool calls for performance
122+
- **Docker-in-Docker detection** — Worker container executes commands directly when already in Docker
123+
124+
### UI Improvements
125+
- **Collapsible Tasks & Resources** — Click section headers in the right panel to collapse/expand
126+
- **Fixed right panel layout** — Right panel no longer causes page scroll issues when toggled
127+
- **Improved scroll behavior** — Message list scrolls correctly without affecting page layout
128+
129+
### Bug Fixes
130+
- Fixed `scrollIntoView` causing entire page to scroll when RightPanel is open
131+
- Fixed `_get_draft` database call using wrong function name
132+
- Fixed test suite failures related to async database mocking
133+
134+
### Internal
135+
- Added `_running_in_container()` detection for Docker Compose worker environments
136+
- Improved test coverage for compute and writing tools
137+
- Updated Settings nav to reflect current menu structure
138+
139+
---
140+
141+
## v0.2.0
142+
143+
Major rewrite of the mode system, paper writing, processing architecture, and UI routing.
144+
145+
### Mode System
146+
- Simplified from Plan/Research/Write to **Plan + Execute** (two modes)
147+
- Plan mode: ask questions, gather context, create plans. No execution.
148+
- Execute mode: all tools available. Follow the plan.
149+
- Toggle with P/E button, `Cmd+B` (Plan), or `Cmd+E` (Execute)
150+
- Amber border for Plan messages, blue border for Execute messages
151+
- Three-layer mode enforcement: system prompt, tool filtering, runtime blocking
152+
153+
### Paper Writing
154+
- Writing tool with **auto-save to database** — survives across workers and restarts
155+
- Paper preview in the **Paper tab** in the UI
156+
- Client-side export to Markdown and LaTeX
157+
- Outline, sections, and bibliography managed as structured data
158+
159+
### Processing Architecture
160+
- **Per-conversation processing state** — multiple conversations run in parallel
161+
- **Background jobs via Celery + Redis** — close the browser, come back later
162+
- **Redis-based interrupt relay** — actually kills running worker tasks, not just a flag check
163+
- **Sub-agent streaming** — research tool shows nested tool calls in real-time
164+
165+
### Settings & UI
166+
- **Settings as routed pages** (`/settings/providers`, `/settings/agent`, `/settings/sandbox`, `/settings/writing`) — no longer a modal
167+
- **Sticky model selection** — persisted per-user in the database
168+
- **Onboarding flow** — guided setup when no LLM provider is configured
169+
- **Route restructure** — app served from `/` instead of `/app`
170+
- **SSE reconnection catch-up** — missed events replayed on reconnect
171+
- **PLAN.md auto-generated** and pinned in resources panel
172+
173+
### Testing & CI
174+
- **149 backend tests + 29 frontend tests** — comprehensive coverage
175+
- **GitHub CI** — tests run on push and pull request
176+
- `make test` runs all tests (backend + frontend + docs build)
177+
- `make test-backend`, `make test-frontend`, `make test-docs` for targeted runs

0 commit comments

Comments
 (0)