All-time range + DB integrity surfacing#3
Open
ezwep wants to merge 3 commits into
Open
Conversation
- Replace UNION-then-date-sort with two FTS5 queries that yield BM25
scores per matching session. Title hits get a 200pt boost so a query
match in the conversation title outranks a body-only match.
- Add a "Best match" sort option (default for active searches) and
preserve existing date/token/cost sorts when the user picks them.
- Pass the FTS5 snippet() output (with <mark> tags) back to the UI,
rendered as a highlighted excerpt under each card so users can see
why a session matches.
- Smarter query parser: multi-word input becomes implicit-AND with
prefix matching on each token; quoted phrases, NEAR, NOT, and column
filters still pass through to FTS5 verbatim.
- Surface invalid FTS syntax (e.g. unbalanced quotes) as an inline
status banner under the search input instead of silently returning
zero results.
- Return shape changed from list to {conversations, search} so the UI
can show match counts and errors. Backward-compatible array fallback
in the JS callsite.
When a single project is selected, the main panel now defaults to a
heuristic project digest that answers "what have I been working on
here." The existing usage dashboard is preserved as a peer sub-view
behind a small toggle.
Backend (dashboard_data.py):
- New get_project_digest() sources entirely from the conversations
table (one row per session) so it stays fast even on a 53k-message
index — measured ~12ms on the largest project (612 sessions).
- Computes a momentum summary: total sessions, active days, tokens,
cost, sessions in last 7d vs prior 7d, plus a trend label
("picking up" / "steady" / "winding down" / "new" / "dormant").
- 42-day daily activity timeline with session count, tokens, cost.
- Heuristic topic extraction: tokenizes titles + excerpts of the most
recent ~60 sessions, filters a curated stopword set (English + Dutch
+ dev/Claude-Code jargon), returns top 10 terms by frequency.
- Returns {unavailable: true} for "all" / null scopes so the UI can
fall back to the existing dashboard view cleanly.
API (app.py):
- New ConversationAPI.get_project_digest() exposed to the webview
frontend, mirroring the get_dashboard_payload pattern.
Frontend (templates/index.html):
- New state.mainSubView ('digest' | 'overview') persisted to
localStorage. When a single project is selected the digest is the
default; "All projects" forces the overview.
- renderMainPanel routes to renderProjectDigest when in dashboard view,
a real project is selected, and the sub-view is 'digest'.
- renderProjectDigest builds a hero card (trend chip + 6 headline
metrics), an activity timeline bar strip, a topic chip cloud sized by
frequency, and a clickable recent-sessions list that opens the
existing session detail view.
- All user-derived strings escaped via existing escHtml(); no new
dependencies; no indexer or schema changes; fully offline.
Risks addressed:
- state.view regression: every existing assignment to state.view in
the project click handler, escape handler, and overview button click
still routes correctly; the dashboard remains the fallback for "all".
- Performance: digest only reads conversations (not messages); topic
extraction capped at 60 sessions and 10 terms.
- XSS: title/excerpt/topic content all interpolated through escHtml().
Three related improvements bundled together so the dashboard keeps
working when the index is broken and can show the full history when
"last 90 days" is too narrow.
1. **All-time range option**
- dashboard_data.py: new _earliest_indexed_day() helper plus an
"all" branch in _resolve_range() that anchors start_day to the
oldest assistant-message timestamp (scoped to the selected
project). End day stays "tomorrow". Label reports the actual
span in days.
- templates/index.html: extra <option value="all"> on the range
select; loadUiState() accepts it as a persisted value.
- Performance: measured ~600 ms for 1812 sessions across 97 days
on the live index — well within the existing rollup budget.
2. **WAL + synchronous=NORMAL + integrity check** (indexer.py)
- PRAGMA synchronous=NORMAL pairs with the already-enabled WAL to
keep durability while reducing fsync churn.
- New check_db_integrity() runs PRAGMA integrity_check and
returns (ok, message). Called once at startup from
ConversationAPI._ensure_db.
- Without this the corrupt-DB event from today silently produced
tracebacks in stderr with zero UI signal.
3. **UI surfacing of DB status** (app.py + templates/index.html)
- ConversationAPI now caches {ok, message, checked_at} from the
initial integrity check and exposes it in get_stats().
- New recheck_db_integrity() method for on-demand verification.
- get_stats() wraps its query batch in try/except sqlite3.DatabaseError
so a corrupt DB no longer crashes the bridge — it updates the
status and returns zeroed counters.
- Frontend adds a red banner above the statusbar when integrity
fails, with Recheck and Rebuild-index buttons. The statusbar
gains a "DB: ok / DB: corrupt" indicator that's green when fine.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bundles three related improvements so the dashboard keeps working when the index is broken and can show the full history when 90 days is too narrow.
1. All-time range option (your concrete request)
2. WAL + synchronous=NORMAL + integrity check
3. UI surfacing of DB status
Test plan