Greptile rerun and the maintainer review both raised the same concern:
state.db and the sidecar self.messages can diverge by hundreds of
messages during recovery / mid-flight writes / pending_user_message.
On the test corpus:
0db167553ac7 db=11 sidecar=12
295592fd560e db=813 sidecar=889
7478cae31f01 db=2730 sidecar=2405
The previous patch's SQL path was reading state.db. The pre-patch
inline walk was reading self.messages (the sidecar). Sidebar stale-row
detection (_looks_like_stale_zero_message_row,
_row_may_need_sidecar_metadata_refresh) consumes this field as if the
sidecar were the source of truth, so swapping the source silently flips
the field's semantics.
Drop the SQL path entirely. The helper now does the same in-memory
walk the pre-patch code did, extracted into a named method for
visibility. The inline role check (dict.get('role') with default
empty string) is 1.7x faster than calling _message_role() per
iteration on the test corpus (2.57ms -> 1.49ms across 8 sidecars,
2405 msgs total).
Bench (2405-msg sidecar): 0.76ms per call, vs ~1.5s for the pre-patch
inline walk on the same sidecar (the original perf hotspot). The
1.5s figure included JSON parse, dict construction, and the SQLite
overhead of the now-removed SQL path; the bare walk is the
0.76ms shown here.
Correctness: 27/27 webui sidecars produce identical counts vs the
pre-patch inline walk. No behavior change for any caller.
Addresses Greptile findings (P1 silent-zero, P2 docstring, P2 dead
allowlist) by removing the SQL path entirely rather than patching
around its divergent-source issue.
Greptile flagged that _compute_user_message_count_lazy silently returns 0
when the SQLite query fails (DB locked, missing file, schema drift).
The pre-patch compact() walked self.messages in Python and returned the
correct count without ever touching the DB, so the silent zero
fallback is a regression that breaks _looks_like_stale_zero_message_row
under contention.
Fix: helper now takes an optional messages argument. On SQLite failure
it falls back to the same in-memory walk the original code used, so
callers see the same number they'd have seen before for any session
where Session.load() populated self.messages.
Also drop the dead ('GET', '/api/session') allowlist entry flagged by
Greptile — the /api/session handler already does its own per-stage
logging via the _t0.._t6 block (with auto-log on slow requests), and
adding RequestDiagnostics on top would just duplicate the slow-request
journal entries without adding information.
Session.compact() called _last_message_timestamp(self.messages) which
reversed-walked all 2,730 messages on every /api/session response.
The messages array is chronologically ordered, so the last non-tool
message sits at the very end. Scan only the last 8 messages; fall
back to a full scan only if no timestamp is found in the window
(unusual sessions with >8 trailing tool rows).
Reverts to identical behavior when the window hits, since the most
recent user/assistant message's timestamp is what compact() emits.
Bench (single-shot, live Chromebook eMMC):
before: get_session=1107ms compact=1725ms total=2880ms
after: get_session= 648ms compact= 288ms total=1313ms
(-80% on compact, -54% on total)
Revert: git checkout master && systemctl --user restart hermes-webui
get_state_db_session_messages() returned archived (active=0) rows that in-place
compaction had marked inactive. WebUI reconciliation feeds this reader straight
into the next model-facing context (reconciled_state_db_messages_for_session,
prefer_context=True), so those archived rows got pulled back in — resurrecting
pre-compaction history and making every later turn re-trigger compression. Now,
WHEN the messages table exposes an `active` column, inactive rows are excluded by
default (`AND (active IS NULL OR active != 0)` — NULL/legacy rows preserved), with
an `include_inactive=True` escape hatch for explicit recovery/audit callers.
Schemas without the column are unaffected (guarded by `'active' in available`).
Contributor stage; ungated on arrival. Co-authored-by: ai-ag2026 <ai-ag2026@users.noreply.github.com>
Closes the crash-window resurrection corner in the #5532 clear-conversation
data-loss fix. /api/session/clear writes a .json.bak before truncating, and
because the cleared sidecar's truncation_watermark is 0.0, startup recovery saw
bak_count > live_count and could restore the backup — resurrecting the cleared
transcript after a crash/restart in the clear window.
Fix: stamp a unique per-clear `clear_generation` UUID on the sidecar when
clearing a session that had messages (persisted via Session model). Recovery's
_session_records_clear_sentinel() returns no_action when the live sidecar carries
a clear_generation the backup doesn't share AND the live sidecar matches the
exact cleared-empty shape (messages/context empty, watermark/boundary 0.0,
pending fields cleared). Same-generation backups stay recoverable; unreadable or
partial matches FAIL OPEN (normal recovery), so a real crash-loss is never
suppressed.
Contributor stage; gate-pass. Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Restore the compact() source-order expectation used by the branch metadata contract test while keeping focused compression recovery markers in the compact payload.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Make focused compression-recovery continuation creation idempotent by marking child sessions and reusing the existing child on repeated start requests. Also surface malformed successful recovery responses in the UI instead of silently re-enabling the action.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
@rodboev independently pushed fixes for messaging-tombstone (matches ours) and an
audit orphan-.bak skip — but the audit skip used _marks_deleted_webui_session
(index heuristic), which re-introduces the crash-suppression bug, and the other 4
resurrection/recovery paths were still open. Rebuilt the stage on rodboev's new
head (fc21c3f3, attribution preserved) and applied the complete superset:
- routes.py claim path: tombstone 404 gated on _state_db_session_source in
('','webui','fork') so foreign-source rows always materialize (finding B)
- models.py sidebar projection: skip tombstoned source='webui' rows w/ no live
sidecar so a deleted session can't resurface in /api/sessions (finding C)
- session_recovery.py orphan-.bak startup: skip restore when durable tombstone
present, durable-only (finding D)
- session_recovery.py audit orphan-.bak: durable-only classify as
deleted_session_skipped (corrects rodboev's index-heuristic version)
- session_recovery.py _read_state_db_missing_sidecar_rows: durable-tombstone-only
so a genuine crash (index intact, no durable tombstone) still repairs (finding F)
Regression tests (all verified fail-without-fix): sidebar-projection, orphan-.bak
startup skip, messaging-not-tombstoned, index-only-crash-repairable. Both advisors
(Codex SAFE TO SHIP + Opus ship-with-fixes) cleared the prior stage; this rebuild
carries the same fixes onto the contributor's latest work.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
_session_message_content_key (the state.db reconciliation key in
api/models.py) normalized whitespace only, while the streaming-side
identity _message_identity strips the workspace prefix for user turns.
WebUI sends the model a workspace-prefixed user_message
([Workspace::v1: /path]\n<text>) while the visible/optimistic bubble and
sidecar row carry the bare <text>. The mismatch made a prefixed state.db
row and a bare sidecar row key differently, so state_db_delta_after_context
failed to align them, treated the state.db copy as new, and appended a
duplicate user turn. The agent-side merge then concatenated the two
adjacent user rows into a permanent composite -- the post-restart
stale-user-prepend bug.
Fix: strip the workspace prefix for role=='user' in the reconciliation
key, reusing the same _strip_workspace_prefix helper the streaming side
uses (lazy import to avoid the api.streaming -> api.models cycle) so the
two dedup layers can't drift again. Assistant/tool keys are unchanged and
prefix-free user messages key identically (idempotent).
Fixes#5339.
Root cause of the silent-crash-after-hours cluster (#4765/#2233/#4633): the
global in-memory SESSIONS LRU evicted with a blind popitem(last=False), which
could drop an actively streaming or not-yet-persisted session (data loss) and
was capped only via an env var. On long-running installs the effective result
was unbounded RAM growth until segfault.
- Add _session_is_evictable(): a session is evictable ONLY when it is not
streaming (no active_stream_id), has no in-flight turn (no pending_user_message
/ pending_started_at), and its full state is proven on disk (sidecar
message_count >= in-memory count; metadata-only stubs and zero-message shells
are trivially safe).
- Add _evict_sessions_over_cap(): replaces all 8 blind popitem loops across
models.py, routes.py, streaming.py. Walks the LRU oldest-first and removes only
provably-safe entries; never acquires LOCK/stream locks itself (caller holds
LOCK) so no lock-ordering deadlock. May briefly exceed the cap rather than ever
evict an active/unsaved session.
- Make the cap configurable via config.yaml webui.sessions_cache_max
(get_sessions_cache_max()); precedence config.yaml -> HERMES_WEBUI_SESSIONS_MAX
(legacy) -> DEFAULT_SESSIONS_CACHE_MAX=300. No new HERMES_* env var. Invalid or
<1 values fall back so a typo can never disable the bound.
- Evicted sessions lazily reload from their JSON sidecar via the existing
get_session() accessor; no call sites changed. _index.json sidebar behavior
unchanged (the sidebar reads the index, not SESSIONS).
- Add tests/test_issue4765_sessions_lru_eviction.py (8 tests): eviction past
cap, active/streaming never evicted, unsaved/stale-tail never evicted, lazy
reload with identical content, and no-data-loss under heavy churn.
- README: document the config.yaml key + safety semantics.
Fixes#4765.
Regression seam behind #5308: a delegated subagent child's sidebar row is built
from a stale sidecar that reports message_count==0, and the state.db count
overlay in _apply_sidebar_state_db_override_metadata was gated on
`state_db_source == 'webui'`. A subagent child (state_db_source=='subagent')
therefore never received its true message count, so the front-end visibility
predicate (_sidebarRowHasVisibleMessages) dropped the row and the subagent
session disappeared entirely (not nested, not orphaned) after #5244+#5306.
Fix: widen the count/last-message overlay to `state_db_source in ('webui','subagent')`,
keeping the same conservative anti-resurrection guard. The source-tag/title
reassignment stays WebUI-only so a subagent child keeps its subagent
classification. Same state.db-blind-metadata root as the #5307 transcript
recovery, fixed server-side rather than by loosening the front-end predicate
(which would fight the #5306 active-parent scoping).
Tests: subagent child gets its count overlaid + classification preserved; a
non-webui/non-subagent foreign source (cron) still gets NO overlay.
Fixes#5308.
Windows locks session JSON files briefly during antivirus scans or
browser polling, causing os.replace() to raise WinError 5
(ERROR_ACCESS_DENIED). Add _safe_replace() helper that retries with
exponential backoff (50ms to 800ms, max 5 attempts) on PermissionError.
No-op on non-Windows platforms.
Replaces all 5 os.replace() call sites in models.py:
- Session.save() (main file + backup)
- _write_session_index() (full rebuild + fast path)
- prune_session_from_index()
Codex deep-gate caught a CORE regression in the original top-N cap: capping
_apply_sidebar_state_db_overrides also capped the SOURCE classification that
/api/sessions filters on BEFORE the lazy lineage correction, so a stale-CLI JSON
row whose state.db source is webui could be silently filtered out of a WebUI-only
sidebar once it fell beyond the cap. Split the read: the indexed sessions-table
source/title lookup runs for ALL rows; only the expensive messages COUNT/MAX
aggregation (the actual 5-18s bottleneck) is capped to top-N. Added a 500-row
end-to-end regression with a real state.db proving the beyond-cap stale-CLI row
stays webui.
On power-user instances with thousands of sessions, GET /api/sessions
blocked 5-18s in the all_sessions.state_db_overrides stage:
_apply_sidebar_state_db_overrides read state.db for every row (2400+ ids)
on every concurrent poll. With the gateway-watcher reading and the agent
writing the same state.db, this became lock/IO contention that piled up
concurrent polls and flapped the UI to 'Connection lost'.
Cap the override lookup to the top-N most-recent (paint-priority) rows
that are actually visible, mirroring the existing lineage-enrichment cap
(#4638). The caller passes an already pinned-first/newest-first sorted
list, so the top-N covers the visible window; rows beyond the cap keep
their JSON source/title and are corrected lazily on history-panel open.
Cap defaults to 300, env-configurable via
HERMES_WEBUI_STATE_DB_OVERRIDE_TOP_N (0 disables). Fails open.
Reported by @latipun, reproduced by @b3nw.
Co-authored-by: latipun <latipun@users.noreply.github.com>
Sidebar rows whose backing state.db session has zero messages (a '+'-click
that opened a row but the first turn never committed, or a sidebar nav that
opened then closed before any message landed) now get pruned instead of
lingering forever. The existing #3238/#4591 orphan-prune path only handled
CLI/API-server sidecars; this adds a parallel pass for native-WebUI rows.
- api/models.py: add agent_session_zero_message_sids() batch helper that
mirrors agent_session_rows_existing()'s safe-degrade contract (returns
frozenset() on any error so a transient failure never causes data loss).
Add a small tombstone file at SESSION_DIR/_pruned_webui_orphans.json
(capped at the last 500 sids) so the next poll's
recover_missing_index_sidecars does not re-add a pruned orphan to the
sidebar index — the per-poll fsync'd index write + state.db probe loop
the original PR would have hit on every poll for every orphan. The
helper, the load, and the full-scan fallback all honor the tombstone.
new_session() and import_cli_session() clear any matching tombstone
entry so a fresh session with the same id isn't shadowed.
- api/routes.py: extract the webui zero-message prune into a private
_prune_orphaned_webui_zero_message_sessions() helper, then call it
from BOTH branches of _build_session_list_cache_payload. Established
installs pin show_cli_sessions=False (per api/config.py:7637-7648)
and are exactly the long-time users who accumulated the #4985 404
orphans — the original PR only ran the prune in the if branch and
silently skipped the else branch (maintainer review
IC_kwDOR1LuPM8AAAABHsyFGg). The helper also honors the tombstone
defensively so the prune can't thrash on a row the index still
carries from a prior poll.
- tests/test_issue4985_orphaned_webui_zero_message.py: 39 regression
tests — the original 25 (7 helper-direct + 12 monkeypatch predicate
tests + 6 real-pipeline end-to-end tests) plus 14 follow-up tests
driven by the MUST-FIX / tombstone / r6 signal-swap reviews:
* test_helper_prunes_orphan_keeps_safety_retained_rows — direct call
to the extracted helper with a row set covering every shape the
gate is responsible for.
* test_real_all_sessions_post1171_prune_fires_when_show_cli_sessions_false
— explicit regression for the established-install case the
original PR missed.
* test_real_all_sessions_post1171_prune_fires_when_show_cli_sessions_true
— companion positive control.
* test_tombstone_persists_across_polls — drives two pipeline builds,
asserts prune fires once total and the tombstoned sid is NOT
re-added to SESSION_INDEX_FILE.
* test_tombstone_does_not_block_new_session_with_same_id — defensive
cover for the new_session() / import_cli_session() tombstone clear.
* test_tombstone_trimmed_to_last_N_entries — verifies the cap.
* test_real_all_sessions_post1171_prune_fires_when_show_cli_sessions_true
— companion positive control (show_cli_sessions=True path).
* test_tombstone_self_heals_when_message_added — confirms
Session.save()'s belt-and-suspenders tombstone clear fires when
a sidecar commit lands with len(messages) > 0.
* test_tombstone_loader_is_fail_open_on_corrupt_file[non-json|version-mismatch|non-dict-root]
— 3-parameter regression for the fail-open corrupt-file contract
(#5023 lesson).
* test_sidecar_only_webui_session_is_retained — sidecar JSON carries
real messages but state.db.messages is empty (mirroring lag); the
r6 loaded-sidecar probe retains the row.
* test_sidecar_only_webui_session_self_heals_tombstone — the r4.5
self-heal path combined with the r6 sidecar-only retention rule.
* test_truly_empty_sidecar_with_title_still_pruned — companion
negative test that the r6 fix does NOT over-retain empty sidecars
with non-Untitled titles.
* test_sidecar_only_webui_session_retained_when_show_cli_sessions_false
— established-install path also respects the r6 retention rule.
- CHANGELOG.md: Unreleased entry under Fixed.
fix: target post-#1171 survivors + hoist + tombstone + r6 sidecar-load probe
The r5 sidecar-only retention signal keyed off the row's cached
``message_count`` — which is stale-positive on the very phantom rows
#4985 exists to prune (sidecar ``messages`` empty but cached count > 0).
The r6 signal swaps that cached-count probe for a ``Session.load(sid)
.messages`` probe, gated to ``state.db``-empty candidates only so the
common live-row path stays a no-op. Three retain tests now write real
populated ``messages`` arrays (not just ``[]``) so the loaded-sidecar
check has something to find; the stale-count phantom test is left as
the canonical r6 regression (its ``messages=[]`` + ``message_count=5``
sidecar is exactly the shape the r6 loaded-sidecar check correctly
classifies as orphan). Maintainer review 4584722701.
t