- _repair_bare_custom_provider_model matches display-named providers via
_custom_provider_slug_from_name (custom:my-proxy matches 'My Proxy').
- _ordered_custom_provider_model_ids now handles dict keys, list strings,
and list dicts with id/model/name, aligned with api/config.py catalog.
- config_obj=None fallback uses get_config() instead of raw cfg.
- _read_profile_model_config returns profile config dict too, avoiding a
second YAML parse on hot display paths.
- Tests added for slug matching, list-form models, and get_config fallback.
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
The two-tier mtime cache from #4783 fixed the per-request SKILL.md rescan
but left two concurrency holes that only bite at container cold start,
when the frontend fires several profile-data requests at once and the
caches are empty:
1. `_get_profile_skills_stats()` had no lock, so concurrent misses on the
same profile each ran `os.walk(followlinks=True)` + parsed every
SKILL.md simultaneously.
2. `_build_profile_rows_fast()` ran outside `_LIST_PROFILES_CACHE_LOCK`
in `list_profiles_api()`, so every concurrent request rebuilt all rows
(each walking every profile's skill tree) at once.
With ThreadingHTTPServer (one OS thread per request) and Docker overlay2,
this stacked thousands of concurrent stat() calls and stalled workers
57-70s (per the report's thread dumps).
Fix:
- Add a per-profile compute lock (registry guarded by a meta-lock) and
use double-checked locking in `_get_profile_skills_stats()`: concurrent
misses on one profile collapse to a single compute, while independent
profiles still compute in parallel.
- Single-flight the row build in `list_profiles_api()` by holding
`_LIST_PROFILES_CACHE_LOCK` across the build + cache write. Lock order
is strictly list-lock -> per-profile skills-lock, so no deadlock.
The report's third suggestion (debounce the mtime probe) is deliberately
NOT taken: the every-call cheap probe is the #4783 out-of-band
change-detection contract (test_issue4783 asserts it MUST run on every
call). Serializing the misses removes the herd without weakening that
contract, since only the expensive compute is guarded, not the probe.
Adds tests/test_issue5364_skills_stats_thundering_herd.py proving the
herd collapses (single compute / single build under a concurrent burst),
independent profiles still parallelize, and the every-call probe contract
is preserved. All existing #4783 contract tests still pass.
Co-authored-by: claw-io <claw-io@users.noreply.github.com>
_session_message_content_key (the state.db reconciliation key in
api/models.py) normalized whitespace only, while the streaming-side
identity _message_identity strips the workspace prefix for user turns.
WebUI sends the model a workspace-prefixed user_message
([Workspace::v1: /path]\n<text>) while the visible/optimistic bubble and
sidecar row carry the bare <text>. The mismatch made a prefixed state.db
row and a bare sidecar row key differently, so state_db_delta_after_context
failed to align them, treated the state.db copy as new, and appended a
duplicate user turn. The agent-side merge then concatenated the two
adjacent user rows into a permanent composite -- the post-restart
stale-user-prepend bug.
Fix: strip the workspace prefix for role=='user' in the reconciliation
key, reusing the same _strip_workspace_prefix helper the streaming side
uses (lazy import to avoid the api.streaming -> api.models cycle) so the
two dedup layers can't drift again. Assistant/tool keys are unchanged and
prefix-free user messages key identically (idempotent).
Fixes#5339.
Root cause of the silent-crash-after-hours cluster (#4765/#2233/#4633): the
global in-memory SESSIONS LRU evicted with a blind popitem(last=False), which
could drop an actively streaming or not-yet-persisted session (data loss) and
was capped only via an env var. On long-running installs the effective result
was unbounded RAM growth until segfault.
- Add _session_is_evictable(): a session is evictable ONLY when it is not
streaming (no active_stream_id), has no in-flight turn (no pending_user_message
/ pending_started_at), and its full state is proven on disk (sidecar
message_count >= in-memory count; metadata-only stubs and zero-message shells
are trivially safe).
- Add _evict_sessions_over_cap(): replaces all 8 blind popitem loops across
models.py, routes.py, streaming.py. Walks the LRU oldest-first and removes only
provably-safe entries; never acquires LOCK/stream locks itself (caller holds
LOCK) so no lock-ordering deadlock. May briefly exceed the cap rather than ever
evict an active/unsaved session.
- Make the cap configurable via config.yaml webui.sessions_cache_max
(get_sessions_cache_max()); precedence config.yaml -> HERMES_WEBUI_SESSIONS_MAX
(legacy) -> DEFAULT_SESSIONS_CACHE_MAX=300. No new HERMES_* env var. Invalid or
<1 values fall back so a typo can never disable the bound.
- Evicted sessions lazily reload from their JSON sidecar via the existing
get_session() accessor; no call sites changed. _index.json sidebar behavior
unchanged (the sidebar reads the index, not SESSIONS).
- Add tests/test_issue4765_sessions_lru_eviction.py (8 tests): eviction past
cap, active/streaming never evicted, unsaved/stale-tail never evicted, lazy
reload with identical content, and no-data-loss under heavy churn.
- README: document the config.yaml key + safety semantics.
Fixes#4765.
Regression seam behind #5308: a delegated subagent child's sidebar row is built
from a stale sidecar that reports message_count==0, and the state.db count
overlay in _apply_sidebar_state_db_override_metadata was gated on
`state_db_source == 'webui'`. A subagent child (state_db_source=='subagent')
therefore never received its true message count, so the front-end visibility
predicate (_sidebarRowHasVisibleMessages) dropped the row and the subagent
session disappeared entirely (not nested, not orphaned) after #5244+#5306.
Fix: widen the count/last-message overlay to `state_db_source in ('webui','subagent')`,
keeping the same conservative anti-resurrection guard. The source-tag/title
reassignment stays WebUI-only so a subagent child keeps its subagent
classification. Same state.db-blind-metadata root as the #5307 transcript
recovery, fixed server-side rather than by loosening the front-end predicate
(which would fight the #5306 active-parent scoping).
Tests: subagent child gets its count overlaid + classification preserved; a
non-webui/non-subagent foreign source (cron) still gets NO overlay.
Fixes#5308.
server.py exited SILENTLY after 9-16h: no traceback, no shutdown-audit line,
no core dump — the log just stopped mid-request. It runs a ThreadingHTTPServer
with daemon_threads=True, so an unhandled exception in a request/SSE/long-poll
handler thread could terminate work with nothing recorded, and faulthandler was
not enabled so a native crash left nothing at all. The WebUI also configures no
logging handlers, so INFO/ERROR records are dropped by logging's lastResort
filter (WARNING+ only) — meaning even the existing shutdown audit never reached
the log.
Add api/crash_visibility.py (stdlib-only, hooks never raise) and wire
install_crash_visibility() into server.main() before any heavy startup:
* faulthandler.enable(all_threads=True) — native crash dumps a C-level
traceback; SIGUSR1 registered for on-demand hang diagnosis.
* threading.excepthook — logs uncaught daemon/handler-thread exceptions
(thread name, ident, traceback) instead of losing them silently.
* sys.excepthook — logs uncaught main-thread exceptions (KeyboardInterrupt
preserved).
* atexit exit-audit breadcrumb — a clean/unwound exit is recorded; its
absence narrows a silent death to an un-unwound kill (OOM/SIGKILL/abort).
All diagnostics are written directly to the fault stream (stderr, which the
bootstrap redirects into the WebUI log) AND mirrored through logging, so the
line is guaranteed to land regardless of logging config.
No request-handling behavior change, no new deps. Paired memory root-cause: #4765.
Fixes#4633.
Structural hardening per user request, following the 3rd round of the
same bypass class on PR #5313's reasoning_efforts feature:
Round 1: plain-ordering regression (provider-config short-circuit ran
before the nested-route deny).
Round 2: a provider-qualified hint (@custom:<slug>:vertex/gemini-...)
left a slug fragment that the deny's prefix-match missed.
Both were legitimate, narrowly-targeted fixes, but the pattern —
_nested_route_reasoning_denied() requiring the model string to START
WITH the route prefix — meant every future wrapper/nesting scheme
would need the strip logic to be updated in lockstep, and a missed
case fails OPEN (reasoning re-enabled on a route that must never show
it), which is the wrong failure mode for a security-adjacent guard.
This commit removes that class of bug instead of patching its latest
instance: _nested_route_reasoning_denied() now searches for the
vertex/gemini- or gemini_cli/gemini- pattern ANYWHERE in the string at
a non-alphanumeric boundary, rather than requiring it at position 0.
Correctness no longer depends on _strip_provider_hint_for_reasoning()
having stripped exactly the right prefix first — any number of opaque
wrapper layers (@provider:, a named custom-provider slug, or any
nesting scheme not yet invented) can precede the route and the deny
still fires.
Verified: all historical bypass strings (round 1 and round 2, plus a
hypothetical deeper double-wrapped case) now correctly deny; embedded
substrings that must NOT match (e.g. 'notvertex/gemini-x') correctly
don't, thanks to the boundary lookbehind.
Added test_nested_route_deny_is_boundary_based_not_prefix_based
locking in the structural invariant directly, independent of any
particular wrapper scheme.
Verification:
./scripts/test.sh tests/test_reasoning_effort_model_capabilities.py tests/test_custom_provider_bare_model_reasoning.py
# 54 passed in 1.31s
Full suite: 11257 passed, 64 pre-existing failures (identical file/test
set as the prior commit's baseline — network/credential/profile-isolation/fd-leak
tests, unrelated to reasoning_efforts).
Resolve gate certifier feedback on PR #5313 (deeper bypass, round 3):
A provider-qualified hint like `@custom:<slug>:vertex/gemini-image-1.0`
still bypassed `_nested_route_reasoning_denied()`. The old
`_strip_provider_hint_for_reasoning()` did a naive first-colon split,
which only removed the leading `@custom:` wrapper and left the named
provider's slug (e.g. `agg:vertex/gemini-image-1.0`) attached to the
model id. That leftover slug fragment no longer starts with
`vertex/gemini-`, so the nested-route deny missed it and a configured
`providers.<name>.reasoning_efforts` / `custom_providers[].reasoning_efforts`
allowlist re-enabled reasoning controls on Gemini image/embedding routes
that must never expose them.
Fix: `_strip_provider_hint_for_reasoning()` now accepts the resolved
provider id and strips the exact `@{provider}:` prefix first (e.g.
`@custom:agg:`), so both the wrapper and the slug are removed in one
pass before the nested-route deny check runs. Falls back to the
original first-colon split when no provider is supplied, preserving
existing behavior for plain `@provider:model` hints.
Verified in-process: both
`@custom:agg:vertex/gemini-image-1.0` and
`@custom:agg:vertex/gemini-embedding-001` now correctly resolve to []
instead of leaking the configured ["low", "high"].
Added regression test extending the existing nested-route-deny test to
cover provider-qualified hinted models.
Verification:
`./scripts/test.sh tests/test_reasoning_effort_model_capabilities.py tests/test_custom_provider_bare_model_reasoning.py`
-> 53 passed in 1.23s
Full suite: 11258 passed (63 pre-existing failures unrelated to this
change — network/credential/profile-isolation/fd-leak tests; confirmed
identical failure set on unmodified branch).
Resolve review feedback on PR #5313:
- Read reasoning_efforts from named custom_providers entries for custom:<name>
providers instead of looking only in providers:<name>.
- Preserve the nested-route deny ordering before the provider-config shortcut
so Gemini image/embedding routes cannot be re-enabled by config.
- Deduplicate configured effort values before returning them.
Add focused regression tests for bare provider config, named custom provider
config, all-invalid fallthrough, ACP hard guards, and nested route denies.
Phase 2 of cleanup now sweeps _index.json for stale entries with no
backing file and no in-memory session, removing them regardless of
title (catches multi-language 'Untitled' variants). Phase 3 only
deletes the index when Phase 1 removed files AND Phase 2 couldn't
run, fixing the cache-busting on every cleanup call.
Fix two phase-interaction bugs flagged by Greptile review:
- Track phase1_removed_ids to prevent Phase 2 double-counting
sessions already removed from disk by Phase 1.
- Track phase2_rewrote_index so Phase 3 skips deletion when
Phase 2 already cleaned the index in-place.
Adds test_issue5331_index_only_ghost_cleanup.py with 12 test cases.
_session_is_subagent_view_only guard added to the remaining paths that could
mutate a stale persisted subagent sidecar:
- _handle_handoff_summary (appends tool msg to state.db)
- _handle_chat_sync (fallback POST /api/chat)
- /api/session/draft POST (composer_draft save)
- /api/personality/set (per-session personality save)
Fixes#5307
Enumerated every /api/session/<mutation> route and applied _session_is_subagent_view_only:
duplicate, branch, retry, undo, toolsets, compress, archive (existing-sidecar path)
now 400 for subagent children; delete/clear/truncate/pin already gated; rename/move/update
route through the gated _get_or_materialize_session (403). This closes the data-loss
paths (delete_cli_session, retry/undo truncate, branch/duplicate fork).
Also: /api/sessions list coercion now also honors state.db source='subagent' for rows
whose stale index says webui/fork, so they can't surface as writable/CLI sidebar rows.
Fixes#5307
The root non-CLI classification surfaced subagent rows in the sidebar without
read_only, exposing delete/truncate/pin — and /api/session/delete calls
delete_cli_session() which erases the child's state.db transcript (data loss).
- /api/sessions list: coerce subagent rows to read_only=True + is_cli_session=False
so the UI offers no mutation affordances.
- New _session_is_subagent_view_only() shared guard; applied to the direct
mutation routes that bypass _get_or_materialize_session(): delete / clear /
truncate / pin now 400 for subagent children. (rename/move already route
through the gated _get_or_materialize_session and 403.)
- tests: guard helper + static contract that all 4 routes + list coercion present.
Fixes#5307
- api/agent_sessions.py is_cli_session_row(): add 'subagent' to non_cli_sources
so EVERY consumer (sidebar rows, /api/sessions, etc.) classifies a delegated
child as non-CLI (the root the per-site fixes were compensating for).
- api/routes.py GET /api/session happy path: coerce is_cli_session=False +
read_only=True in the serialized payload for a subagent child before redaction,
so a stale writable sidecar can't be exposed as writable to the browser.
Fixes#5307
- _get_or_materialize_session happy path: reject an already-persisted subagent
sidecar (source_tag/raw_source=='subagent' or state.db subagent) even when it
was stored read_only=False (pre-fix materialization), so chat-start can't write it.
- import_cli existing-session refresh: coerce read_only=True on the persisted
sidecar (and response) when it's a subagent child.
- test: persisted read_only=False subagent sidecar is still refused by the helper.
Fixes#5307
- import_cli _read_only_view now also treats resolved cli_meta source_tag/raw_source
=='subagent' as view-only (all_profiles=true resolves cli_meta from a non-active
profile, which the active-profile state.db _sa_child check could miss).
- import_cli existing-session refresh branch no longer hardcodes is_cli_session=True
for a subagent child (both the persisted update and the response payload).
Fixes#5307
Codex round 4 found POST /api/session/archive's missing-sidecar fallback also
calls import_cli_session(). grep confirms exactly 3 import_cli_session() call
sites in routes.py; all 3 now refuse source='subagent' children:
- 4792 _get_or_materialize_session (chat-start) -> PermissionError
- 22382 import_cli endpoint -> read-only view payload
- 13885 archive fallback -> 400 'Subagent sessions cannot be archived'
So no path can materialize a delegated child as a writable WebUI sidecar.
Fixes#5307
Codex found a 3rd writable path: POST /api/chat/start -> _get_or_materialize_session()
materialized source='subagent' as a writable sidecar before the not_claimable guard.
Fix: refuse subagent children (PermissionError) at that shared chokepoint, checked via
_is_subagent_child_session_id(sid) (state.db source, independent of cli_meta) BEFORE the
materialize path — so all three entry points (GET synth, import_cli, chat-start) now
consistently keep a delegated child view-only. CLI/TUI/Desktop materialization preserved.
Tests: materialize helper refuses subagent child + still allows tui.
Fixes#5307
- api/routes.py GET /api/session synthesized response: stop hardcoding
is_cli_session=True; serialize bool(synth.is_cli_session) so a recovered
subagent child stays not-CLI-classified (was overriding the helper's False).
- api/routes.py POST /api/session/import_cli: gate source='subagent' into the
read-only view payload (is_cli_session=False, imported=False) BEFORE
import_cli_session(), so a subagent child can never be materialized as a
writable WebUI sidecar via this endpoint.
- tests: assert import_cli routes subagent children read-only (no materialize).
Both were paths that bypassed the _is_claimable_cli_source denylist. Fixes#5307
Reworked per the Codex gate finding: the first approach widened the client
import predicate, which (a) conflicted with #3603's intentional _isExternalSession
gate and (b) let import_cli persist the subagent child as a WRITABLE, CLI-classified
session that then passed the poll-skip/active-refresh gates.
Corrected to a server-side, view-only recovery:
- api/routes.py: mark source='subagent' as NON-claimable in _is_claimable_cli_source
(both cli_meta and state.db source denylists). A subagent child now resolves via
the not_claimable branch -> read-only Session with its state.db transcript, and
build_session takes is_cli_flag (False for subagent children) so the recovered
session is NOT CLI-classified and can't widen the frontend _isExternalSession gates.
- static/sessions.js: REVERTED to master (no client change needed; #3603 contract intact).
- tests: assert reason=not_claimable, read_only=True, is_cli_session!=True, transcript present;
#2782 deleted-webui 404 preserved; #3603 _isExternalSession contract preserved.
58 tests green (5307 + 3603 + claim-cli + core-data-loss). Fixes#5307
A delegated subagent child (source='subagent' in state.db) usually has no WebUI
sidecar but is registered in the WebUI index sharing the parent's lineage. That
made GET /api/session -> _claim_or_synthesize_cli_session return 'was_webui' ->
404, so the child pane opened empty despite state.db holding messages.
- api/routes.py: add _state_db_session_source() + _is_subagent_child_session_id();
exclude subagent children from the was_webui 404 gate so they recover their
state.db transcript (the #2782 self-heal 404 for deleted WebUI sessions is kept).
- static/sessions.js: add _isSubagentChildSession() + _sessionNeedsServerImportForLoad()
(kept separate from _isExternalSession to avoid widening refresh-gating); the
main session-tap, lineage-segment, and child-open handlers now trigger the
import/merge path for subagent children.
- tests: test_5307_subagent_child_transcript.py (7 tests).
Fixes#5307
- Add _repair_bare_custom_provider_model() shared by fast/slow paths (#5314)
- Use ordered model id list (config declaration order) for deterministic repair
- Shrinks fast-path block so test_issue1855 fast path stays before catalog call
Refs #5314
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
On subsequent turns (such as Turn 2), the client-side select dropdown can automatically normalize and strip the provider namespace prefix from the selection. On the next user input, the client POSTs the bare model ID (e.g. `grok-composer-2.5-fast`), which the server failed to repair back to its full qualified name unless it matched the suffix of the profile's configured default model.
This updates both fast-path and slow-path repair checks in `_resolve_compatible_session_model_state()` to dynamically query the active custom provider's configured models list in `config.yaml` (`custom_providers`). This guarantees that any bare model belonging to the configured custom provider is dynamically re-qualified back to its fully-namespaced form, regardless of whether it is the profile's configured default model.
Refs #5314
Co-authored-by: b3nw <b3nw@users.noreply.github.com>
Addresses Greptile review feedback on PR #5313:
1. Move cursor-acp/copilot-acp guard before step 0 so a stray config
entry can't surface unsupported effort options for those providers.
2. Only short-circuit when the filtered list is non-empty; an all-invalid
list (e.g. typos) falls through to heuristics instead of hiding
reasoning support from the UI.
Add a config-driven path to resolve_model_reasoning_efforts() that reads
providers.<name>.reasoning_efforts from config.yaml. This lets users
explicitly list valid reasoning effort levels per provider, so the
WebUI dropdown only shows options the model actually supports.
Handles both custom:<name> and bare registered provider names.
Falls through to existing heuristics (Copilot per-model lists, LM Studio
live API, models.dev) when no config entry is present.
Reapply footer control ordering on current upstream/master while preserving the required situational chip renderer. Persist composer_control_order with backend validation, keep the settings descriptions reorder-aware, and make primary/situational chip renderers participate in same-group drag ordering.
Verified with: node --check static/boot.js; node --check static/panels.js; git diff --check; ./scripts/test.sh tests/test_issue4598_composer_control_visibility.py
The embedded workspace terminal spawns a PTY shell that runs arbitrary
commands as the server-process user. check_auth() returns True
unconditionally when no password/passkey is configured (the default
out-of-the-box state), so without a network-scope gate the terminal
endpoints were reachable by any unauthenticated caller able to hit the
port — which on a passwordless public bind is remote code execution.
Apply the same local-origin gate the onboarding/bootstrap endpoints use
(_onboarding_gate_allows) to /api/terminal/{start,input,resize,close} and
/api/terminal/output: with auth disabled, accept only loopback/private
origins, ignore spoofable X-Forwarded-For/X-Real-IP unless
HERMES_WEBUI_TRUST_FORWARDED_FOR=1, and honor HERMES_WEBUI_ONBOARDING_OPEN=1
as the explicit opt-out for a deliberately-exposed server. Auth-enabled
servers (cookie already verified upstream) and genuine same-host clients
are unaffected.
Also fixes a latent test-isolation leak in
test_extension_route_remains_behind_webui_auth: it set HERMES_WEBUI_PASSWORD
but never invalidated the process-wide password-hash cache, so its result
depended on suite execution order (exposed when the new test file shifted
ordering). Invalidate before+after so it reads the env var deterministically.
12 new gate tests in tests/test_cvd3_terminal_local_origin_gate.py.
nesquena-hermes