The two-tier mtime cache from #4783 fixed the per-request SKILL.md rescan
but left two concurrency holes that only bite at container cold start,
when the frontend fires several profile-data requests at once and the
caches are empty:
1. `_get_profile_skills_stats()` had no lock, so concurrent misses on the
same profile each ran `os.walk(followlinks=True)` + parsed every
SKILL.md simultaneously.
2. `_build_profile_rows_fast()` ran outside `_LIST_PROFILES_CACHE_LOCK`
in `list_profiles_api()`, so every concurrent request rebuilt all rows
(each walking every profile's skill tree) at once.
With ThreadingHTTPServer (one OS thread per request) and Docker overlay2,
this stacked thousands of concurrent stat() calls and stalled workers
57-70s (per the report's thread dumps).
Fix:
- Add a per-profile compute lock (registry guarded by a meta-lock) and
use double-checked locking in `_get_profile_skills_stats()`: concurrent
misses on one profile collapse to a single compute, while independent
profiles still compute in parallel.
- Single-flight the row build in `list_profiles_api()` by holding
`_LIST_PROFILES_CACHE_LOCK` across the build + cache write. Lock order
is strictly list-lock -> per-profile skills-lock, so no deadlock.
The report's third suggestion (debounce the mtime probe) is deliberately
NOT taken: the every-call cheap probe is the #4783 out-of-band
change-detection contract (test_issue4783 asserts it MUST run on every
call). Serializing the misses removes the herd without weakening that
contract, since only the expensive compute is guarded, not the probe.
Adds tests/test_issue5364_skills_stats_thundering_herd.py proving the
herd collapses (single compute / single build under a concurrent burst),
independent profiles still parallelize, and the every-call probe contract
is preserved. All existing #4783 contract tests still pass.
Co-authored-by: claw-io <claw-io@users.noreply.github.com>
Codex round-4 found my followlinks=True change introduced a perf regression: the
probe os.walk lacked the .git/.venv/node_modules/site-packages + skill-support-dir
pruning that iter_skill_index_files applies, so a skill vendoring a dependency
tree made this EVERY-CALL probe walk thousands of irrelevant files — defeating the
cache's whole purpose. The probe now prunes dirnames[:] with the same
EXCLUDED_SKILL_DIRS + SKILL_SUPPORT_DIRS sets as the compute path. Regression test
(change deep inside node_modules/ must not move the probe) added + non-vacuous.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Codex round-3 reproduced: _skill_tree_max_mtime_ns used os.walk(followlinks=False)
while the compute path iter_skill_index_files uses followlinks=True, so an edit to
a symlinked skill's target SKILL.md was invisible to the probe (stale up to TTL).
Added followlinks=True to the probe walk so it descends symlinked skill dirs and
sees target mtime changes. Regression test added + proven non-vacuous.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Codex re-gate: my prior 'TTL-expired + unchanged-mtime -> refresh TTL, no recompute'
branch could leave counts stale INDEFINITELY for an mtime-PRESERVING out-of-band
change (e.g. a git checkout restoring the old mtime) — worse than master's 8s
recompute. Codex proved it (cached (2,2) stayed (2,2) while a fresh compute returned
(1,2)). Now: within TTL + unchanged mtime serves cached (perf win kept); mtime
changed OR TTL expired both force a full recompute (the TTL is a genuine safety
net). Updated the row-2 test to assert the safety recompute.
Co-authored-by: Rod Boev <rod.boev@gmail.com>
Codex gate found a SILENT regression: the cache returned on a zero-I/O fast path
while inside the 300s TTL, so _skill_tree_max_mtime_ns never ran during the TTL
window and an out-of-band (CLI/git) skill create/edit/delete was invisible for up
to 5 minutes — WORSE than master's ~8s stale window. Restructured so the cheap
stat-only mtime probe runs on EVERY call (prompt change detection); the TTL now
only bounds the expensive SKILL.md recompute. Unchanged mtime still serves cached
+ refreshes TTL with no recompute (the perf win is preserved). Updated 2 tests
that encoded the flawed zero-I/O design + added a within-TTL-change regression
(proven non-vacuous).
Co-authored-by: Rod Boev <rod.boev@gmail.com>
The registry-derived scrub union is best-effort and returns empty when
hermes_cli.auth can't be imported (WebUI-only CI/test env). That made
ANTHROPIC_TOKEN / CLAUDE_CODE_OAUTH_TOKEN scrubbing depend on the agent being
installed, so the quota-subprocess regression failed on CI (agent absent) while
passing locally (agent present). Add both token names to the fail-closed floor
set so the scrub covers them regardless of the registry import — strictly more
correct (defense-in-depth) and deterministic across environments.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Final credential-surface item: agent/azure_identity_adapter.py treats
IDENTITY_ENDPOINT / MSI_ENDPOINT as ManagedIdentityCredential sources, so an
empty named profile could inherit the host's managed identity on the quota +
detached-worker paths. Add IDENTITY_ENDPOINT / IDENTITY_HEADER / MSI_ENDPOINT /
MSI_SECRET to the non-registry scrub set (config var AZURE_FOUNDRY_BASE_URL kept).
Detached-worker regression extended.
This completes the agent's enumerable credential-env read surface (verified from
source). Decision (with maintainer): stop the Codex deep-gate here — the
remaining theoretical surface is cloud-SDK-internal credential-source endpoints
on pre-existing infra, not vectors introduced by this PR; validated via Opus +
full suite. The PR's #3961 scope (os.environ mirror race on /api/providers +
/api/models) is fully closed plus the broader same-class hardening.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Round-3 gate found two more credential env vars the agent resolves cross-profile
that the scrub set missed, verified against the installed agent source:
- AWS_CONTAINER_CREDENTIALS_FULL_URI — read by model_switch._has_aws_creds:1360
(botocore container-credentials provider; RELATIVE_URI was present, FULL_URI
was not).
- Azure identity family — AZURE_CLIENT_ID / AZURE_CLIENT_SECRET / AZURE_TENANT_ID
/ AZURE_FEDERATED_TOKEN_FILE (agent/azure_identity_adapter.py service-principal
+ workload-identity model auth) and AZURE_FOUNDRY_API_KEY
(runtime_provider.py).
Enumerated the agent's full credential-env read surface from source to close the
whole class in one pass rather than one var per gate round. Deliberately EXCLUDES
config/region vars (AWS_REGION, AWS_DEFAULT_REGION, AZURE_FOUNDRY_BASE_URL) — the
child probe may legitimately need those. Regressions extended to pin FULL_URI +
the Azure family.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Codex re-gate found the registry-derived scrub still missed credential env vars
the agent resolves via raw os.getenv() that aren't in PROVIDER_REGISTRY:
CUSTOM_API_KEY (hermes_cli/models.py), AWS_BEARER_TOKEN_BEDROCK
(hermes_cli/model_switch.py), AZURE_ANTHROPIC_KEY (runtime_provider.py), and the
boto3 AWS_* credential family (agent/bedrock_adapter.py). Same cross-profile leak
class on custom/AWS-backed provider quota + detached-worker paths.
- Add _NON_REGISTRY_AGENT_CREDENTIAL_ENV_NAMES fail-closed set, union into
_profile_secret_env_names so BOTH leak sites pick it up.
- _account_usage_subprocess_env now relies on the centralized scrub set (WebUI-
only fallback only if the api.profiles import fails), dropping the partial
local AWS pair.
- 2 more regressions (quota subprocess + detached worker) covering CUSTOM_API_KEY
+ AWS family, fail without the fix.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Codex re-gate of #4544 reproduced a residual cross-profile leak: the PR closed
OPENROUTER_API_KEY but ANTHROPIC_TOKEN / CLAUDE_CODE_OAUTH_TOKEN still leaked the
server-process credential to an empty named profile on /api/provider/quota
(anthropic) and the detached-worker model-rebuild path, because the scrub set was
built only from the WebUI _PROVIDER_ENV_VAR map (which omits OAuth/token-flow
provider env vars).
Widen the shared _profile_secret_env_names() scrub source to union in the agent
registry's api_key_env_vars (hermes_cli.auth.PROVIDER_REGISTRY) so every
credential env var the agent runtime can resolve via raw os.getenv() is scrubbed
in profile-scoped reads. Fixes both leak sites at once (both route through
_profile_secret_env_names). + 2 regressions that fail without the fix.
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
* stage #4620 (franksong2702): snapshot isolated-profile opt-in at import (fixes#4590)
Security/deployment-hardening follow-up to #4586/#4589. Snapshots
HERMES_WEBUI_ISOLATED_PROFILE at module import (_INITIAL_ISOLATED_PROFILE_OPT_IN)
so _isolated_profile_opt_in() reads the startup snapshot, not live os.environ — a
pinned profile .env loaded later can't flip the isolation posture. _reload_dotenv
key filter kept as defense-in-depth. Adds one-time warning when HERMES_HOME has a
profile-dir shape but isolated wasn't opted in at startup. Code byte-faithful from
PR head 5ef14264.
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
* Release v0.51.566 — Release TY (isolated-profile opt-in hardening; #4620)
---------
Co-authored-by: nesquena-hermes <agent@nesquena-hermes>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
Codex re-gate (CORE, reproduced): _reload_dotenv() was guarded, but get_profile_runtime_env() also
copies a profile .env into runtime env, which profile_env_for_background_worker()/streaming apply to
os.environ -> HERMES_WEBUI_ISOLATED_PROFILE=0 in a profile .env still escaped isolation on that path.
Fix: filter _PROTECTED_ENV_KEYS in get_profile_runtime_env's .env loop + add the key to
_BLOCKED_RUNTIME_ENV_KEYS (gateway-parity filter). + runtime-path regression test.
Codex gate (CORE, reproduced): _reload_dotenv() copies a profile's .env into os.environ, so a
contained user could set HERMES_WEBUI_ISOLATED_PROFILE=0 in their own profile .env and silently
escape isolation. Fix: _PROTECTED_ENV_KEYS — _reload_dotenv refuses to override operator/deployment
keys from a profile .env (logs a warning). + regression test proving .env can't disable isolation.
Profile-isolation regression: isolated mode was inferred from HERMES_HOME path shape (*/profiles/<name>),
which the launcher exports for ANY normal named profile -> single-users on a named profile lost profile
switching + saw only 1 profile. Fix: HERMES_WEBUI_ISOLATED_PROFILE explicit opt-in is now the PRIMARY gate.
Codex + Opus both independently caught a CORE gap in the first cut: the
bounded /api/models rebuild runs on a detached 'models-catalog-rebuild' daemon
thread that inherits neither the request-profile thread-local (#798) nor
os.environ. So on a non-default profile the worker probed the DEFAULT profile's
credentials and, when the 4s budget was exceeded, published the rebuilt catalog
to the DEFAULT profile's disk cache (cross-contamination) — exactly the slow
path a non-default cold rebuild takes.
Fix:
- profile_scope_for_detached_worker(profile_name): sets the request-profile TLS
AND applies the profile .env on the worker thread, restoring both on exit
(no-op for default). Distinct from profile_env_for_active_request (which reads
the current thread's TLS and must not clear it).
- get_available_models() captures the active profile on the request thread and
wraps the rebuild worker body (probe + over-budget publish + disk save +
fingerprint) in that scope; the legacy synchronous rebuild applies the profile
env on the foreground. /api/models route no longer wraps (the work moved into
get_available_models so ALL callers — chat/start, resolution — are fixed).
- 2 new regression tests incl. the worker-thread before/inside/after assertion.
Empirically verified: a fresh worker thread resolved models_cache.json/default
WITHOUT the scope (the bug) and models_cache.work.json + the work .env + the
work auth.json WITH it.
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>
On a non-default profile, Settings → Providers timed out and the model
picker showed only the default profile's models. WebUI profile switching is
per-client/cookie-scoped (#798), but two read-only paths resolved from the
process-global default profile:
- Facet A: /api/providers + /api/models did not apply the active profile's
.env around the read, so get_auth_status() / provider_model_ids() / custom
key lookups resolved the default profile's credentials. On a non-default
profile the auth probes could stall past the 30s frontend abort.
- Facet B: the /api/models disk cache was a single import-time
STATE_DIR/models_cache.json shared across every profile, while the cache
fingerprint is profile-specific -> a non-default profile rejected the shared
snapshot every read and cold-rebuilt (the slow serial-probe path).
Fix:
- api.profiles.profile_env_for_active_request(): applies the active
per-request profile's .env for the duration of the read (delegates to the
existing profile_env_for_background_worker used by streaming). No-op for the
default/root profile, so single-profile deployments are byte-identical.
- api.config._get_models_cache_path(): profile-keys the disk cache filename
(models_cache.<profile>.json) derived from the default path; default profile
keeps models_cache.json unchanged (no file migration).
- routes.py: wrap both GET handlers in profile_env_for_active_request.
- conftest: restore api.profiles._active_profile + clear request-profile TLS
after each test (a pre-existing isolation hole that profile-keyed cache
paths newly surface under sharding).
Tested: 10 new regression tests; live before/after on isolated servers shows
a non-default profile now surfaces its configured provider (deepseek) with
its own credentials + its own cache file, where master showed only the
default profile's providers and one shared cache.
Co-authored-by: nesquena-hermes <nesquena-hermes@users.noreply.github.com>
list_profiles_api() builds rows from cheap upstream helpers and skips find_alias_for_profile (which reads ~2.4GB of binaries from ~/.local/bin per dropdown open), with a 4s TTL cache + graceful fallback. 4700ms->250ms cold, 0ms cached. Full suite 8183 passed, Codex SAFE, Opus SHIP-IT. Supersedes #3772.
## Release v0.51.266 — Release IH (stage-r16)
One agent-authored APPROVED fix + two un-held streaming/SSE fixes.
### Fixed
| Issue/PR | Author | Fix |
|----------|--------|-----|
| #3635 (#3637) | @nesquena-hermes (nesquena APPROVED) | Composer profile chip reads `S.activeProfile` again — a #3331 regression keyed it on the loaded session's profile, so opening a cross-profile session made the chip disagree with the dropdown checkmark and misrepresent where the next message routes. #3331's project/session-op scoping is unaffected. |
| #3587 (#3605) | @rodboev | Reasoning persists to the correct intermediate assistant message in multi-turn tool flows. The index only advanced in `on_interim_assistant` (suppressed for contentless tool-call messages) → post-tool reasoning was mis-attributed; it now also advances at the `on_tool` boundary, guarded against over-increment. **(un-held — finding resolved)** |
| #2660 (#3558) | @franksong2702 | Session-event SSE no longer wakes every tab across profiles and never drops a relevant refresh — profile attached when known, root/`default` aliases stay unscoped, and the `maxsize=1` queue falls back to unscoped refresh-all on a profile-mismatch coalesce. **(un-held — both findings resolved)** |
### Gate
- Full pytest suite: **7770 passed, 0 failed**
- ESLint: CLEAN · ruff: CLEAN · browser-smoke: CLEAN
- Codex (regression): **SAFE TO SHIP** — chip matches dropdown/routing (no #3331 scoping regression), reasoning-index advance composes with the agent's tool/interim callback ordering, session-events coalesce safely with no dropped refresh and no profile data leak (`/api/sessions` still server-side filtered).
Co-authored-by: nesquena <nesquena@users.noreply.github.com>
Co-authored-by: rodboev <rodboev@users.noreply.github.com>
Co-authored-by: franksong2702 <franksong2702@users.noreply.github.com>
nesquena-hermes