The Codex OAuth onboarding worker introduced in #1652 had a cancel-vs-worker
race: a `cancel_onboarding_oauth_flow` request that arrived while the worker
was mid-network-call (between the `live = dict(...)` snapshot and the next
status check) would be silently overridden:
1. User clicks Cancel → server sets flow.status = "cancelled" and drops
sensitive lifecycle fields under the lock.
2. Worker is mid-`_poll_codex_authorization` / `_exchange_codex_authorization`
using the local `live` snapshot it captured before the cancel.
3. Worker calls `_persist_codex_credentials(...)` — auth.json gets written.
4. Worker calls `_set_flow_status(flow_id, "success")` — overrides the
cancelled status.
Net effect: the user's explicit cancel is ignored, credentials are persisted,
and the UI reports success. Reproduced with a behavioural harness that drove
a real worker thread against patched network helpers and confirmed:
pre-fix : flow status `success`, auth.json written despite cancel
post-fix: flow status `cancelled`, auth.json NOT written
The fix re-checks the flow status under `_OAUTH_FLOWS_LOCK` after the token
exchange completes and before persisting. If the status is no longer
`pending`, the worker exits without persisting credentials and without
overwriting the terminal status.
Regression test `test_cancel_during_token_exchange_does_not_persist_credentials`
drives the worker against threading.Event-gated network stubs to reproduce
the race deterministically and lock the new invariant.
Trace verified against fresh hermes-agent tarball — credential_pool entry
shape (`auth_type=oauth`, `source=manual:device_code`, `priority=0`, base_url)
remains compatible with `agent.credential_pool.load_pool("openai-codex")` and
the agent CLI's `_save_codex_tokens` legacy fallback path.
Tests:
- 10/10 in tests/test_issue1362_codex_oauth_onboarding.py
- Full suite: 4230 passed, 57 skipped, 3 xpassed, 0 failed in 33.82s
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Constituent PRs:
#1637 by @Michaelyklam — protect raw pre from glued-bold lift (closes#1451)
#1639 by @bergeouss — macOS auto-scroll race + custom:* provider list (closes#1360, #1619)
#1642 by @nesquena-hermes — YAML/JSON/diff code block newlines (closes#1618, #1463)
Opus advisor SHIP verdict on stage-295. One observation absorbed:
- api/config.py:2533 dead-code comment per Opus (defensive belt-and-braces
for #1619 fallback; load-bearing fix is in routes.py /api/models/live)
PR #1641 (Michaelyklam parallel-discovery duplicate of #1642) closed as
superseded; UI media adopted with co-author trailer.
4245 → 4255 tests passing (+10).
#1360 — On macOS WKWebView, trackpad momentum scrolling fires scroll
events that interleave with the _programmaticScroll setTimeout(0) guard.
A mid-momentum scroll event either gets swallowed (_programmaticScroll
still true) or falsely reports nearBottom (momentum hasn't settled),
keeping _scrollPinned=true and snapping the viewport back down.
Fix: rAF-debounce the scroll listener so the nearBottom check runs at
the next paint frame when the browser's scroll position has settled.
Added a hysteresis counter requiring 2 consecutive near-bottom samples
before re-pinning, preventing accidental re-pin during deceleration.
#1619 — When a custom:* provider (e.g. custom:relay via custom_providers)
has models that overlap with auto-detected models from base_url /v1/models,
the dedup logic at config.py:2263 skipped them all. The named custom
group ended up empty, and the continue at line 2334 silently discarded
the auto-detected models. Result: only the default model appeared.
Fix 1 (config.py): When custom:* named group has 0 models after dedup,
fall back to auto_detected_models_by_provider instead of dropping them.
Fix 2 (routes.py): Extended /api/models/live fallback to handle
custom:* slugs (not just bare "custom") for both custom_providers
config lookup and base_url live fetch.
Closes#1633. STATE_DIR/models_cache.json was persisted across server
restarts without any version stamp, so a Docker container update from
version A to B read the cache file written by version A — users saw
stale picker contents (missing models, phantom provider groups) for
up to 24 hours until either the TTL expired, an unrelated provider
edit triggered invalidate_models_cache(), or they manually deleted
the file.
Reporter Deor (Discord) updated to v0.50.292 — which contained fixes
for #1538, #1539, and #1568 — did a hard refresh and cleared site
data, and still saw byte-for-byte identical picker contents because
the server kept reading the v0.50.281 cache file off the host-mounted
state volume.
Fix:
* _save_models_cache_to_disk() stamps payloads with _webui_version
(resolved lazily from api.updates.WEBUI_VERSION via sys.modules
lookup to avoid the api.config <-> api.updates circular import)
and _schema_version = 2.
* New _is_loadable_disk_cache() validator checks both stamps in
addition to shape. Mismatch on either field rejects the load.
* _load_models_cache_from_disk() calls the new validator and
strips the disk-only metadata before returning, so the rest of
the code sees the same shape it always did.
* _is_valid_models_cache() kept loose (shape-only) so in-memory
cache writes that never touch disk don't fail validation.
Schema version is independent of the WebUI version stamp so future
cache-shape changes can invalidate older releases without relying
on a tag bump alone.
Early-init edge case (api.updates not yet loaded) skips the version
check rather than wedging the boot — at worst an unstamped file is
written once and rejected on the next call.
Updated existing tests/test_model_cache_metadata.py to use subset/
round-trip semantics rather than byte-for-byte equality, since the
disk payload now has additional stamps. The four response-shape
fields still round-trip verbatim; the load result is unchanged
(stamps stripped). 19 new regression tests.
4180 -> 4199 tests pass.
SHOULD-FIX: rate-limit _repair_stale_pending repair-firing telemetry. Switch
from unconditional logger.warning to age-keyed: WARNING when pending_age <
5min (the diagnostically valuable race window — actual leak-path candidates
that slipped past the grace guard) and DEBUG for the long-tail (orphaned
sidecars from prior process lifetimes). Prevents reconnect loops on stuck
sessions from flooding the log while preserving the diagnostic signal we
want for tuning _REPAIR_STALE_PENDING_GRACE_SECONDS empirically.
NIT: _LOCAL_SERVER_PROVIDERS expanded with lm-studio (hyphenated alias used
in some custom_providers configs and already recognized at api/config.py:2189
for SSRF host trust) and localai (LocalAI project). Test parametrize expanded
from 7 to 11 names, also covering pre-existing koboldcpp and textgen for
symmetry. +4 regression tests.
NIT (docs): CHANGELOG callout for the RFC1918 behavior change. Internal-
network OpenAI-compatible proxies now preserve the model prefix on private-IP
base_urls. Documented the migration path: configure as a custom_providers
entry to bypass the local-server detection.
NIT (deferred, optional): narrowing the heuristic to is_loopback only is
left as future work; the broader scope was an explicit goal in the bug
body and Opus flagged it as SHOULD-DISCUSS-but-not-block.
4184 -> 4188 passing. 0 regressions. ~10 LOC absorbed total.
Closes#1623 — Lower SSE app heartbeat from 30s to 5s at every long-lived
handler (main agent, terminal, gateway-watcher, approval-poller, clarify-poller).
Kernel TCP keepalive declares peer dead at 25s worst-case (10s KEEPIDLE +
5s KEEPINTVL * 3 KEEPCNT, added v0.50.289 #1581). 30s app heartbeat let the
kernel tear sockets down on flaky networks before the app sent its first
keepalive byte — drops at ~10s during long thinking phases. New named
constant _SSE_HEARTBEAT_INTERVAL_SECONDS=5; regression test pins the
inequality (app_heartbeat * 2 <= kernel_window) so future tuning can't
re-introduce the misalignment.
Closes#1624 — Add 30s grace period to _repair_stale_pending() trigger.
Without it, any narrow race between the streaming thread clearing
pending_user_message and STREAMS.pop(stream_id) produces a false-positive
'Previous turn did not complete.' marker on a turn that finished correctly
(reproducible after every command-approval turn). Defense-in-depth, not
the root-cause fix — the actual streaming-thread leak path is tracked
separately. Falsy pending_started_at (legacy sidecars) treated as
'old enough' so legitimate legacy-data recovery still works. Plus
logger.warning telemetry on every legitimate repair so the next batch of
user reports tells us whether the underlying race still fires.
Closes#1625 — Local model servers (LM Studio, Ollama, llama.cpp, vLLM,
TabbyAPI, koboldcpp, textgen-webui) now keep the full HuggingFace-style
model id (e.g. 'qwen/qwen3.6-27b' instead of stripped 'qwen3.6-27b'). New
_LOCAL_SERVER_PROVIDERS set + _base_url_points_at_local_server() loopback/
RFC1918 heuristic — either signal triggers no-strip. Backward compat
preserved for OpenAI-compatible proxies on public hosts (LiteLLM at
litellm.example.com still strips openai/gpt-5.4 -> gpt-5.4). Updated the
existing #230/#433 test to reflect that #1625 supersedes the strip-on-custom
rule for loopback hosts (see api/config.py and test_model_resolver.py
docstring update). Reported by @akarichan8231 in Discord on 2026-05-04.
42 regression tests across:
tests/test_issue1623_sse_heartbeat_alignment.py (3)
tests/test_issue1624_repair_stale_pending_grace.py (9)
tests/test_issue1625_local_server_model_id_preservation.py (30)
4142 -> 4184 passing. 0 regressions.
SHOULD-FIX #1 (renamed-root client cross-alias): drop strict-equality client
filter at static/sessions.js:1853. Server-side _profiles_match cross-aliases
'default'-tagged rows to a renamed root 'kinni'; the strict-equality client
would reject them, dropping every legacy session for renamed-root users. The
server is now solely authoritative for profile scoping.
SHOULD-FIX #2 (messaging-source dedupe ordering): _keep_latest_messaging_session_per_source
now runs AFTER the profile filter at api/routes.py:2078. Before, it ran on
the merged-cross-profile list with profile-blind keys, discarding the older
profile's row across profiles before the scope filter — leaving zero rows for
any messaging identity the active profile shared with another profile.
NIT #3: _projects_migrated flag now set only AFTER successful save_projects.
NIT #4: cleaned dead test code in test_is_root_profile_invalidation_drops_stale.
NIT #5: _create_profile_fallback's clone_from=='default' literal now routes
through _is_root_profile() for parity with the 5 other callsites.
+2 regression tests pin the SHOULD-FIX shapes:
- test_keep_latest_messaging_runs_after_profile_filter (source-string ordering)
- test_static_sessions_js_trusts_server_profile_scoping (no client re-filter)
4173 -> 4175 tests pass. 0 regressions.
Closes#1611 — /api/sessions filters by active profile by default; ?all_profiles=1
opt-in for aggregate views; new _profiles_match() helper honours renamed-root
cross-aliasing; static/sessions.js drops the s.is_cli_session bypass; toggle-on
re-fetches with all_profiles=1 instead of slicing client-cached rows.
Closes#1612 — new _is_root_profile() central helper consults list_profiles_api()
for is_default=True matches alongside the legacy 'default' alias. Replaces five
literal-default callsites in api/profiles.py. Memoized with explicit invalidation
hooks at create + delete. Sticky active_profile file write now stores '' for
renamed root, consistent with the legacy empty==root contract.
Closes#1614 — projects carry a profile field stamped at create-time;
/api/projects filters by active profile; /api/projects/{create,rename,delete}
and /api/session/move reject ops on cross-profile projects with 404; new
_PROJECTS_MIGRATION migration in load_projects() back-tags untagged projects
from any session that uses them, fall back to 'default'; ensure_cron_project
keys lookup by (name, profile) so each profile gets its own Cron Jobs project.
31 regression tests (9+11+11) pin the renamed-root resolution, server-side
profile scoping shape, helper invariants, cross-alias matching, migration
behavior, and active-profile guards on every project mutation endpoint.
4148 tests pass.
Reporter: @stefanpieter
Co-authored-by: stefanpieter <noreply@github.com>
Remove the try/except Exception wrapper around
cron_profile_context_for_home(...).__enter__() in _run_cron_tracked.
A silent fallback to ctx=None would leave the worker thread unpinned
against process-global HERMES_HOME, silently corrupting cross-profile
state — the same class of bug as #1573.
Add regression test to catch any future re-introduction.
Switch the per-turn duration fallback from `is not None` to a truthy check so
None, missing-attr, and an explicit 0 all uniformly fall back to time.time().
Without this, a 0 timestamp (e.g. via a buggy migration or manual file edit)
would yield `time.time() - 0` ≈ wall-clock-since-epoch, displaying nonsense
like 'Done in 56 years 4 months ...'. In practice pending_started_at is always
set via int(time.time()) so this is a hardening fix, not a live-bug fix.
Also drop the brittle source-string assertion in the regression test that
pinned the literal expression. The behavioural test
test_done_handler_persists_duration_on_last_assistant_message already proves
the duration field is set; pinning the source line broke twice during the
v0.50.290 release pipeline alone (Opus tightening + maintainer revert).
Fixes#1595
Signed-off-by: Sanjay Santhanam <51058514+Sanjays2402@users.noreply.github.com>
Closes#1579.
api/updates.py was building the GitHub compare URL from local HEAD short SHA:
repoUrl + '/compare/' + curSha + '...' + newSha
where curSha = `git rev-parse --short HEAD`
Whenever local HEAD diverges from upstream — unpushed work, dirty stage
branches, forks, in-flight rebases, release-time merge commits whose SHA
only lives in the maintainer's local history — the compare URL points at
a SHA github.com has never seen and returns the standard 404 page.
Reporter (@ai-ag2026) observed:
https://github.com/nesquena/hermes-webui/compare/c660c7f...86cb22e
→ 404 because c660c7f was an unpushed local commit.
The right base is `git merge-base HEAD <compare_ref>` — the most recent
commit local and upstream share. Since `git fetch` succeeded just before,
the merge-base is guaranteed to exist on the upstream GitHub repo.
Behavior matrix:
Pure-behind clone (no local commits): merge-base == HEAD; URL unchanged.
Behind + local-only commits (#1579): merge-base != HEAD; URL points at
public ancestor instead of local HEAD.
merge-base failure (shallow clone): current_sha=None; JS link guard
suppresses link rather than emitting
a known-broken URL.
Also hardens static/ui.js: reset the link's href and display:none on every
banner render, so a stale link from a prior render can't survive a re-render
where the new payload has current_sha=null.
Tests:
- test_current_sha_is_merge_base_not_local_HEAD — reporter's scenario
- test_current_sha_equals_HEAD_when_no_local_commits — backward compat
- test_current_sha_falls_back_to_None_when_merge_base_fails — defensive
- test_whats_new_link_resets_display_and_href_on_every_render
- test_whats_new_link_suppressed_when_curSha_falsy
- test_reporter_url_shape_no_longer_produces_invalid_compare_url
4094 → 4100 passing. 0 regressions.
Three small follow-ups from the review:
1. Remove the over-broad except Exception around get_active_hermes_home()
in _handle_cron_run. The function is in-memory dict reads + one
Path.is_dir() stat — if it raises from inside a request handler,
api.profiles is in a state we shouldn't be making cron decisions in.
A silent fallback to _profile_home=None re-introduces the exact
bug #1573 fixes (worker thread runs unpinned against process-global
HERMES_HOME). Better to 500 the request than risk silent cross-
profile state corruption.
2. Add a thread-safety note on os.environ mutation in api/profiles.py
explaining why _cron_env_lock is sufficient — CPython env-var
assignment is GIL-protected at the bytecode level but the multi-step
read-modify-write pattern (snapshot prev → assign new → restore on
exit) is not atomic without explicit serialization. The lock makes
the entire context-manager body run-to-completion serially, including
any subprocess.Popen() calls inside run_job() that inherit the env.
3. New regression test (test_cron_run_does_not_silently_swallow_profile_resolution_errors)
pinning the no-silent-fallback contract via source-level assertion.
Catches future re-introduction of the over-broad except clause.
Co-authored-by: kowenhaoai <kowenhaoai@users.noreply.github.com>
Manfred