[Bug] Hosted OpenHands Cloud returns 401 token_not_found_in_db after restoring default managed LLM profile

[allenzhang0182]

Is there an existing issue for the same bug?

• I searched existing issues.

This appears closely related to previously reported/fixed managed LiteLLM credential issues, especially:

• Add managed LiteLLM key refresh and startup self-healing OpenHands#15022
• fix: add managed LLM key refresh endpoint OpenHands#15023
• Self-heal stale managed LLM keys before conversation startup OpenHands#15024
• [Bug]: OpenHands Cloud chats fail with 401 token_not_found_in_db (Invalid proxy server token passed) for all OpenHands provider models OpenHands#15280
• [Bug]: OpenHands#15282

However, I am reporting this because the issue is still reproducible on Hosted OpenHands Cloud after the managed-key refresh/self-healing work was merged and released.

Bug Description

Hosted OpenHands Cloud (app.all-hands.dev) cannot start conversations using the default OpenHands-managed LLM.

Every new conversation fails with:

401
Authentication Error, Invalid proxy server token passed.
Unable to find token in cache or LiteLLM_VerificationTokenTable
type: token_not_found_in_db

This occurs with the default hosted configuration:

Agent: OpenHands / CodeActAgent
Model: openhands/deepseek-v4.1-flash
Active profile: Default

I previously temporarily configured an OpenAI/Codex API key while testing another model.

That configuration was later removed, and the default OpenHands-managed configuration was restored.

The visible profile/settings now appear normal, but Hosted OpenHands continues to send a managed proxy credential that LiteLLM rejects.

This may therefore be a regression or an account state not covered by the managed LiteLLM key refresh/self-healing logic introduced for OpenHands/OpenHands#15022.

Expected Behavior

When the managed LiteLLM credential associated with a Hosted OpenHands Cloud account is stale or missing upstream, OpenHands should detect that state, refresh/rotate the managed credential, and start the conversation normally.

A new Hosted OpenHands conversation using:

openhands/deepseek-v4.1-flash

should successfully start without requiring the user to manually supply an API key.

Actual Behavior

This bug occurs in Hosted OpenHands Cloud, not in a locally installed SDK.

Current active profile information:

{
  "name": "Default",
  "model": "openhands/deepseek-v4.1-flash",
  "base_url": "https://llm-proxy.app.all-hands.dev",
  "api_key_set": false,
  "provider_connection_id": null
}

Relevant /api/settings state:

agent_kind: openhands
agent: CodeActAgent

agent_settings.llm.model:
openhands/deepseek-v4.1-flash

agent_settings.llm.api_key:
null

agent_settings.llm.base_url:
null

llm_api_key_set:
true

No user-supplied API key is currently configured in the active LLM profile.

Starting a new conversation and sending:

Reply exactly: OPENHANDS_OK

fails immediately with:

Error code: 401

Authentication Error, Invalid proxy server token passed.

Unable to find token in cache or LiteLLM_VerificationTokenTable

type: token_not_found_in_db
code: 401

The actual API key and token hash have intentionally been redacted.

I also tested the historical workaround described in OpenHands/OpenHands#15282:

Settings
→ LLM
→ leave the default configuration unchanged
→ Save
→ start a new conversation

This did not repair the credential.

A newly created conversation still fails with the same 401 token_not_found_in_db error.

Because this failure occurs entirely in Hosted OpenHands Cloud, there is no meaningful local python, pytest, uv, or pip command that reproduces it.

Steps to Reproduce

Hosted Cloud reproduction:

1. Sign in to https://app.all-hands.dev.

2. Use the OpenHands / CodeActAgent agent.

3. Use the active Default LLM profile.

4. Use:

   openhands/deepseek-v4.1-flash
   

5. Confirm the active profile reports:

   api_key_set: false
   provider_connection_id: null
   

6. Start a new Hosted OpenHands conversation.

7. Send:

   Reply exactly: OPENHANDS_OK
   

8. The conversation fails immediately with:

   401
   token_not_found_in_db
   LiteLLM_VerificationTokenTable
   

Additional recovery attempt:

1. Open Settings → LLM.
2. Leave the default configuration unchanged.
3. Click Save.
4. Start another new conversation.
5. Send the same minimal message.
6. The same 401 error still occurs.

This is a Hosted Cloud reproduction path. No local SDK installation is involved.

Acceptance Criteria

• A Hosted OpenHands Cloud account with a stale or missing managed LiteLLM credential can start a new conversation without receiving 401 token_not_found_in_db.
• Conversation startup detects an invalid managed OpenHands/LiteLLM credential and refreshes or rotates it automatically, as intended by the #15022/#15023/#15024 managed-key self-healing work.
• The repaired credential is persisted correctly and recognized by llm-proxy.app.all-hands.dev.
• The fix does not overwrite or delete user-provided BYOK/provider credentials.
• Existing Cloud workspaces do not need to be deleted or reset to recover the account-level managed credential.
• If automatic self-healing cannot repair the credential, the Hosted UI provides a clear recovery path or actionable error rather than repeatedly returning token_not_found_in_db.

Installation Method

Hosted OpenHands Cloud:

https://app.all-hands.dev

No local OpenHands SDK installation is involved.

If you selected "Other", please specify

Hosted OpenHands Cloud / SaaS.

SDK Version

Not applicable.

This failure occurs on Hosted OpenHands Cloud rather than a locally installed OpenHands SDK.

Version Confirmation

The issue is reproducible on the currently deployed Hosted OpenHands Cloud service as of 2026-09-19.

Python Version

Not applicable.

No local Python/SDK execution is involved in reproducing the issue.

Model Name (if applicable)

openhands/deepseek-v4.1-flash

Operating System

Windows / Chrome browser.

The failure appears account/backend related rather than OS-specific.

Logs and Error Messages

Redacted error:

Error code: 401

Authentication Error, Invalid proxy server token passed.

Received API Key = [REDACTED]
Key Hash (Token) = [REDACTED]

Unable to find token in cache or LiteLLM_VerificationTokenTable

type: token_not_found_in_db
param: key
code: 401

Sensitive values have intentionally been removed.

Minimal Code Sample

Not applicable.

This is reproducible directly in the Hosted OpenHands Cloud UI and does not require SDK code.

Minimal Hosted Cloud input:

Reply exactly: OPENHANDS_OK

Screenshots and Additional Context

This looks very similar to the previously documented managed LiteLLM key failure in:

• Add managed LiteLLM key refresh and startup self-healing OpenHands#15022 — managed LiteLLM key refresh and startup self-healing
• fix: add managed LLM key refresh endpoint OpenHands#15023 — managed LLM key refresh endpoint
• Self-heal stale managed LLM keys before conversation startup OpenHands#15024 — self-heal stale managed LLM keys before conversation startup
• [Bug]: OpenHands Cloud chats fail with 401 token_not_found_in_db (Invalid proxy server token passed) for all OpenHands provider models OpenHands#15280
• [Bug]: OpenHands#15282

The earlier #15282 report was explicitly described as a per-account managed LiteLLM credential problem rather than a normal OpenHands API key problem.

The managed-key fix was subsequently merged and released, so the fact that the same failure is still reproducible may indicate:

1. a regression in the startup self-healing path,
2. a managed credential state not covered by the existing repair logic, or
3. inconsistency between the Hosted OpenHands account's stored managed credential and the LiteLLM verification database.

Current visible configuration appears restored to the hosted default:

Agent: OpenHands / CodeActAgent
Profile: Default
Model: openhands/deepseek-v4.1-flash
Profile api_key_set: false
Provider connection: null
Managed proxy: https://llm-proxy.app.all-hands.dev

The historical Settings → LLM → Save workaround also fails to recover the account.

If account-specific information is required to investigate or repair the managed credential, please provide a private support channel where I can send identifiers such as my Hosted OpenHands account email, organization ID, or conversation ID.

I intentionally have not posted publicly:

• API keys or partial API keys
• token hashes
• Authorization headers
• cookies/session credentials
• OpenHands account email
• organization ID
• repository names
• workspace/conversation IDs
• HAR files

I also have an existing Hosted OpenHands workspace containing uncommitted development files.

Please avoid deleting or resetting existing workspaces as part of account-level credential remediation unless absolutely necessary.

Scope / Routing Note

This is specifically a Hosted OpenHands Cloud (app.all-hands.dev) managed-credential failure, not a local openhands-sdk execution failure.

The SDK repository's automated readiness check requests a local python, pytest, uv, or pip reproduction command, but such a command is not applicable to this Hosted Cloud failure.

If OpenHands/software-agent-sdk is not the correct tracker for this Hosted Cloud credential regression, please transfer or route this issue to the appropriate OpenHands Cloud/backend tracker rather than treating the absence of a local SDK command as a reproduction failure.

[github-actions]

✅ ready-for-dev label applied

This issue meets the readiness criteria and is ready for contribution.
A contributor can now open a PR that links this issue (Fixes #<number>).

This is an automated check — no AI was used to generate this comment.

[allenzhang0182]

Updated the issue body with the full Hosted Cloud reproduction, exact acceptance criteria, references to #15022/#15023/#15024, and confirmation that the historical Settings → LLM → Save workaround still results in 401 token_not_found_in_db.

[pMertDogan]

Same issue

[aivong-openhands]

Cross-reference: enterprise-side heal verification confirms this is the runtime-agent gap

This matches a caveat surfaced while verifying the enterprise-side self-heal (OpenHands/enterprise#437 — "heal stale org-level BYOR LLM key on conversation start", now merged). That PR heals the app-server / DB path (clears the stale org._llm_api_key and rotates a fresh managed sk-… key on conversation start), but a Replicated-VM verification run showed the fix does not reach the sandbox runtime agent's LLM on the triggering conversation — which is exactly the token_not_found_in_db / stale-managed-credential symptom reported here.

Observed timeline from that run (UTC), broken state re-injected, then a real conversation started through the UI:

19:15:48.46  clear_stale_org_level_llm_key                       (heal fired)
19:15:48.60  managed_llm_key_refresh:healed_after_clear  rotated
19:15:50.86  app_conversation_start:building_start_request  conv=4d4e6921...
19:15:51/52  litellm 401 Received=dumm****odel   <-- triggering convo's runtime agent
             STILL used the stale key AFTER the DB was healed

• DB confirmed healed immediately after (org._llm_api_key=None, member sk-e…, a direct completion returned 200), yet conversation 4d4e6921 (the one that triggered the heal) still 401'd because its sandbox runtime agent used the stale key.
• The next conversation (41659260) succeeded: LiteLLM POST /chat/completions 200 OK, no stale 401.
• Also observed a transient KeyNotFoundError key=92ca… not found in db for a freshly rotated managed key around the same window.

So the real-world UX is "the first conversation after the fix 401s once, then self-heals for every subsequent conversation." The remaining gap is that a refreshed/rotated managed key needs to propagate into the conversation's runtime agent LLM (the sandbox), not just the app-server fallback path.

Likely-related root-cause launch-path bug in this repo: #4533 ("Conversation launch drops LLM api_key when the seeded default agent profile is active"), where the launched conversation persists agent.llm.api_key = null even though settings report the key is set.

This comment was created by an AI agent (OpenHands) on behalf of @aivong-openhands.

[aivong-openhands]

Proposed resolution plan (shared root cause with #4533)

After comparing this with #4533, both are symptoms of one root cause: the sandbox runtime agent's LLM credential is snapshotted at launch (the secret value is copied into the conversation / base_state.json) instead of being resolved by reference at call time.

Issue	Snapshot contains	LiteLLM error
#4533	api_key = null (dropped on the seeded default launch path)	AuthenticationError: api_key must be set
this issue (#5189)	a stale managed key (already rotated/healed in the DB)	401 token_not_found_in_db

Same mechanism (launch-time copy), opposite failure — one drops the key, the other carries an outdated one.

Plan

1. Durable fix — reference-only credential delivery (Design: reference-only credentials and safe runtime delivery #4288). Conversation launch drops LLM api_key when the seeded default agent profile is active (litellm AuthenticationError) #4533's targeted patch (PR fix(agent-server): backfill redacted LLM api_key on agent_settings la… #4539) was closed unmerged precisely because the correct fix is Design: reference-only credentials and safe runtime delivery #4288. Resolving the managed credential by reference at call time fixes both: the key can't be dropped (Conversation launch drops LLM api_key when the seeded default agent profile is active (litellm AuthenticationError) #4533) and a healed/rotated key is always current (this issue -> no more "first conversation 401s once"). This is the primary fix and should close the runtime half of [Bug] Hosted OpenHands Cloud returns 401 token_not_found_in_db after restoring default managed LLM profile #5189.
2. Near-term mitigation specific to this issue — refresh-on-401 in the runtime LLM path. When LiteLLM returns token_not_found_in_db, re-resolve the managed key and retry once. This removes the "first conversation fails once, then self-heals" UX before the full Design: reference-only credentials and safe runtime delivery #4288 architecture lands. (Conversation launch drops LLM api_key when the seeded default agent profile is active (litellm AuthenticationError) #4533 does not need this — a null key can't be retried into existence.)
3. Confirm the enterprise/rotation side. The transient KeyNotFoundError key=92ca... not found in db observed during verification points at a rotation race tracked in OpenHands/enterprise#439 (PR OpenHands/enterprise#440). [Bug] Hosted OpenHands Cloud returns 401 token_not_found_in_db after restoring default managed LLM profile #5189 likely has two contributors — the SDK snapshot staleness and that rotation race — so verifying Fix websocket infinite loop on user disconnect #439/Add Chromium binary check and auto-install if not available #440 ensures a freshly rotated key is present in LiteLLM before the runtime uses it.

TL;DR

• Don't write a targeted patch for Conversation launch drops LLM api_key when the seeded default agent profile is active (litellm AuthenticationError) #4533 — it folds into Design: reference-only credentials and safe runtime delivery #4288.
• This issue ([Bug] Hosted OpenHands Cloud returns 401 token_not_found_in_db after restoring default managed LLM profile #5189) is the one that needs an explicit fix: Design: reference-only credentials and safe runtime delivery #4288 for the durable path, optionally a runtime refresh-on-401 retry as a bridge, plus confirming enterprise#439/#440.

This comment was created by an AI agent (OpenHands) on behalf of @aivong-openhands.