Skip to content

fix(agent-server): evict idle Docker conversation runtimes #5196

Description

@neubig

Actual Behavior

Start the SDK Agent Server with uv run openhands-agent-server and conversation_runtime: docker. When the outer Agent Server uses this mode, every accessed conversation gets a dedicated agent-server-conversation-* container. DockerConversationRegistry removes owned containers at outer-server startup, explicit deletion, or shutdown, but it has no idle-runtime eviction loop. The inner Agent Server evicts its in-process conversation after conversation_idle_ttl_seconds, while its container continues running indefinitely.

On the OSS Agent Canvas VM this left 32 containers running for up to 32 hours. Each had an approximately 169 MB writable layer, filling the 100 GB system disk and leaving only 9.9 MB free.

Steps to Reproduce

  1. Start Agent Server with conversation_runtime: docker and the default conversation idle TTL.
  2. Open several conversations and allow them to reach a terminal or idle state.
  3. Wait longer than the configured idle TTL.
  4. Observe that the inner conversation is evicted but each agent-server-conversation-* container remains running and continues consuming memory and disk.

Desired Behavior

The Docker conversation registry should stop an inactive runtime after the configured idle TTL while preserving the conversation, provisioning identity, workspace, and persisted event history. A later interaction should reprovision the runtime normally. Running conversations and recently accessed runtimes must not be stopped.

Acceptance Criteria

  • Docker-backed runtimes are stopped after the configured conversation idle TTL when their conversation is in a terminal or idle state.
  • Conversations with a running agent are never evicted solely because no outer HTTP request arrived during the TTL.
  • Accessing a runtime refreshes its activity deadline.
  • Reopening an evicted conversation reprovisions its container and preserves persisted history and workspace contents.
  • Registry shutdown cancels and awaits the eviction task without leaking tasks or containers.
  • Disabling the configured idle TTL disables runtime eviction.
  • Tests cover terminal-runtime eviction, running-runtime retention, access refresh, disabled eviction, and clean shutdown.

OpenHands AI triage

The following comments and acceptance criteria were added by the OpenHands AI agent.

Triage

Confirmed against openhands-agent-server/openhands/agent_server/docker_runtime/registry.py: DockerConversationRegistry only removes owned containers at startup cleanup, explicit delete, or shutdown — it has no idle-runtime eviction loop, so a container outlives the inner conversation that is evicted after conversation_idle_ttl_seconds. The fix is scoped to the Docker runtime registry adding an idle-eviction loop keyed on the existing Config.conversation_idle_ttl_seconds (null already means "never evict"). It must stop an inactive runtime while preserving the provisioning identity, workspace bind mount, and persisted history, then let a later access reprovision normally, and must skip running/subscribed conversations.

Non-goals: no change to inner-server in-process eviction, no change to the provisioning/persistence format, no new config knobs, and no change to the non-Docker (local) runtime path.

Acceptance Criteria

  • With conversation_runtime: docker and a set conversation_idle_ttl_seconds, a container whose conversation is in a terminal or idle state and unaccessed for at least the TTL is stopped (its agent-server-conversation-* container is no longer running) and no longer held in the registry's in-memory container map.
  • A conversation whose agent is actively running is never stopped solely because no outer HTTP/WebSocket request arrived during the TTL.
  • Each access that resolves or starts a runtime (conversation/proxy routes and socket proxies) refreshes that conversation's activity deadline.
  • Accessing a previously evicted conversation reprovisions its container and serves the preserved persisted event history and workspace contents unchanged.
  • Setting conversation_idle_ttl_seconds to null disables eviction entirely (no container is stopped for idleness).
  • Registry shutdown() cancels and awaits the eviction task, leaving no orphaned task and no owned container still running.
  • Automated tests cover terminal/idle-runtime eviction, running-reactivity retention, access refresh, disabled eviction, and clean shutdown.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:highFor bugs, affecting nearly all users and degrading performance or UX.ready-for-devIssue meets development readiness criteria

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions