Skip to content

Claude backend: WebSearch and WebFetch are deferred tools, so the agent rarely searches the web (speculative) #459

Description

@alex-clickhouse

Summary

Status: speculative. We do not know yet if more web search helps the agent, or if the token cost is acceptable.

On the Claude backend, the agent almost never calls WebSearch or WebFetch. The Claude Code CLI marks both tools as deferred. The model gets only the tool names. Before it can call a deferred tool, it must call ToolSearch to load the schema. Nerve sets alwaysLoad only on the nerve MCP server (nerve/agent/tools/claude_sdk_adapter.py), so the built-in web tools stay deferred.

Evidence

Langfuse traces from one production deployment (direct Anthropic API, no proxy):

Tool span Count, all time
Bash 45,433
ToolSearch 34
WebFetch 15
WebSearch 2
  • ToolSearch loaded WebSearch 4 times and WebFetch 9 times. The agent cannot call a deferred tool before it loads it, so these counts agree with the call counts.
  • The tracing is complete. The langsmith PreToolUse hook traces all tools (matcher=None), and the two WebSearch calls returned normal results.
  • The workload is also a cause. This deployment does mostly code, GitHub, and database work. Its curl calls go to known APIs, not to search engines.
  • Codex turns are not traced, so this issue is about the Claude backend only.

CLI behavior

Examined in the CLI bundled with claude-agent-sdk 0.2.158:

  • WebSearch and WebFetch have shouldDefer: true.
  • The CLI sends the deferred tool names in a deferred_tools_delta message attachment, not in the system prompt. The Nerve system prompt does not remove them.
  • ENABLE_TOOL_SEARCH sets the mode. A false value loads all tools at the start.
  • The CLI turns tool search off when ANTHROPIC_BASE_URL is not a first-party Anthropic host and ENABLE_TOOL_SEARCH is not set. Thus proxy mode (CLIProxyAPI) and a gateway ANTHROPIC_BASE_URL already load all tools at the start.
  • There is no setting to load one built-in tool at the start. tengu_non_deferrable_builtins in the bundle is a server-side feature flag.

Options

  1. Set ENABLE_TOOL_SEARCH=false in ClaudeBackend._build_env. All tools load at the start, including the tools of all external MCP servers. This adds input tokens to each turn.
  2. Tell the model in the prompt templates to load WebSearch and WebFetch with ToolSearch when it needs current information. The cost is low, but the model can ignore it.
  3. Add a config key (for example agent.tool_search) that sets ENABLE_TOOL_SEARCH, so operators can choose.
  4. Make no change. The agent's work may not need web search often.

Open questions

  • Is low web search use a problem? We have no reports of wrong answers caused by missing search.
  • What is the token cost of option 1 with a typical set of MCP servers?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions