Skip to content

Fix RubyLLM 2.x tool-calling turns and cache token counts - #243

Merged
David Elner (delner) merged 6 commits into
mainfrom
fix/ruby-llm-2-turn-spans
Nov 16, 2026
Merged

David Elner (delner) merged 6 commits into
mainfrom
fix/ruby-llm-2-turn-spans

Conversation

@delner

@delner David Elner (delner) commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

A fork of #230 to fix CI, get this merged; thanks Ritchie C (@infinityrobot)

RubyLLM 2.x runs the whole tool loop inside Chat#complete, so tracing that method alone drops the final answer from the chat span and records only the last request's tokens. This traces each model request through Chat#generate, groups a turn under a ruby_llm.complete span, captures the 2.x cache-token fields, and avoids mutating tool schemas while recording metadata. The 1.x tracing path is preserved.

Fixes #226.

ruby_llm 2.x runs the whole tool loop inside Chat#complete (step until
complete?), so wrapping complete gave one span per turn. On tool-calling
turns that span recorded the tool-call request as its output and only the
last request's tokens.

On 2.x, trace Chat#generate instead, which step calls once per model
request. 1.x keeps wrapping complete, which recurses after tool calls.
The variant is picked when the patch is applied, by whether the class
defines a public generate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Tracing generate() alone made a tool-calling turn's chat and tool spans
siblings, so a caller without its own span got one trace per request.
On 2.x, complete() now opens a task span that parents them. It carries
the turn's input and final output but no metrics, since each request's
tokens are on its ruby_llm.chat span. 1.x is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Filtering to ruby_llm.chat alone let a double-wrapped complete() pass the
idempotency test. Also expect exactly one ruby_llm.complete span on 2.x
and none on 1.x.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ruby_llm 2.x renamed the cache counts to cache_read_tokens and
cache_write_tokens, so cached input never reached prompt_tokens and a
cached request reported only its few uncached tokens. Read either name in
one helper shared by the streaming and non-streaming paths.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
format_tool_schema deleted strict and additionalProperties from the
tool's parameters hash in place unless it was frozen. That hash is the
tool's cached schema, so instrumenting a chat stripped the constraints
from the requests RubyLLM sends to the provider, for every later instance
and even for untraced chats. Build trimmed copies instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
faraday-net_http 3.5.0 (2026-10-11) is past faraday 2.14.4's `< 3.5` cap,
so the ruby_llm appraisals backtracked to faraday 2.0.0, which cannot load
the net_http adapter, and every recorded RubyLLM test errored. Require
faraday >= 2.14 there.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@delner
David Elner (delner) merged commit efebc77 into main Nov 16, 2026
10 checks passed
@delner
David Elner (delner) deleted the fix/ruby-llm-2-turn-spans branch November 16, 2026 16:41
@delner

Copy link
Copy Markdown
Collaborator Author

Ritchie C (@infinityrobot) this was released in 0.6.1. Thanks for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ruby_llm: on RubyLLM 2.x, tool-calling turns drop the final answer and undercount tokens

2 participants