Skip to content

fix(mistralai): emit chunked assistant content as content blocks - #3761

Open
HarianthK wants to merge 2 commits into
Arize-ai:mainfrom
HarianthK:mistral-content-chunks
Open

HarianthK wants to merge 2 commits into
Arize-ai:mainfrom
HarianthK:mistral-content-chunks

Conversation

@HarianthK

@HarianthK HarianthK commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Description

Mistral's reasoning models (magistral-*) return the assistant message content as a list of chunks rather than a string: a thinking chunk, which itself holds a list of text chunks, followed by a text chunk with the answer (AssistantMessageContent = Union[str, List[ContentChunk]] in the SDK). _get_attributes_from_chat_completion_message yielded that list straight into message.content. OpenTelemetry refuses it:

Invalid type ThinkChunk in attribute 'llm.output_messages.0.message.content' value sequence

so the output message on the span carried neither the reasoning nor the answer.

This keeps the string case exactly as it was and, for a list, emits message.contents.N blocks: each text inside a thinking chunk as a reasoning block and each text chunk as a text block, the same shape the LiteLLM, OpenAI and (as of #3754) Groq instrumentors use for reasoning. Chunk types with no text (images, references) are skipped, as the request side already does by serialising them.

Two tests with a mocked /v1/chat/completions response in the style of the existing ones: the magistral shape, and a plain string response pinned as unchanged. The first fails on main with the warning above and a KeyError on the reasoning block.

Checked with tox run -e test-mistralai, test-mistralai-latest and ruff-mypy-mistralai, all clean.

Streaming

The streaming accumulator kept the message content as a _StringAccumulator, which ignored any list-valued delta, so a streamed magistral answer reached the extractor with no content at all (as the review below found). A _ContentAccumulator now takes either text or chunk lists; a delta that continues the trailing chunk of the same type extends it, so token-by-token thinking and text fold into one reasoning block and one text block, the same shape the non-streamed response gives. A third test serves a magistral stream as server-sent events through respx and reads the blocks back; it fails without the accumulator change with a KeyError on the reasoning block.

Reasoning models (magistral) return the assistant content as a list of
chunks, a thinking chunk followed by the text of the answer. The
response extractor set that list straight on message.content, which
OpenTelemetry rejects as an attribute value, so the output message lost
both the reasoning and the answer.

A string is emitted as before. A list of chunks now goes out as
message.contents blocks: each text inside a thinking chunk as a
reasoning block, and each text chunk as a text block, the shape the
other instrumentors use for reasoning.

@feiiiiii5 feiiiiii5 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran this locally before commenting. Environment: mistralai + respx + OTel SDK in a fresh venv, packages taken from this PR's head tree via PYTHONPATH, -p no:randomly.

What I verified works

  • tests/.../test_reasoning.py → 2 passed. The magistral list-content shape maps to message.contents.0 (type=reasoning) and message.contents.1 (type=text), and the plain-string guard test confirms message.content is still used and no message.contents.* appears for a string reply.
  • Emitting content blocks rather than a raw list is the right target shape: message.contents is already what the OpenAI request extractor (_request_attributes_extractor.py:131) and the LangChain tracer (_tracer.py:885,889) use, so this brings the mistralai output side in line instead of inventing a new layout.
  • _flatten_content_chunks yields one block per inner thinking text, which matches how a multi-chunk thinking payload arrives.
  • No regression on this machine. The whole tests/ directory for this package gives the identical 7 pre-existing failures at the base commit (5da0a966) and at this head — same node IDs: test_entrypoint_for_opentelemetry_instrument plus the six *_streaming_* cases, which look like cassette/network artifacts in my environment rather than code failures — while the pass count goes 18 → 20. So the two new tests add coverage without breaking anything. (Both runs took ~7.5 min, same command, -p no:randomly.)

Question about scope, because the streamed path looks unaffected

The change is in _response_attributes_extractor.py, i.e. the non-streamed response object. For client.chat.stream(...), attributes come from _ChatCompletionAccumulator, and there:

  • _response_accumulator.py:68 declares the message field as content=_StringAccumulator();
  • the merge at _response_accumulator.py:147-149 is elif isinstance(self_value, _StringAccumulator): if isinstance(value, str): self_value += value, so a list-valued content delta is not appended and not replaced — it is dropped;
  • __iter__ at :131-133 only yields the key when str(value) is non-empty, so the accumulated message ends up with no content at all, and this extractor change never sees a list on that path.

So if magistral streams the same chunked content shape it returns in the non-streamed response, a streamed call would still export neither the reasoning nor the answer text. I did not verify what magistral actually puts in delta.content on a live stream, hence the question rather than a claim:

  1. Does magistral stream chunked content, or only plain strings per event?
  2. If it streams chunks, should this PR also make the accumulator list-aware (mirroring how tool_calls uses _IndexedAccumulator), or is the stream case deliberately left to a follow-up? A line in the PR description would be enough for reviewers to know they are not looking at a half-finished fix.
  3. If chunked content can reach the accumulator at all, a chat.stream() regression test alongside the two non-streamed ones would pin it.

Related, in case it is useful for deciding the shape once: I filed #3785 for the OpenAI instrumentor, where string fields that are simply absent from the declared schema (refusal, reasoning_content) survive as a plain str and are then overwritten per chunk, so only the final fragment is exported. Same underlying question — the declared schema is the single source of truth for what a stream can reconstruct — and fixing it per field in each package will keep drifting.

The streaming accumulator kept the message content as a string
accumulator, which ignored any delta that was a list, so a streamed
reasoning answer reached the extractor with no content at all. A content
accumulator now takes text or chunk lists, folding a delta into the
trailing chunk of the same type, so the accumulated message has the
same blocks a non-streamed response has. One streamed test served as
server-sent events.
@HarianthK

Copy link
Copy Markdown
Contributor Author

Thanks for the careful run. To your questions: magistral streams the same chunk shape it returns non-streamed (thinking chunks in delta.content as a list, then text chunks), so the accumulator did need to understand it, and I have done that here rather than leave it to a follow-up. _ContentAccumulator replaces the string accumulator for message content: text deltas are joined as before, chunk lists are collected, and a delta that continues the trailing chunk of the same type extends it, so the streamed span ends with one reasoning block and one text block like the non-streamed one. A third test serves a magistral stream as SSE through respx and checks the blocks; without the accumulator change it fails with a KeyError on the reasoning block. The PR description now says so too.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants