feat(meai): map mid-conversation system messages for Opus 4.8#207
feat(meai): map mid-conversation system messages for Opus 4.8#207PederHP wants to merge 5 commits into
Conversation
TomerAberbach
force-pushed
the
feat/meai-mid-conversation-system-messages
branch
from
a9e0271 to
79f3ae9
Compare
…-fallback handler (anthropics#56) Helpers/StainlessHelperHeader.cs (internal) carries the lowercase header key, the closed value vocabulary, and MergedValue() — comma-appends to any existing tag (case-insensitive, deduped, single line). BetaRefusalFallbackHandler now appends fallback-refusal-middleware to every applicable request — original and each hop — folded into the same headers copy the betas-append already does (AppendBetas → WithHandlerHeaders). BetaToolRunner now references the constant instead of an inline literal.
The MEAI IChatClient mapping previously hoisted every ChatRole.System
message into the top-level `system` property regardless of position, so
the Opus 4.8 mid-conversation system message feature was unreachable.
A ChatRole.System message is now emitted as a `{"role":"system"}` message
at its position when the placement is valid (immediately follows a user
turn and either ends the array or precedes an assistant turn) and the
resolved model supports it (claude-opus-4-8). Leading system messages,
invalid positions, and unsupported models fall back to the top-level
`system` property, preserving prior behavior. Consecutive system messages
are merged into one, per the API constraint.
Applied to both the stable and beta IChatClient mappings, with shared
tests covering valid placement, mid-list placement, hoist on unsupported
model, hoist after an assistant turn, consecutive-merge, leading-only,
and cache control.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PederHP
force-pushed
the
feat/meai-mid-conversation-system-messages
branch
from
79f3ae9 to
4ee477d
Compare
| /// </summary> | ||
| private static bool ModelSupportsMidConversationSystem(MessageCreateParams createParams) => | ||
| createParams.Model.Raw() is { } modelId | ||
| && modelId.StartsWith("claude-opus-4-8", StringComparison.Ordinal); |
There was a problem hiding this comment.
Bedrock sends anthropic.claude-opus-4-8, so the prefix check misses it. We could trim anthropic. to fix this. Foundry, though, breaks even that: per the docs, model is the deployment name, which can be anything.
I haven't seen model-driven behavior in other IChatClients e.g. Microsoft.Extensions.AI.OpenAI or Google.GenAI. I think is fine, but I suspect it would be a first.
There was a problem hiding this comment.
An alternative to the model-driven behavior would be a builder style WithInterleavedSystemMessages() on IChatClient or something passed in ChatOptions. Downside is developers having to know to use them. The ergonomics of having to check the model are a bit rough if the developer has to do it.
I can change to either of these if you prefer - or maybe add one and keep the current behavior , with the trim added. That provides a path for foundry users while keeping ergonomics convenient for those with the right model name?
There was a problem hiding this comment.
I like the idea of having an extension, I think a WithInterleavedSystemMessages(this ChatOptions, InterleavedSystemMessageMode mode = Auto) that adds an additional property, similar to WithCacheControl, should work.
Auto or unspecified can keep parsing the model id, while Enabled/Disabled provide an avenue for providers with custom model ids.
The extension is just for discoverability that a bare additional property would otherwise lack. Models that don't support this today will eventually phase out, at which point the extension becomes a no-op that we can then obsolete -> remove.
There was a problem hiding this comment.
Scratch Auto mode, it would be better to make this explicit only and avoid setting the model-driven precedent.
4 participants
Summary
The MEAI IChatClient mapping previously hoisted every ChatRole.System message into the top-level
systemproperty regardless of position, so the Opus 4.8 mid-conversation system message feature was unreachable.A ChatRole.System message is now emitted as a
{"role":"system"}message at its position when the placement is valid (immediately follows a user turn and either ends the array or precedes an assistant turn) and the resolved model supports it (claude-opus-4-8). Leading system messages, invalid positions, and unsupported models fall back to the top-levelsystemproperty, preserving prior behavior. Consecutive system messages are merged into one, per the API constraint.Applied to both the stable and beta IChatClient mappings, with shared tests covering valid placement, mid-list placement, hoist on unsupported model, hoist after an assistant turn, consecutive-merge, leading-only, and cache control.
Note for reviewers
This adds a couple of longer-than-usual explanatory comments in the extensions classes — most notably the block above the placement/hoist logic in
GetMessageCreateParams. I know long comments are sometimes asked to be trimmed, but the placement rules here are non-obvious (the "immediately follows a user turn AND ends the array or precedes an assistant turn"constraint, plus why the validity check can be judged per-message after merging) and there's existing precedent for keeping comments of this kind in these files. Happy to trim if you'd prefer — flagging it proactively.
Two intentional simplifications, also worth a look:
claude-opus-4-8model-id prefix. The feature is unavailable on Bedrock/Vertex/Foundry regardless of model; if those deployments ever route through this mapping, the gate would need a platform check (noted in the helper's doc comment).Test plan
dotnet test— 410 MEAI extension tests pass across the stable and beta clients (net8.0).dotnet csharpier check,dotnet format style,dotnet format analyzers— all clean.🤖 Generated with Claude Code
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com