Handle [DONE] terminal event in SSE streams from Azure APIM hosted Anthropic models#202
Handle [DONE] terminal event in SSE streams from Azure APIM hosted Anthropic models#202brused27 wants to merge 2 commits into
Conversation
|
Hey @brused27 thanks for the proposal! I tried to reproduce this and so far haven't been able to trigger the What I tested:
The Could you share:
|
|
Hey @dtmeadows-ant - thank you for the review and you are spot on here that it does seem to be due to us passing through an Azure APIM now that I troubleshot this further. We currently use this enterprise wide to manage our endpoints with subscription keys to make sure our LLM traffic is attributed properly against business units and ensure high availability cross-region for some scenarios we need if we exceed TPM limits. However, we have configured this to be a passthrough policy ultimately so that existing Claude based apps will work properly (Claude Code, Anthropic SDK, etc).
We are using new AnthropicClient(new() { BaseUrl = endpoint, ApiKey = apiKey });
Global Standard,
Yes, Azure APIM policy.
Included below - Pure Microsoft Foundry vs Azure APIM in my environment curl request/responses. I also included our Azure APIM policy below, which is purely a passthrough policy as seen below. I did find that the way the SSE processing code works in this SDK currently, it drops back to take a default SSE structured event as assuming My hope is that we can bring it back in because this SDK works properly on v12.11.0 but the SSE parsing code seems to have been rewritten more recently, and it has broken our integration when using the pure SDK now. Azure APIM (stream DOES contain
|
|
As a follow up comparison, I am finding the current Anthropic Python SDK is gracefully handling this scenario against the APIM - I am hoping the .NET SDK can be re-aligned to that behavior the same way. Here is a small sample script I am finding is succeeding against Azure APIM without error: import os
from anthropic import Anthropic
client = Anthropic(api_key=os.getenv("APIM_API_KEY"), base_url="https://{REMOVED}.azure-api.net/anthropic")
print("Response from Claude:")
with client.messages.stream(
model="claude-haiku-4-5",
max_tokens=1024,
messages=[
{"role": "user", "content": "hello"}
]
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
print() |
brused27
changed the title
brused27
force-pushed
the
fixes/foundryDONEResponse
branch
from
9fe6cf4 to
9a00df9
Compare
2 participants
Added a guard for
data: [DONE]SSE terminal event emitted by Microsoft Foundry during SSE streams. Without this guard, the SSE parser assigns a defaultEventTypeof"message"to such events, causing the SDK to attempt JSON deserialization on the literal string"[DONE]". This results in aJsonException('D' is an invalid start of a value).Test case:
dotnet test src/Anthropic.Tests/Anthropic.Tests.csproj --filter "SseTest.Sse_SkipsDoneTerminalEvent"Previous exception seen with Anthropic model usage on Microsoft Foundry using streaming chat completion through Microsoft.Extensions.AI: