test(vertexai): stabilize search & anthropic beta integration tests#1842
Open
Mason Daugherty (mdrxy) wants to merge 1 commit into
Open
test(vertexai): stabilize search & anthropic beta integration tests#1842Mason Daugherty (mdrxy) wants to merge 1 commit into
Mason Daugherty (mdrxy) wants to merge 1 commit into
Conversation
…n tests The Google Cloud Build llm-integration-tests job was failing on tests unrelated to the package-version metadata feature (#1841): - test_search_builtin used a context-dependent follow-up ("that last story") that the model could answer with a clarification, returning no grounding metadata. Use a self-contained current-events query so the invoke path reliably grounds while still asserting grounding metadata and annotations for both streaming and invoke. - The anthropic async beta tests hard-coded an expired/unaccepted beta header value, causing anthropic.BadRequestError. Skip when the live endpoint rejects the beta header so beta rotation no longer fails CI, and add deterministic unit tests that verify betas route through the Anthropic beta client for async invoke and async streaming. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The Google Cloud Build
langchain-google-vertexai-us(llm-integration-tests) job was failing on tests unrelated to the package-version metadata feature (#1841), which only adds_add_version(...)validators and does not touch Anthropic beta handling or Vertex search. This makes those live tests deterministic while preserving their intent.test_search_builtin: replaced the context-dependent follow-up ("that last story"), which the model could answer with a clarification (no grounding), with a self-contained current-events query. Still asserts grounding metadata/annotations for both streaming and invoke.context-1m-2025-08-07was rejected (BadRequestError). Now skips when the live endpoint rejects the beta header, and adds deterministic unit tests verifyingbetasroute through the Anthropic beta client for async invoke and stream.Release Note
none
Test Plan
uv run --group test pytest tests/unit_tests/test_model_garden_betas.py(passes locally)Made by Open SWE