feat: support Vertex AI for Google CUA agents#2365
Open
viviluccioli wants to merge 2 commits into
Open
Conversation
Routes provider "vertex" (explicit or via a vertex/ model prefix) to GoogleCUAClient and initializes @google/genai in Vertex mode, honoring service-account auth, express-mode API keys, project/location provider options, and ambient ADC. Relaxes VertexModelConfigObject so auth and providerOptions are individually optional, since express keys and ADC need neither. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The vertex CUA change made VertexModelConfigObject.auth and .providerOptions optional but left the generated OpenAPI spec stale, which would have carried the old required-fields contract into the Stainless-generated SDKs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 436d0e8 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
This PR is from an external contributor and must be approved by a stagehand team member with write access before CI can run. |
github-actions
Bot
added
external-contributor
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
5 issues found across 10 files
Confidence score: 3/5
- In
packages/core/lib/v3/agent/AgentProvider.tsandpackages/core/lib/v3/agent/GoogleCUAClient.ts, explicitprovider: "vertex"routing/acceptance is not tightly gated to valid Google Vertex CUA models, so unsupported model IDs can be sent down the Vertex path and fail at runtime instead of being rejected early — add provider+model allowlist validation before request dispatch. - In
packages/server-v3/openapi.v3.yaml, the relaxedVertexModelConfigObjectshape appears to be a Stagehand API contract change without the required server integration coverage, which raises regression risk between client/server behavior — add an integration test underpackages/server/testthat exercises the new schema behavior before merging. - In
packages/core/lib/v3/agent/GoogleCUAClient.ts, top-levelClientOptions.headersare dropped for Vertex CUA unless duplicated in provider-specific options, which can silently strip auth/tenant headers in production calls — merge generic headers intohttpOptions.headersfor the Vertex request path. - In
packages/core/lib/v3/v3.ts, current tests miss the V3 prefix-to-provider branch formode: "cua"withmodel: "vertex/gemini-...", so this routing regression could ship undetected — add a V3-level test that asserts correct provider propagation for that model prefix.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/core/lib/v3/agent/GoogleCUAClient.ts">
<violation number="1" location="packages/core/lib/v3/agent/GoogleCUAClient.ts:73">
P2: The Vertex client accepts an explicit `provider: "vertex"` without validating that the model is one of the Vertex CUA models, so invalid model ids can slip through to the API call.</violation>
<violation number="2" location="packages/core/lib/v3/agent/GoogleCUAClient.ts:135">
P2: Vertex CUA requests drop top-level `ClientOptions.headers` unless callers duplicate them under `providerOptions.vertex.headers`. Merge the generic headers into `httpOptions.headers` so the public client-option contract holds for the new provider.</violation>
</file>
<file name="packages/core/lib/v3/v3.ts">
<violation number="1" location="packages/core/lib/v3/v3.ts:2070">
P3: A regression in V3’s prefix-to-provider propagation would not be caught: current Vertex tests bypass this new branch entirely. Add a V3-level test for `mode: "cua"` with `model: "vertex/gemini-..."` that asserts the handler receives the stripped model name and `clientOptions.provider === "vertex"`.
(Based on your team's feedback about adding unit tests for new behavior.) [FEEDBACK_USED]</violation>
</file>
<file name="packages/core/lib/v3/agent/AgentProvider.ts">
<violation number="1" location="packages/core/lib/v3/agent/AgentProvider.ts:94">
P2: Vertex routing is accepted for any explicitly provided provider, so non-Google CUA models can still reach the new Vertex client instead of being rejected early.</violation>
</file>
<file name="packages/server-v3/openapi.v3.yaml">
<violation number="1" location="packages/server-v3/openapi.v3.yaml:390">
P2: Custom agent: **Any breaking changes to Stagehand REST API client / server implementation must be covered by an integration test under packages/server/test**
The VertexModelConfigObject schema was relaxed by making `auth` and `providerOptions` optional, which introduces new server-side code paths for express-mode API key and ADC authentication flows. These paths should be covered by integration tests in `packages/server-v3/tests/integration/` (e.g., `start.test.ts` or a dedicated vertex test file) to verify the server correctly accepts and processes requests without these previously-required fields. Currently no vertex-related integration tests exist in the server-v3 integration suite.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Client as Client (agent.execute)
participant V3 as V3 engine
participant Resolve as resolveModel()
participant AgentProv as AgentProvider
participant GCAClient as GoogleCUAClient
participant GenAI as GoogleGenAI SDK
participant VertexAPI as Vertex AI API
participant Env as Env Vars / ADC
Note over Client,VertexAPI: New Vertex AI CUA flow (vertex/ prefix)
Client->>V3: execute(model="vertex/gemini-3.5-flash", ...)
V3->>Resolve: resolveModel("vertex/gemini-3.5-flash")
Resolve-->>V3: modelName="gemini-3.5-flash", provider="vertex"
alt provider is "vertex" and clientOptions.provider not set
V3->>V3: set clientOptions.provider = "vertex"
end
V3->>AgentProv: getAgentProvider("vertex/gemini-3.5-flash")
AgentProv->>AgentProv: strip "vertex/" prefix, map to "google"
alt stripped model is a Google CU model
AgentProv->>AgentProv: return "vertex" (not "google")
else not a Google CU model
AgentProv-->>V3: throw UnsupportedModelError
end
AgentProv-->>V3: provider = "vertex"
V3->>AgentProv: getClient("gemini-3.5-flash", {provider:"vertex", ...})
AgentProv->>GCAClient: new GoogleCUAClient("vertex", "gemini-3.5-flash", ...)
rect rgb(240,240,240)
Note over GCAClient: Vertex auth branch
GCAClient->>GCAClient: isVertex=true, strip "vertex/" from modelName
alt auth.type === "googleServiceAccount"
GCAClient->>GCAClient: apiKey="", googleAuthOptions from auth.credentials
else clientOptions.apiKey or GOOGLE_VERTEX_AI_API_KEY is set (Express mode)
GCAClient->>GCAClient: apiKey=key, no project/location
else ambient ADC (no service account, no express key)
GCAClient->>Env: read GOOGLE_APPLICATION_CREDENTIALS
GCAClient->>GCAClient: project from options or GOOGLE_CLOUD_PROJECT, location from options or GOOGLE_CLOUD_LOCATION
end
Note over GCAClient: No fallback to GEMINI_API_KEY in Vertex mode
end
GCAClient->>GCAClient: build genAIOptions {vertexai:true, ...}
GCAClient->>GenAI: new GoogleGenAI(genAIOptions)
V3->>GCAClient: executeCUA(input, ...)
loop CUA inference rounds
GCAClient->>GenAI: send request with model="gemini-3.5-flash"
GenAI->>VertexAPI: call Vertex AI endpoint (aiplatform.googleapis.com)
VertexAPI-->>GenAI: response (actions, screenshots)
GenAI-->>GCAClient: parsed result
Note over GCAClient,V3: Multiple rounds for multi-step tasks
end
GCAClient-->>V3: final result (actions, usage)
V3-->>Client: result
Note over Client,VertexAPI: Vertex model usage recorded as provider=vertex, model=gemini-3.5-flash
Tip: cubic used a learning from your PR history. Let your coding agent read cubic learnings directly with the cubic MCP.
Re-trigger cubic
Comment on lines
+135
to
+139
| if (vertexOptions?.headers) { | ||
| genAIOptions.httpOptions = { | ||
| ...(genAIOptions.httpOptions ?? {}), | ||
| headers: vertexOptions.headers, | ||
| }; |
Contributor
There was a problem hiding this comment.
P2: Vertex CUA requests drop top-level ClientOptions.headers unless callers duplicate them under providerOptions.vertex.headers. Merge the generic headers into httpOptions.headers so the public client-option contract holds for the new provider.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/agent/GoogleCUAClient.ts, line 135:
<comment>Vertex CUA requests drop top-level `ClientOptions.headers` unless callers duplicate them under `providerOptions.vertex.headers`. Merge the generic headers into `httpOptions.headers` so the public client-option contract holds for the new provider.</comment>
<file context>
@@ -70,19 +70,90 @@ export class GoogleCUAClient extends AgentClient {
+ if (this.baseURL) {
+ genAIOptions.httpOptions = { baseUrl: this.baseURL };
+ }
+ if (vertexOptions?.headers) {
+ genAIOptions.httpOptions = {
+ ...(genAIOptions.httpOptions ?? {}),
</file context>
Suggested change
| if (vertexOptions?.headers) { | |
| genAIOptions.httpOptions = { | |
| ...(genAIOptions.httpOptions ?? {}), | |
| headers: vertexOptions.headers, | |
| }; | |
| if (clientOptions?.headers || vertexOptions?.headers) { | |
| genAIOptions.httpOptions = { | |
| ...(genAIOptions.httpOptions ?? {}), | |
| headers: { | |
| ...(clientOptions?.headers ?? {}), | |
| ...(vertexOptions?.headers ?? {}), | |
| }, | |
| }; | |
| } |
| tools, | ||
| ); | ||
| case "google": | ||
| case "vertex": |
Contributor
There was a problem hiding this comment.
P2: Vertex routing is accepted for any explicitly provided provider, so non-Google CUA models can still reach the new Vertex client instead of being rejected early.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/agent/AgentProvider.ts, line 94:
<comment>Vertex routing is accepted for any explicitly provided provider, so non-Google CUA models can still reach the new Vertex client instead of being rejected early.</comment>
<file context>
@@ -91,6 +91,7 @@ export class AgentProvider {
tools,
);
case "google":
+ case "vertex":
return new GoogleCUAClient(
type,
</file context>
| process.env.GOOGLE_API_KEY || | ||
| ""; | ||
| this.baseURL = clientOptions?.baseURL as string | undefined; | ||
| const isVertex = type === "vertex" || clientOptions?.provider === "vertex"; |
Contributor
There was a problem hiding this comment.
P2: The Vertex client accepts an explicit provider: "vertex" without validating that the model is one of the Vertex CUA models, so invalid model ids can slip through to the API call.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/agent/GoogleCUAClient.ts, line 73:
<comment>The Vertex client accepts an explicit `provider: "vertex"` without validating that the model is one of the Vertex CUA models, so invalid model ids can slip through to the API call.</comment>
<file context>
@@ -70,19 +70,90 @@ export class GoogleCUAClient extends AgentClient {
- process.env.GOOGLE_API_KEY ||
- "";
- this.baseURL = clientOptions?.baseURL as string | undefined;
+ const isVertex = type === "vertex" || clientOptions?.provider === "vertex";
+ if (isVertex && this.modelName.startsWith("vertex/")) {
+ this.modelName = this.modelName.slice("vertex/".length);
</file context>
| required: | ||
| - modelName | ||
| - provider | ||
| - auth |
Contributor
There was a problem hiding this comment.
P2: Custom agent: Any breaking changes to Stagehand REST API client / server implementation must be covered by an integration test under packages/server/test
The VertexModelConfigObject schema was relaxed by making auth and providerOptions optional, which introduces new server-side code paths for express-mode API key and ADC authentication flows. These paths should be covered by integration tests in packages/server-v3/tests/integration/ (e.g., start.test.ts or a dedicated vertex test file) to verify the server correctly accepts and processes requests without these previously-required fields. Currently no vertex-related integration tests exist in the server-v3 integration suite.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/server-v3/openapi.v3.yaml, line 390:
<comment>The VertexModelConfigObject schema was relaxed by making `auth` and `providerOptions` optional, which introduces new server-side code paths for express-mode API key and ADC authentication flows. These paths should be covered by integration tests in `packages/server-v3/tests/integration/` (e.g., `start.test.ts` or a dedicated vertex test file) to verify the server correctly accepts and processes requests without these previously-required fields. Currently no vertex-related integration tests exist in the server-v3 integration suite.</comment>
<file context>
@@ -379,16 +379,15 @@ components:
$ref: "#/components/schemas/VertexModelProviderOptions"
required:
- modelName
- provider
- - auth
- - providerOptions
</file context>
| // resolveModel drops the parsed provider prefix; "vertex/" must reach | ||
| // the agent client since it changes the endpoint/auth, not the model. | ||
| if (provider === "vertex" && !clientOptions.provider) { | ||
| clientOptions.provider = "vertex"; |
Contributor
There was a problem hiding this comment.
P3: A regression in V3’s prefix-to-provider propagation would not be caught: current Vertex tests bypass this new branch entirely. Add a V3-level test for mode: "cua" with model: "vertex/gemini-..." that asserts the handler receives the stripped model name and clientOptions.provider === "vertex".
(Based on your team's feedback about adding unit tests for new behavior.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/v3.ts, line 2070:
<comment>A regression in V3’s prefix-to-provider propagation would not be caught: current Vertex tests bypass this new branch entirely. Add a V3-level test for `mode: "cua"` with `model: "vertex/gemini-..."` that asserts the handler receives the stripped model name and `clientOptions.provider === "vertex"`.
(Based on your team's feedback about adding unit tests for new behavior.) </comment>
<file context>
@@ -2057,12 +2057,19 @@ export class V3 {
+ // resolveModel drops the parsed provider prefix; "vertex/" must reach
+ // the agent client since it changes the endpoint/auth, not the model.
+ if (provider === "vertex" && !clientOptions.provider) {
+ clientOptions.provider = "vertex";
+ }
+
</file context>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Google's Computer Use models were only reachable through the Gemini API (
GEMINI_API_KEY). Teams running on Google Cloud need them through Vertex AI instead — for service-account and ADC auth, GCP project billing, and region pinning for data residency.Vertex was already a supported provider for regular LLM calls (
LLMProvider.ts), but the CUA path had no Vertex branch, somode: "cua"was unusable on Vertex.What changed
AgentProvider: routes thevertexprovider toGoogleCUAClient. Avertex/model prefix selects the provider — the prefix decides the endpoint and auth, not the model — and is rejected for non-Google-CU models.GoogleCUAClient: initializes@google/genaiwithvertexai: true, resolving auth in order: explicit service account (googleAuthOptions), express-mode API key, then ambient ADC. Honorsproject/locationfrom provider options orGOOGLE_CLOUD_PROJECT/GOOGLE_CLOUD_LOCATION. Deliberately does not fall back to Gemini API keys in Vertex mode. Express mode omits project/location, which the SDK rejects alongsideapiKey.VertexModelConfigObject:authandproviderOptionsare now individually optional — express keys and ADC need neither.packages/server-v3/openapi.v3.yaml: regenerated so the relaxed contract reaches the Stainless-generated SDKs. Without this the generated clients would keepauth/providerOptionsrequired, making express-mode and ADC unusable from them.AVAILABLE_CUA_MODELS: adds the fourvertex/-prefixed Google CU models.The service-account →
googleAuthOptionsmapping intentionally mirrors the existing non-CUA path inLLMProvider.ts, andGOOGLE_VERTEX_AI_API_KEYmatchesproviderEnvVarMapinlib/utils.ts. That does duplicate the mapping in two places — happy to factor it into a shared helper if you'd prefer.E2E Test Matrix
Live runs used
vertex/gemini-3.5-flashwith a service account via ADC (GOOGLE_APPLICATION_CREDENTIALS, no express-mode API key), against the globalaiplatform.googleapis.comendpoint withlocation: "us"— not a regional one.agent.execute(): navigate google.com, click the search bar, type a query, click search, click the Images tab, select from dropdown menusagentExecuterounds, all returning HTTP 200 from the stagehand serveragent.execute(): search and select from a dropdown (Color → "Black and white" on Google Images)agentExecuteround, HTTP 200provider=vertex,model=gemini-3.5-flashvertex/prefix is stripped on the live path — the model id sent to the API is clean, notvertex/gemini-3.5-flash— and that usage attributes to the vertex provider rather than google.vitest run packages/core/tests/unit/google-cua-vertex.test.ts(new)vertex/→vertex,google/staysgoogle, non-CU models rejected), prefix stripping, and client construction for service-account / express-mode / ADCnew GoogleGenAI({vertexai: true})doesn't validate credentials at construction. Supporting evidence; the live runs above are what prove Vertex works. Asserts the Stagehand-visible surface rather than@google/genaiinternals, which differ across SDK builds.vitest run packages/core/tests/unit/api-variables-schema.test.ts(updated)auth/providerOptionspnpm lint(eslint + typecheck, all packages)Tasks: 7 successful, 7 totalpnpm --filter @browserbasehq/stagehand-server-v3 run gen:openapire-run against the committed specopenapi.v3.yamlis exactly what the generator emits, so the Stainless upload won't drift.Express-mode API key auth is covered by construction tests only — I don't have an express-mode key to exercise it live, so that path is unverified end-to-end. Service account via ADC is the path with live coverage.
Summary by cubic
Adds Vertex AI support for Google Computer Use Agents. You can now call Google CUA models via the
vertexprovider or avertex/model prefix with service-account, express API key, or ADC auth.vertexprovider andvertex/-prefixed Google CUA models toGoogleCUAClient, strip the prefix before API calls, and propagate the provider through the CUA path.@google/genaiwithvertexai: true; auth order: service account (googleAuthOptions), express-modeapiKey/GOOGLE_VERTEX_AI_API_KEY, then ambient ADC. Supportproject/locationvia options orGOOGLE_CLOUD_PROJECT/GOOGLE_CLOUD_LOCATION. No fallback to Gemini API keys in Vertex mode.vertex/gemini-2.5-computer-use-preview-10-2025,vertex/gemini-3-flash-preview,vertex/gemini-3.5-flash, andvertex/gemini-3-pro-previewto available CUA models.VertexModelConfigObject.authand.providerOptionsoptional and regeneratepackages/server-v3/openapi.v3.yamlso SDKs reflect the relaxed schema.Written for commit 436d0e8. Summary will update on new commits.