Skip to content

[Bug]: flaky: cli-webmcp iframe tests intermittently miss the iframe's tools on Linux/macOS Chrome #42869

Description

@Abnoz01

Version

main (observed on 1.64.0-next between 2026-09-21 and 2026-09-22)

Steps to reproduce

The three iframe-based tests in tests/mcp/cli-webmcp.spec.ts register a WebMCP tool in the main frame and another in an <iframe>, run cli('open', url), then immediately webmcp-list / webmcp-call:

  • webmcp-call disambiguates same-name tools in identical same-origin frames
  • webmcp-call disambiguates duplicate tool names by frame
  • webmcp-list stitches tools across frames

Expected behavior

The listing includes the iframe's tools every time.

Actual behavior

Intermittently only the main frame's tools are collected:

Expected substring: "is registered in multiple frames, retry with the frame parameter"
Received string:    "### Result
Called WebMCP tool "echo" in http://localhost:8907/. ... "text": "main"
Expected substring: "- frame: http://localhost:8907/widget.html (frame 1)"
Received string:    "### Result
Found 1 WebMCP tool(s). ...

From the CI results database (last ~470 runs per test, expected_status = passed):

Test runs failed runs
disambiguates same-name tools in identical same-origin frames 469 11
disambiguates duplicate tool names by frame 469 6
webmcp-list stitches tools across frames 469 3

Failures occurred on main (2026-09-22 00:12, 00:14, 16:32) and on unrelated PRs (#42787, #42789, #42818, #42821, #42832, #42835, #42845), only on mcp-ubuntu-latest-chrome/chromium and mcp-macos-latest-chrome/chromium, never on Windows.

Additional context

listWebMCPTools evaluates collectToolsInPage in every frame and treats a rejected or timed-out evaluation as "no tools here". open waits for load before the snapshot collects tools, so the iframe has loaded and its script has run by then. The remaining suspect is that Chromium's getTools() reflects a freshly registered tool from a child frame asynchronously, so a listing taken right after navigation can still miss it.

I can send a PR that polls the snapshot header for the expected tool count before asserting in these three tests, unless a product-side wait is preferred.

Environment

- CI: mcp-ubuntu-latest-chrome, mcp-ubuntu-latest-chromium, mcp-macos-latest-chrome, mcp-macos-latest-chromium
- Playwright: main

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions