Skip to content

fix(ios): use the resolved display for screenshots and visual verification #2728

Description

@thymikee

Part of #2725. Investigation evidence and reproducible probe: #2727 (comment).

Resolved design / dependencies

Use the resolved app window's screen, not XCUIScreen.main, for iOS app-screen capture and visual verification. Resolve app.windows.firstMatch.frame before obtaining that window's private screen; screenshot the resulting XCUIScreen. Open, half-open, closed, a single-panel control, and a native fixture alert were measured successfully. Do not choose by a hardcoded display number or the first screen in XCUIScreen.screens.

Prerequisite: PR #2724; implement on its branch with an explicit PR base, or after it merges. No unresolved design decision remains. Consumer-level checks below remain required implementation validation; the spike proves the capture primitive, not the complete migration.

Small owning interface

Introduce one iOS capture helper, conceptually captureResolvedAppScreen(app:) -> CapturedAppScreen, returning an upright image and actual display ID, encoded pixel width/height, and pixels-per-point. Keep the private window/screen lookup exception-safe in the native XCTest bridge. Reuse the window-resolution order established by #2724; do not maintain independently drifting display resolvers or introduce a session-wide display cache.

Normalize the image by drawing it in its oriented logical size with its actual scale before PNG encoding. Measured open Duo: window 951x669, UIImage logical size about 950.667x668.667, scale 3, raw CGImage/PNG 2006x2852, UIImage orientation right; upright drawing yields 2852x2006. Do not infer orientation from raw CGImage dimensions, rely on PNG metadata being respected by every consumer, or stretch the one-pixel XCTest discrepancy to CoreDevice's nominal 2853x2007 size.

Wire and failure contracts

Carry screenshot-only metadata with the runner screenshot result (e.g. screenshotMetadata: {displayID, pixelWidth, pixelHeight, pixelsPerPoint}), alongside the existing file/base64 payload. Declare it at the owning cross-language contract, not by per-consumer casts. Host normalization uses those actual source facts, never SIMULATOR_MAINSCREEN_SCALE for an image from another panel.

An unresolved/invalid app window or screen fails required screenshot capture with a typed reason, without selecting main. Optional visual checks return their existing unavailable/unknown result; no sample is not evidence of stability or navigation change. Do not make an observation activate an app merely to obtain a screen; use the caller's intended app context.

Bounded migration

  • Route the iOS screenshot case, captureRunnerFrame, and navigation fallback visual capture through this helper. Keyboard settling already consumes frame capture and keeps its separate keyboard visibility check.
  • Update packages/platform-apple/src/core/screenshot.ts runner fallback normalization to consume source metadata.
  • Preserve macOS/tvOS behavior. Do not redesign gestures, text entry, host display inventory, or recording touch overlays (Recording touch overlay exports at ~220x480 or all-black whenever touches are drawn #2707).
  • Follow the repository split-before-behavior rule for oversized sources, with mechanical structural changes separated from behavior.

Acceptance and validation

  • Force simctl screenshot failure: runner fallback captures the visible inner content upright at correct density.
  • Navigation change/no-change controls and keyboard settling observe the lit display. A failed sample cannot satisfy a positive visual assertion.
  • Test open, half-open, closed, and a single-panel control; preserve a visible native overlay in the full-screen capture. Record pose, display, image orientation, actual pixel dimensions, scale, and app state.
  • Repeat the pose matrix on the implementation PR head. During the spike, moving Device Hub from the second host display to the main display cleared device-hub-window-missing; document this troubleshooting detail rather than treating it as a missing simulator.
  • Add focused contract/encoding tests, native builds and XCTest selection checks, affected gates, and local semantic evidence. Update ADR 0025 only to claims proved by those checks.

Ready for implementation with the explicit #2724 branch/merge prerequisite. Follow AGENTS.md; keep one task per PR and clean up manual sessions.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions