fix(ios): report capture refusals as typed reasons in back-fallback and recording

[thymikee]

Summary

iOS back fallback and record start now report a capture that did not happen as a typed reason instead of a false verdict, finishing the failure contracts #2741 left papered over (#2728).

• In-app back coordinate fallback: a nil screenshot sample was read as "no visual change", and when the app resolved no window it sampled SpringBoard, so two identical home-screen captures reported in-app back control is not available after a real leading tap had already run. It now observes only the app's own resolved window and returns the display refusal as an unverified outcome carrying APP_SCREEN_*. An observable no-change is unchanged.
• record start: every no-frame bootstrap collapsed to an untyped NSError; it now throws the shared APP_SCREEN_* reason, so a runtime with no resolvable window fails closed with a typed code.
• A zero-pixel capture is refused at the capture type.

macOS/tvOS keep their host-display capture and generic bootstrap error. 6 Swift files, one host doc comment, one ADR paragraph. Builds on #2741; addresses #2728 (parent #2725).

Validation

Commit 7b5b5043, rebased on main (0.21.12).

• pnpm check:affected --run: all runnable checks passed (Swift runner builds and replay lanes are GitHub/device-lane only).
• build:xcuitest:ios → ** TEST BUILD SUCCEEDED **; check:packaged-runner-swift parses + line parity; xctest-selection 276 declared, 0 unreachable; typecheck, lint, format, check:layering, check:fallow clean.
• New runner unit tests pin the observation split, the refusal-selection, and the typed-vs-generic error mapping. Mutation: revert the iOS typed throw to the old NSError and the mapping test fails.

Outstanding: live iPhone Duo re-verification of these paths (this host has no iPhone Duo / Xcode 27.1). A device handoff runbook is provided for the Duo lane before merge-ready.

[thymikee]

Live iPhone Duo verification handoff — #2728 capture failure contracts

Update after the Duo run + reviewer follow-up. PR head is now e5ad8662c (rebased onto main; was 6 behind). The two changed routes — the back fallback's app-only observation and the record start bootstrap refusal — are not reachable through the simctl terminate → back / record start recipe below on a simulator: the runner's activation guard relaunches a terminated-but-installed app before the capture resolves (ACTIVATE … reason=stale_target → Launch …), and record start on a simulator uses the simctl recordVideo backend, so it never asks the runner for a bootstrap frame. Those two contracts are pinned by automatic runner unit tests (§0) that now drive the production samplers, not re-created inline copies; a device pass should cover the no-change regression and the pose matrix (§1a, §3–§5), and not chase APP_SCREEN_WINDOW_UNRESOLVED on a simulator.

Toolchain: Xcode 27.1 beta + booted iPhone Duo (iOS 27.1). Build with unit tests: pnpm install --frozen-lockfile && pnpm build && AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS=1 AGENT_DEVICE_XCUITEST_PLATFORM=ios pnpm build:xcuitest:ios.

0. Runner unit tests — these own the two failure contracts (automatic)

The app-only decision for the back fallback now lives in one static the production code calls: navigationFallbackSample(resolvingApp:systemSurface:encoding:). It samples only the app's own resolved screen and never consults systemSurface; a screenshot still uses selectObservedScreenCapture (unchanged from 7b5b504), which does fall back to the system surface on an unresolved window. The distinction — back must never read SpringBoard's home screen as "unchanged" — is now the tested contract, not a re-created inline literal.

• Host lane (macOS, every PR): testVerifyNavigationFallbackOutcomeReportsUnresolvedWindowWithoutSystemSurface calls the production navigationFallbackSample, handing it a systemSurface closure that XCTFails the test if consulted, and asserts .unverified(APP_SCREEN_WINDOW_UNRESOLVED) through inAppBackOutcome; testNavigationVisualSampleDistinguishesEncodedFrameFromRefusal; plus testNavigationVisualVerificationSeparatesNoChangeFromNoSample, testNavigationFallbackReportsTheRefusalItHitNotADefaultCode, testRecordingBootstrapErrorKeepsTheTypedRefusalAndFallsBackGenerically, testRecordingStartSurfacesACaptureRefusalAsATypedCode.
• iOS PR lane (ios.yml): testRecordStartThrowsTheCaptureRefusalItReceived drives ScreenRecorder.start with an always-refusing capture and maps the actual thrown error through recordingStartErrorPayload(for:), asserting the host code is APP_SCREEN_WINDOW_UNRESOLVED.

1. back coordinate fallback — observable outcomes (Duo)

Bound to com.callstack.agentdevicelab on a screen with no real back control so the fallback runs.

• (a) Observable no-change — REGRESSION (verified): leading tap navigates nowhere → back = in-app back control is not available; runner.log: AGENT_DEVICE_RUNNER_IN_APP_BACK_VISUAL_VERIFICATION beforeBytes=A afterBytes=A changed=no, with a real leading tap synthesized. Byte-stable vs pre-change.
• (b) Observable change — leading tap navigates → back ok, log ... changed=yes (A != B).
• (c) Unobserved — the app-only refusal that reports APP_SCREEN_WINDOW_UNRESOLVED (via navigationFallbackSample, which never samples the system surface) is a unit-test contract (§0). Not simulator-reachable via simctl terminate (the guard relaunches the app). Do not spend the device run here.

2. record start typed refusal

The bootstrap refusal contract is owned by testRecordStartThrowsTheCaptureRefusalItReceived (§0). On a simulator record start uses simctl recordVideo and never requests a runner bootstrap frame, so verify only the success path on-device (records normally, exports the lit panel).

3. Pose matrix (open / half-open / closed) — verified

agent-device fold open|half-open|closed, re-snapshot after each. The capture resolves the app onto the currently-lit panel: closed → LCD 466×678; half-open / open → LCD-1 669×951. Both panels report pointScale: 3; the unlit panel reads backlight is off. Hinge read-back: open=180°, half-open=130°, closed=0°. The CoreDevice Monitor Motion / device info displays stream on the 27.1 beta is intermittently flaky — retry a failed fold after a short cooldown (needs the 27.1 toolchain; the selected Xcode returns an empty display table).

4. Single-panel control — verified

Closed pose is the single-panel control: outer LCD 466×678 at pixelsPerPoint=3.

5. macOS/tvOS regression

AGENT_DEVICE_XCUITEST_PLATFORM=macos pnpm build:xcuitest:macos → SUCCEEDED. A macOS record start with no host frame still fails with the generic untyped record error, never an APP_SCREEN_* code (pinned on the host lane).

Report back

The changed=no no-change line, per-pose hinge angle + lit-panel name, and pixelsPerPoint. The two failure codes are reported by §0's automated tests, not the device run.

Mutation sanity (observed failing on a real build at e5ad8662c):

• Making navigationFallbackSample consult the system surface on .unresolvedWindow (i.e. reverting the fallback to sample SpringBoard) fails testVerifyNavigationFallbackOutcomeReportsUnresolvedWindowWithoutSystemSurface with XCTAssertFalse failed — the in-app fallback samples the app only, never SpringBoard. This is the reviewer's key finding: the test now drives the production sampler, so a revert of the fix turns it red.
• Reverting the refusal->code mapping (navigationVisualSample's .failure branch dropping the code) fails both testVerifyNavigationFallbackOutcome… and testNavigationVisualSample….
• Reverting the #if os(iOS) typed throw in RunnerTests+ScreenRecorder.swift to recordingBootstrapError(from: nil) fails testRecordStartThrowsTheCaptureRefusalItReceived.

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	4.78 MB	4.79 MB	+5.1 kB
Package (unpacked)	4.78 MB	4.79 MB	+5.1 kB
Package (download)	1.43 MB	1.43 MB	+1.3 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	29.1 ms	28.7 ms	-0.4 ms
CLI --help	82.4 ms	87.7 ms	+5.3 ms

[thymikee]

At 7b5b504, the typed refusals aren't pinned to the code paths that produce them. RunnerTests+RecordingTests.swift:527 calls the two static helpers directly, while ScreenRecorder.start's bootstrap loop (ScreenRecorder.swift:49-71) runs in no test — reverting line 70 to the old NSError keeps both tests green. The back fix is the same story: Navigation.swift:184 swaps in captureResolvedAppScreen, but restoring captureObservedScreen also leaves every test green, since the suite only covers the pure classifier and navigationFallbackErrorPayload. So both shipped behaviors — record start carrying a typed refusal, and back refusing to sample a second process — can regress under a green suite. Can you inject the capture/sample source the same way selectObservedScreenCapture is injected, call ScreenRecorder.start(capture: { .failure(.unresolvedWindow) }) under AGENT_DEVICE_RUNNER_UNIT_TESTS and assert the thrown error maps through recordingStartErrorPayload, and add a verifyNavigationFallbackOutcome test where an app-side .unresolvedWindow yields .unverified(APP_SCREEN_WINDOW_UNRESOLVED) without ever calling the system-surface closure?

There's no live run at this head. The handoff comment in RunnerTests+Navigation.swift:225 names a stale SHA (cd758bd), and its recipe — terminate the bound app, then run back or record start — can't reach the new paths: back, backInApp and recordStart are non-lifecycle commands, so prepareActiveCommandContext relaunches the bound app first, and even without the relaunch, resolveRunnerWindow falls back to app.frame (0x0), so the tap returns .unavailable before any capture happens; record start still uses captureObservedScreen, which just captures SpringBoard instead of throwing APP_SCREEN_WINDOW_UNRESOLVED. Following that runbook on a device either looks like the PR failed (old "not available" outcome, successful recording) or passes without touching the changed route. What route or fixture reaches the .unobserved branch through production code — for example a top-leading tap that backgrounds the app, or a Duo pose where the window names no display — and what does runner.log show, specifically AGENT_DEVICE_RUNNER_IN_APP_BACK_VISUAL_VERIFICATION ... changed=unknown with details.runnerErrorCode set to an APP_SCREEN_* code? A single-panel control run showing changed=no and changed=yes matching main would confirm nothing else moved. For record start, either show a real state that produces a typed code on-device, or say plainly that its coverage is unit-only.

Is there a smaller shape here? Net production is around 235 lines, well under the usual size line, but it spans the runner and the TS contract doc and adds three new types. NavigationVisualSample duplicates Result<Data, RunnerAppScreenCaptureFailure> — could it carry that Result directly and derive the code and hint from the failure, dropping the struct and the nil-code default branch? The recorder closure only reads image and pixel size, so could it take Result<RunnerImage, RunnerAppScreenCaptureFailure> instead, which would also remove the macOS placeholder CapturedAppScreen (displayID 0, pixelsPerPoint 1) at Lifecycle.swift:84 and the invalid state it represents? Both would be local to the runner, so nothing else needs to change first.

Not blocking: the SpringBoard fallback in the ADR (docs/adr/0025-foldable-apple-panels.md:116) captures the whole display rather than SpringBoard's own window, which was already true at base on single-panel devices and only surfaces on a Duo whose SpringBoard sits on the dark panel, so the fix could either constrain the fallback to when it provably names the app's display or just correct the ADR wording, and separately, when the before-sample resolves but the leading tap backgrounds or terminates the app, head now returns .unverified/APP_SCREEN_WINDOW_UNRESOLVED where base fell back to SpringBoard and returned ok, so it's worth deciding and documenting whether that case counts as a change; both can be taken or left.

I didn't build or run the Swift runner — the claim that the current tests can't catch these mutations comes from reading the test bodies, not from executing one. I also didn't measure what XCUIApplication.frame returns for a terminated app; the preflight relaunch makes that moot on these two routes anyway. And I couldn't measure how often, on a single-panel device, windows.firstMatch misses while resolveRunnerWindow still finds a usable frame.

CI is green across all 18 checks, but the Swift runner's device lanes don't exercise the live back or record-start refusal routes, so that green doesn't cover the behavior this PR changes. Pin both fixes with tests that go through ScreenRecorder.start and an injected back-fallback sample source, then run live on a route that reaches changed=unknown with details.runnerErrorCode set to an APP_SCREEN_* code, plus the single-panel control, before this is ready to merge.

[thymikee]

Duo-lane live verification — iPhone Duo, iOS 27.1

Ran this PR's runner (built from 7b5b50433; identity confirmed by APP_SCREEN_WINDOW_UNRESOLVED / APP_SCREEN_CAPTURE_UNRENDERABLE / the "unknown outcome" hint present in the deployed xctest binary — not an adopted build) against the booted iPhone Duo, session on an isolated state-dir.

Proven live

• No-change contract preserved. Healthy app → back --in-app: the runner synthesized a real leading tap at (37.3, 105.1) on the resolved app window (466×678, displayID 1) and logged …IN_APP_BACK_VISUAL_VERIFICATION beforeBytes=255626 afterBytes=255626 changed=no → host returns in-app back control is not available. fix(ios): capture the display that hosts the app, not the main screen #2741's resolvable-window path is intact.

• Capture names the lit panel; pixelsPerPoint = 3. Every pose resolves the app onto the currently-lit panel; both panels report pointScale: 3:

  Pose	Hinge	App panel	Native size	Lit panel
  closed	0°	LCD	466×678	LCD on / LCD-1 off
  half-open	130°	LCD-1	669×951	LCD-1 on / LCD off
  open	180°	LCD-1	669×951	LCD-1 on / LCD off

  Closed is the single-panel control (outer LCD, 466×678 @3x). Attached below: live capture of the Agent Device Tester on the outer (lit) panel through this runner.

• Normal recording works (record start/stop → valid .mov).

Two live checks from the handoff's recipe do NOT reach their expected codes on an iOS simulator

1. terminate → back never yields APP_SCREEN_WINDOW_UNRESOLVED. The activation guard self-heals a terminated-but-installed app: ACTIVATE … state=1 reason=stale_target → Launch … relaunches it and resolves its window, so back sees a live window → changed=no → "not available". The unobserved branch is unreachable while the app is still installed and relaunchable.
2. record start can't produce the runner capture refusal on a simulator. It also self-heals, and its backend is simctl recordVideo — it never requests a runner bootstrap frame, so the APP_SCREEN_* bootstrap path isn't exercised at all on iOS sim.

Net: on an iOS simulator the only failure-contract evidence available here is the unchanged no-change contract (above) plus the runner unit tests (RunnerTests+Navigation.swift, +AppScreenCaptureTests, +RecordingTests) that pin refusal-selection and the typed-vs-generic mapping. To reproduce the unobserved / record-refusal codes live, we'd need a guard-defeating state (a foreground app whose scene has no resolvable window) or the runner-native recording backend / a physical device where record requests the bootstrap frame.

Also (not this PR)

The iOS 27.1 beta's CoreDevice Monitor Motion / device info displays stream is intermittently flaky — fold half-open/closed failed once and succeeded on retry after a cooldown; the selected Xcode Beta.6 returns an empty display table, so fold requires the 27.1 toolchain. Worth a line in docs/agents/device-verification.md.

Verdict

CI is green and the branch is MERGEABLE / CLEAN. The no-change contract and Duo capture-naming / pixelsPerPoint are live-verified. Flagging the recipe correction above so the two failure-code branches are either re-triggered via a guard-defeating scenario or documented explicitly as unit-test-covered.

[thymikee]

Thanks for the Duo run at 7b5b504. It confirms the unchanged path: back --in-app still logs changed=no on a resolved window, and capture names the lit panel at 3x in every pose.

It also confirms that the two changed routes can't be reached live on a simulator: the activation guard relaunches the app before back, and record start uses simctl recordVideo, so it never asks for a bootstrap frame. That makes the unit tests the only evidence for those routes. As noted earlier, they don't reach them yet: reverting ScreenRecorder.swift:70 or Navigation.swift:184 keeps the suite green.

The next step is the one from the earlier comment. Can you drive ScreenRecorder.start with an injected failing capture, and add a verifyNavigationFallbackOutcome test where an app-side .unresolvedWindow gives APP_SCREEN_WINDOW_UNRESOLVED without calling the system-surface closure? Please also fix the stale SHA and recipe in the handoff comment at RunnerTests+Navigation.swift:225, since that recipe can't reach these paths.

[thymikee]

Thanks for the updated handoff. I checked the pushed head 7b5b504, and it does not contain testRecordStartThrowsTheCaptureRefusalItReceived or testVerifyNavigationFallbackOutcomeReportsUnresolvedWindowWithoutSystemSurface. The PR still has one commit. Could the tests be in a local commit that is not pushed yet?

The earlier finding stays open until those tests are on the branch. Once they are pushed, I will check that each one fails when its route is reverted, as the handoff describes.

[thymikee]

At 16b39d2, the fix still has a gap: the regression test does not exercise the production sampler, so it cannot catch a revert of the fix. This is a follow-up on the earlier review (7b5b504, #2766 (comment)); the typed-reason plumbing for capture refusals is in place, but the test-coverage finding from that review still applies.

RunnerTests+Navigation.swift:395 builds selectScreenCapture(resolving: { .failure(.unresolvedWindow) }, fallback: .never) inline instead of calling the production sampler at RunnerAppScreenCapture.swift:178-193. With .never, selectScreenCapture just returns resolving(), so the test only checks that its own literal is the identity, not that captureNavigationFallbackVisualState picks the right thing. Two reverts of this fix would keep this test green: calling captureObservedAppScreen(app:) in captureNavigationFallbackVisualState, or passing .onUnresolvedWindow { self.captureResolvedAppScreen(app: self.springboard) }. That matters because the shipped rule for back --in-app is that the fallback never samples a second process — it must report APP_SCREEN_WINDOW_UNRESOLVED as an unknown outcome instead of reading SpringBoard as "unchanged" and returning a false "in-app back control is not available." The Duo run showed this route can't be reached live on a simulator, so this unit test is the only evidence backing that rule, and right now it doesn't test it. The fix is to move the choice that captureNavigationFallbackVisualState makes into a static the instance method calls, for example static func navigationFallbackSample(resolvingApp: () -> Result<...>, systemSurface: () -> Result<...>, encoding:), with production passing the real SpringBoard resolver; the test then passes a systemSurface closure that calls XCTFail and asserts .unverified(APP_SCREEN_WINDOW_UNRESOLVED) through inAppBackOutcome, so either revert turns it red.

Not blocking: the SystemSurfaceFallback enum and its .never case in RunnerAppScreenCapture.swift:57 add a seam with an 8-line comment for a path that has no behavior, and could fold into the fix above or be dropped for a direct captureResolvedAppScreen call; and the payload assertion in RunnerTests+RecordingTests.swift:91 hand-builds .unresolvedWindow instead of mapping the thrown error through recordingStartErrorPayload(for:) as the existing test at line 44 already does — take or leave both.

Does this need the new SystemSurfaceFallback type at all? Its .never case is an identity, so it can't be what makes the back route testable. A smaller design is a static navigation sampler that takes both the app resolver and the system-surface resolver, called by captureNavigationFallbackVisualState, with tests handing it a system-surface closure that fails if invoked; selectObservedScreenCapture could stay as it was at 7b5b504.

Smoke Tests fails in RunnerTests.testAbandonedTreeCaptureSkipsQuerySweepAndHonorsWarmupExemption (SnapshotCapturePlanOccupancyTests.swift:106/111): the viewport read timed out before the tree XPC was abandoned, which reads as a timing race on the snapshot capture-plan and occupancy route, and this PR only touches app-screen capture, record start, back fallback, the ADR, and the TS contract, so the failure looks unrelated. The claim that either revert keeps the current test green comes from reading the code, not from a run.

Before merge, the Navigation.swift:395 test needs to go through the production sampler so that reverting the fallback to SpringBoard sampling turns it red, and then Smoke Tests should be re-run to confirm the capture-plan timing failure is a flake and not tied to this change.

[thymikee]

Addressed in e5ad8662c (rebased onto main; was 6 behind). Follow-up to #2766 (comment).

Blocking — the test now goes through the production sampler. captureNavigationFallbackVisualState delegates to a new static navigationFallbackSample(resolvingApp:systemSurface:encoding:) that holds the app-only decision and never consults systemSurface. testVerifyNavigationFallbackOutcomeReportsUnresolvedWindowWithoutSystemSurface calls that same static and hands it a systemSurface closure that XCTFails if consulted. Confirmed on a real iOS build at e5ad8662c: making the static consult the system surface on .unresolvedWindow turns it red with XCTAssertFalse failed — the in-app fallback samples the app only, never SpringBoard; the clean build is green. The mapping revert still fails it too.

Dropped SystemSurfaceFallback / .never as suggested — its .never case was an identity that couldn't make the route testable. selectObservedScreenCapture is back to its 7b5b504 form; the enum and selectScreenCapture are gone.

Recording test now maps the actual thrown error through recordingStartErrorPayload(for:) rather than a re-typed literal, so the capture→throw→host-payload path is exercised end to end.

Smoke Tests (testAbandonedTreeCaptureSkipsQuerySweepAndHonorsWarmupExemption) — this PR touches only app-screen capture, record start, the back fallback, the ADR, and the TS contract, none of which reach the snapshot capture-plan/occupancy route; re-running on the fresh head to confirm the timing flake.

[thymikee]

This looks ready for review at e5ad866. The earlier findings on 16b39d2 (#2766 (comment)) are addressed: the regression test now calls the production navigationFallbackSample static, and the recording test maps the real thrown error through recordingStartErrorPayload.

Not blocking: navigationFallbackSample takes systemSurface only to discard it, so a revert at the one-line call site in captureNavigationFallbackVisualState would not turn the test red. Keeping that method a single-expression pass-through keeps the call site easy to check.

I read the code and did not run the Swift tests. All checks were still queued or running with no failures; the iOS Smoke Tests and runner unit lanes exercise the changed Swift, so they need to finish green on e5ad866.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-23 13:18 UTC