Skip to content

feat(apple): add watchOS Simulator runtime - #3003

Open
csark0812 wants to merge 9 commits into
callstack:mainfrom
csark0812:chris/agent/watchos-runtime
Open

csark0812 wants to merge 9 commits into
callstack:mainfrom
csark0812:chris/agent/watchos-runtime

Conversation

@csark0812

@csark0812 csark0812 commented Sep 27, 2026 •

Copy link
Copy Markdown

Summary

  • discover installed watchOS Simulator devices as first-class Apple targets
  • add an isolated CoreSimulator HID backend for touch, single-pointer gestures, Digital Crown scrolling, and Crown navigation
  • serve watchOS accessibility snapshots through the existing host AX bridge and screenshots through simctl
  • admit app lifecycle and installation only for watchOS Simulator while keeping physical devices and unsupported operations fail-closed

Verification

  • pnpm format:check
  • pnpm lint
  • pnpm typecheck
  • pnpm build
  • pnpm test:unit — 1,414 files; 11,475 passed; 1 skipped
  • pnpm package:npm release pipeline components: all Apple runner builds and macOS helper passed; Android assets prepared with API 36; pnpm check:package passed
  • live watchOS 27.0 Simulator: discovery, Calculator launch, 29-node accessibility snapshot, semantic ref tap, screenshot, Crown scroll, and Crown navigation

Runtime boundary

The backend checks the selected Simulator's simctl io ... enumerate output for a LegacyHID display and derives its pixel geometry and scale before advertising interaction. Text entry, app switcher, orientation, settings, multi-touch, and physical watchOS devices remain unsupported.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 25 files

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread packages/platform-apple/src/inventory-classification.ts Outdated
Comment thread docs/adr/0019-request-bound-platform-runtime.md Outdated
Comment thread docs/adr/0019-request-bound-platform-runtime.md Outdated
Comment thread apple/watch-helper/WatchControl.m
Comment thread packages/platform-apple/src/watch/interactor.ts
Comment thread packages/platform-apple/src/runtime.test.ts
Comment thread docs/adr/0009-apple-platform-consolidation.md Outdated
Comment thread packages/platform-apple/src/watch/watch-helper-cache.ts
Comment thread packages/platform-apple/src/snapshot-source/cache.ts Outdated
Comment thread packages/platform-apple/src/runtime.ts Outdated
@thymikee

Copy link
Copy Markdown
Member

This adds real value (a watchOS Simulator runtime) but at 3995110 it isn't ready to merge, mostly because the physical-watch boundary and the classification it depends on aren't actually enforced.

watch-helper-cache.ts:29 (ensureWatchHelperBinary, createWatchHelperCacheHost, compileWatchHelper, watchHelperBuildFailed) copies fold-helper-cache.ts line for line, down to the fake host in the test file. A fix to one helper's cache or timeout handling will drift from the other. Can this become one parameterized host-helper cache next to native-build-cache.ts, taking {sourceFilename, binaryFilename, cacheDir, lockDescription, hint, reason, argv}, with the fold and watch helpers rebased onto it and one shared fake-host fixture?

WatchControl.m:79's simulatorDevice() resolves the UDID only through defaultDeviceSetWithError:, but discovery stamps simulatorSetPath on watch simulators and runWatchHelper only passes device.id. With --ios-simulator-device-set, every tap, press, longpress, gesture, scroll, back and home on a watch simulator will fail with COMMAND_FAILED ('Watch Simulator HID client is unavailable') while facts still say the device is available. Every host-side call that addresses a simulator should resolve it through simulatorAddressFor(device); can the resolved set path be passed to the helper and opened with the set-path SimServiceContext API, with a test asserting the helper argv carries the set?

inventory-classification.ts:6's APPLE_WATCH_PATTERN (/\b(apple watch|watchos|watch)\b/i) is tested against every descriptor, including device.name for simulators and the xctrace/devicectl name labels for physical devices. An iOS simulator or iPhone named 'Watch QA' gets classified as appleOs: 'watchos', losing typing, fill, app switcher and XCTest routing (or hitting the UNSUPPORTED_PLATFORM sentinel for a physical device), which regresses existing iOS targets. Can watchOS classification key only on identifiers the OS stamps — the simctl runtime key (SimRuntime.watchOS-) or deviceTypeIdentifier (SimDeviceType.Apple-Watch-), and the devicectl platform or productType — never the user-chosen name? An inventory row for an iOS simulator named 'Watch' would catch a regression here.

The watch alternation in APPLE_PRODUCT_TYPE_PATTERN (inventory-classification.ts:3) makes devicectl list a paired physical Apple Watch as kind: 'device', appleOs: 'watchos'. Since the watchos leaf refusals were removed from appleGesturePlanFact and appleGestureViewportFact (gesture-facts.ts:60), and runtime.ts:299-300 makes ensureReady/bootTarget available for every watchOS kind, a physical watch now passes admission for gesture, scroll and readiness and only fails inside createAppleInteractor's UNSUPPORTED_PLATFORM throw — contradicting the PR's stated physical-watch boundary, and untested because the fact tables only carry a watch simulator row. Should every watchOS fact require kind === 'simulator' (or should physical watches never reach inventory at all)? The simplest fix is reverting the productType alternation; otherwise every fact needs to route through one appleWatchSimulator predicate, with a watchOS kind: 'device' row added to the gesture, runtime and navigation fact tables.

watch/interactor.ts:122's scrollWatch reads only options.amount, so scroll down --pixels 300 sends the default Crown delta while scrollResult (daemon/scroll-runtime.ts:423) still echoes pixels: 300 as honored, and scroll left/scroll right map to a vertical Crown delta while the response still reports direction: 'left'/'right'; durationMs is dropped too. The response claims a distance and direction the device didn't perform, so agents and replays act on false movement. Can pixels, horizontal directions and durationMs be refused with a typed UNSUPPORTED_OPERATION, or declared unavailable in the watch scroll facts? This also needs watch/interactor.test.ts, since the one-to-one test topology rule requires it anyway, asserting the refusal codes.

snapshot-route.ts:265 now admits watchOS simulators as isEligible, but every failure path and off-route case ends in fallback(), and the watch interactor's snapshot() always throws UNSUPPORTED_OPERATION. opensGenerationCircuit retires the app generation after one non-preparing failure, so after one transient bridge failure, every later snapshot of that generation throws an admission-style refusal while captureSnapshot is still advertised as available — a mechanism failure reported as unsupported. For watchOS, can the bridge failure surface as a typed failure carrying its bridgeFailureCode, skip the generation circuit since there's no fallback to open onto, and refuse preferredBackend through facts instead?

Not blocking: the only test of the new backend (watchos-sentinel.test.ts:351) just checks methods are toBeTypeOf('function') with no watch/interactor.test.ts mirror, watchViewport probes simctl io enumerate before ensureBootedSimulator boots the simulator (and its cache Map is never invalidated), and the docs/ADR/CHANGELOG/comment updates (ADR 0009, commands.md:69,230, screenshot-crop-target.ts:95, device.ts:274's missing explicit watchos case) are all worth doing but can be taken or left for a follow-up.

Is the size of this change proportionate to what it needs to do? The production diff is +641/−60 lines under the usual threshold, but it still touches kernel, discovery, facts and snapshot with roughly a dozen scattered appleOs === 'watchos' ? (kind === 'simulator' ? … : …) : … branches in runtime.ts, navigation/runtime.ts, gesture-facts.ts and deployment/runtime.ts; would one declared watchOS-simulator fact profile, owned by the platform-apple runtime, collapse those branches into a single owning type, alongside dropping the productType watch alternation, keying watchOS on runtime/device-type identifiers only, and sharing one host-helper cache?

I did not run this on a device, so I can't confirm the HID message layout in WatchControl.m or the host AX bridge behavior on watchsimulator, the darwin -Werror compile of WatchControl.m (so the missing <math.h> for isfinite is unconfirmed), that current Xcode devicectl actually lists paired Apple Watches with a productType starting Watch… (f4 depends on this), or how often the watch bridge hits non-preparing failures in practice (relevant to f6). On a watchOS Simulator at this PR's head, merge-readiness needs the raw output of devices --platform apple showing the watch as appleOs: watchos (and an iOS simulator named 'Watch' staying ios), open <watch bundle>, snapshot with diagnostics showing the host bridge built with watchsimulator, click @ref returning backend watchos-coresimulator, scroll down plus the refusal codes for scroll left and scroll down --pixels 300, back/home via the Crown, screenshot, and a repeat of tap and scroll with --ios-simulator-device-set pointing at a non-default set containing the watch.

CI shows one check reported for this cross-repo PR, and it's green; no failing job overlaps the diff, but the full unit suite, typecheck and the darwin -Werror helper gate aren't visible here, so this green result doesn't cover those lanes.

The path to merge is: address the shared-cache duplication, the device-set-aware helper addressing, identifier-only watchOS classification, the physical-watch fact gap, honest scroll-input refusals, and typed bridge failures for watchOS snapshots, then attach the live watchOS Simulator transcript described above.

@thymikee

Copy link
Copy Markdown
Member

@csark0812 can you send some demos of how this works?

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 25 files (changes from recent commits).

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread packages/platform-apple/src/runtime.ts Outdated
Comment thread packages/platform-apple/src/snapshot-route.ts Outdated
Comment thread packages/platform-apple/src/inventory-classification.test.ts
Comment thread packages/platform-apple/src/runtime.test.ts Outdated
Comment thread packages/platform-apple/src/watch/interactor.test.ts
Comment thread packages/platform-apple/src/snapshot-route.ts Outdated
@csark0812

Copy link
Copy Markdown
Author

I added regression coverage for the physical-watch boundary, missing LegacyHID refusal, and normalized tap coordinates; the iOS XCTest build-for-testing also compiles the Watch runtime source. I cannot provide a live watchOS demo from this host: CoreSimulatorService is refusing connections and reports no available runtimes. The demo/live runtime proof remains outstanding, and I have not claimed it.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 9 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread packages/platform-apple/src/snapshot-route.ts Outdated
Comment thread packages/platform-apple/src/runtime.test.ts Outdated
Comment thread packages/platform-apple/src/watch/interactor.test.ts
@csark0812

Copy link
Copy Markdown
Author

Live watchOS Simulator demo run on Apple Watch SE 3 (40mm), watchOS 27.0 (UDID 669DABEF-3AB2-41B1-B86A-DEB883C7FC48): opened the installed Calculator app with agent-device, captured its accessibility tree (29 nodes including Calculator and semantic button refs), tapped digit 5 twice through the watchos-coresimulator backend, and captured the screen showing 55; then cleared the input and closed the session. This verifies the public open/snapshot/tap/screenshot path on a real booted watch runtime. Boundary: this host does not have the companion-marker/nested-widget fixture installed on this unclaimed Watch, so this is not the full app+WidgetKit fixture demo. The Calculator AX tree also does not expose its changing numeric display value, so the post-tap oracle here is the screenshot.

@thymikee

Copy link
Copy Markdown
Member

Thanks for the update. The Crown scroll, back and home work and the snapshot route look better since 3995110, but I found two problems at 105a776 that need a fix before merge.

The first problem is a regression for existing iOS targets. The comment in inventory-classification.ts says watchOS identity must never come from a user-editable label. But resolveAppleOs still tests the watch pattern against every descriptor, and the callers still pass user names: device.name in simulator-inventory.ts:45, the xctrace name in physical-inventory.ts:55, and device.name in devicectlLabels. An iOS simulator named "watchOS Companion", or an iPhone named "iPhone (watchOS pair)", becomes appleOs: watchos. That target then loses XCTest routing, type, fill and app-switcher, and goes to the watch HID and snapshot paths, or becomes a refused physical watch. The new test "simulator display names never override the Xcode runtime platform" only uses the name "Watch", so it does not cover this. The rule should be: decide watchOS only from OS-stamped identifiers, never from a name. That means the runtime key and deviceTypeIdentifier for simctl, hardwareProperties.platform and productType for devicectl, and the osVersion string only for xctrace. Please pass identifiers and labels to resolveAppleOs separately and test the watch pattern against identifiers only. Please also add inventory rows for an iOS simulator and a devicectl iPhone that carry that name, and assert both stay ios.

The second problem is the HID probe in runtime.ts. On every fact inspection, inspectWatchHidAvailability runs simctl io <udid> enumerate with a 1.5 s timeout, and it returns false on any failure. That includes a shut-down simulator, a cold CoreSimulatorService that takes longer than 1.5 s, and a thrown error. Then tap, longPress, focus, scroll and gestures are refused with unsupported-device-kind, and back and home with unsupported-platform-leaf, before the interactor runs. So the new ensureBootedSimulator and the typed watchos-simulator-hid-unavailable refusal never run on the real route. A timing or state failure is reported as a permanent capability refusal with the wrong reason, and each watch request pays one or two extra simctl spawns. The gesture-facts test for the HID case asserts unsupported-device-kind for a failed probe, so it locks in the wrong reason. The rule should be: facts decide availability from static identity only (kind, simulatorSetPath, runtime), and the runtime HID check belongs to the interactor, which boots first and then refuses with the typed reason. Could the smaller design work here? Keep the watchOS facts static and let the interactor's existing typed HID refusal own the runtime check. That would delete the probe, the watchHidAvailable parameter threaded through gesture-facts.ts, navigation/runtime.ts and runtime.ts, and the override in runtime.fixtures.ts. Nothing has to change first, and it is a deletion inside platform-apple. If the probe has to stay, please run it only for booted simulators and report an unavailable result with watchos-simulator-hid-unavailable.

Not blocking, and you can take or leave these: in snapshot-route.ts:320 a "preparing" bridge failure on watchOS throws COMMAND_FAILED with retryable: true, but nothing reads details.retryable, so waiting for the in-flight build within the capture deadline may be better; in runtime.test.ts:431 the back and home expectations come from binding.facts.operations.tapPoint.available, which is production output, so literal per-fixture expectations would be safer; the two watch interactor refusals are matched by message rather than typed reason; and no test captures a second time after a watch bridge failure to show that the generation circuit stays closed.

The one reported check passes. This PR shows no full unit, typecheck or darwin -Werror helper lanes, so a green result does not cover them. I did not run tests or builds, I did not compile WatchControl.m with -Werror, and I did not confirm that simctl io <udid> enumerate fails on a shut-down simulator. The shut-down case depends on that, but the timeout and thrown-error cases do not.

Your demo covers open, snapshot, tap and screenshot on a booted watch, and I have not seen it run myself. Before merge, please attach raw CLI output from a watchOS Simulator at 105a776 for these: devices --platform apple showing the watch as appleOs: watchos while an iOS simulator named "watchOS Companion" stays ios; scroll down and scroll up returning backend watchos-coresimulator, with a before and after snapshot showing the list moved; scroll left and scroll down --pixels 300 returning UNSUPPORTED_OPERATION; back and home through the Crown, each with a snapshot or screenshot showing the navigation happened; click on the watch in a non-default --ios-simulator-device-set returning the typed refusal; and, after the second fix, click against a shut-down watch simulator either booting it and succeeding, or refusing with the HID-specific reason and not unsupported-device-kind.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants