diff --git a/skills/cu-automation/SKILL.md b/skills/cu-automation/SKILL.md new file mode 100644 index 0000000..e4b139e --- /dev/null +++ b/skills/cu-automation/SKILL.md @@ -0,0 +1,128 @@ +--- +name: cu-automation +description: Automate Windows desktop and GUI applications via cu CLI using evidence-bound observations and finite action plans. Use when controlling desktop apps, clicking UI elements, typing into Windows windows, or automating desktop workflows. +--- + +# cu Automation Skill + +Automate Windows GUI tasks safely using the `cu` CLI. `cu` is an evidence-bound computer-use interface that executes finite pointer and keyboard plans against fresh desktop observations. + +## Core Rules & Invariants + +1. **Strict Evidence-Bound Authority (Single-Use Lifecycle)**: + - Input requires an active, unconsumed `observationId`. + - **One observation = exactly one `act`**: Any `act` command (whether `completed` or `checkpoint`) immediately consumes that `observationId`. + - Never reuse an `observationId`. For the next action, either use the new `checkpoint.observationId` returned in the receipt, or run `cu observe` again. +2. **Normalized Coordinates (`0..999`)**: + - Coordinates (`{"x": 0..999, "y": 0..999}`) are relative to the *observed image crop*, with origin `(0,0)` at the top-left and `(999,999)` at the bottom-right. Never use physical screen pixels in action files. +3. **Always Pass `--json`**: + - All commands must include `--json` for predictable, parseable machine receipts. +4. **Automatic Checkpoint Segmentation**: + - `cu` automatically ends an action segment at visual decision boundaries (before a second coordinate action, or after `drag`, `wheel`, focus keys like `Tab`/`Enter`/arrows, or `chord`). + - When a boundary is hit, `cu` executes only the safe prefix, captures a new observation, and returns `outcome: "checkpoint"`. **The unexecuted trailing actions are discarded and never run automatically.** +5. **No Blind Replay**: + - On exit 4 (`partial`) or exit 5 (`indeterminate`), never replay the old plan. Run `cu observe` anew to inspect the real desktop state before deciding the next step. +6. **Workspace Directory Stickiness & Run IDs**: + - All state is stored locally under `.cu/` in the current working directory. Keep your shell cwd consistent. + - `run_id` must use lowercase ASCII letters, digits, dots, underscores, and hyphens (`[a-z0-9._-]+`), e.g., `work-1`. + +--- + +## Standard Loop (Observe -> Inspect -> Plan -> Act -> Verify) + +``` +[cu init] -> [cu observe] -> [Inspect PNG] -> [Write action.json] -> [cu act] -> [Check Outcome / Loop] +``` + +### Step 1: Initialize Workspace (Once per task) +```powershell +cu init --json +``` + +### Step 2: Observe Target Region +`cu` requires an explicit region selector. Unrestricted full-desktop capture is blocked in preview. +Syntax: `--region pixel:,,,` (L, T, W, H): +```powershell +cu observe work-1 --region pixel:100,100,1200,800 --json +``` +*(Optional: Run `cu displays --json` only if multi-monitor topology inspection or `--display ` binding is required).* + +Receipt: +```json +{ + "kind": "cu.observe.result/v1", + "runId": "work-1", + "observationId": "obs_0123456789abcdef0123456789abcdef", + "imagePath": ".cu/work-1/captures/obs_0123456789abcdef0123456789abcdef.png", + "coordinateSpace": "normalized_999_top_left", + "expiresAt": "2026-07-23T00:05:00.000Z", + "actionable": true +} +``` +*Note*: Observations expire after 300 seconds by default. Act promptly, or re-run `observe` if expired. + +### Step 3: Inspect the Screenshot & Locate Coordinates +Inspect `imagePath` using your image inspection tool. Calculate target coordinates directly from this image: +- Top-left: `{"x": 0, "y": 0}` +- Center: `{"x": 500, "y": 500}` +- Bottom-right: `{"x": 999, "y": 999}` + +### Step 4: Plan & Execute Actions +Write the plan to a UTF-8 (no BOM) JSON file, then execute: +```json +// action.json +{ + "kind": "cu.action/v1", + "observationId": "obs_0123456789abcdef0123456789abcdef", + "coordinateSpace": "normalized_999_top_left", + "actions": [ + { "kind": "click", "at": { "x": 450, "y": 320 } }, + { "kind": "type_text", "text": "Hello World" } + ] +} +``` +```powershell +cu act work-1 --action-file action.json --json +``` + +> **Best Practice**: Keep plans micro-batched (1 coordinate action + subsequent text/key). Because a second click or a focus key cuts a checkpoint and discards trailing actions, planning one interaction at a time prevents discarded steps. + +### Step 5: Check Outcome & Branch + +Read the JSON result: +- **`outcome: "completed"`**: All actions executed. Re-observe if further actions are needed. +- **`outcome: "checkpoint"`**: Truncated at a boundary. Receipt contains: + ```json + { + "kind": "cu.act.result/v1", + "outcome": "checkpoint", + "checkpoint": { + "observationId": "obs_fedcba9876543210fedcba9876543210", + "imagePath": ".cu/work-1/captures/obs_fedcba9876543210fedcba9876543210.png", + "actionable": true + } + } + ``` + **Use `checkpoint.observationId` and inspect `checkpoint.imagePath` for your next plan.** Do not replay the old plan. + +--- + +## Exit Code & Failure Recovery Matrix + +| Exit Code | Classification | What Happened | Action to Take | +| :---: | :--- | :--- | :--- | +| **0** | Success / Checkpoint | Execution succeeded or reached a clean checkpoint. | Proceed to next step, or consume `checkpoint.observationId`. | +| **2** | Action File Error | Bad JSON, unrecognized keys, invalid key names, or coordinates out of range. | No desktop effect. Fix `action.json` syntax and retry with the **same** `observationId`. | +| **3** | Blocked | Action blocked before any input was emitted (e.g. `observation_expired`, `observation_consumed`, `run_busy`). | Inspect error receipt `retryable` field. If `retryable: true`, run `cu observe` again and retry with the new `observationId`. If `run_busy`, check `cu status work-1 --json`. | +| **4** | Partial Effect | Some input was emitted, but execution was interrupted or could not be proven. | **DO NOT replay the plan.** Run `cu observe` to inspect current desktop state. | +| **5** | Indeterminate | State could not be proven after input emission. | **DO NOT replay the plan.** Run `cu observe` to inspect current desktop state. | + +--- + +## Diagnosis & Cleanup + +- **Diagnose run state**: Run `cu status work-1 --json` to inspect lifecycle, locks, and active observation validity without side effects. +- **Cleanup when task is complete**: Captures consume quota in `.cu//captures/`. Run `cu clearall work-1 --json` only after the entire task is finished (it invalidates the current live observation). + +For the action file JSON schema and key whitelist, see: +- [references/action-file-spec.md](references/action-file-spec.md) diff --git a/skills/cu-automation/references/action-file-spec.md b/skills/cu-automation/references/action-file-spec.md new file mode 100644 index 0000000..0ad6982 --- /dev/null +++ b/skills/cu-automation/references/action-file-spec.md @@ -0,0 +1,130 @@ +# cu.action/v1 Specification & Reference + +`cu.action/v1` is an exact-key, strictly validated JSON format for emitting pointer and keyboard operations. + +## Root Object Format + +```json +{ + "kind": "cu.action/v1", + "observationId": "obs_0123456789abcdef0123456789abcdef", + "coordinateSpace": "normalized_999_top_left", + "actions": [] +} +``` + +- `kind`: must be literal `"cu.action/v1"`. +- `observationId`: copy verbatim from the latest `observe` or `checkpoint` receipt (`obs_` + 32 lowercase hex characters). Never invent or construct IDs. +- `coordinateSpace`: must be literal `"normalized_999_top_left"`. +- `actions`: array of actions (total leaf actions capped at 32; keep plans to 1–2 actions for predictable micro-batching). + +Validation rules: +- **Exact keys only**: Any extra or unknown key at any depth rejects the entire file with no desktop effect. +- No duplicate object keys. +- UTF-8 without BOM, max 64KB, max nesting depth 64. + +--- + +## Primary Action Variants + +### 1. `click` (Most Common) +```json +{ + "kind": "click", + "at": { "x": 500, "y": 500 }, + "button": "left", + "count": 1 +} +``` +- `at`: required `Point` (`x: 0..999, y: 0..999`). +- `button` (optional): `"left"`, `"middle"`, or `"right"`. Default is `"left"`. +- `count` (optional): `1` (single click) or `2` (double click). Default is `1`. + +### 2. `type_text` (Text Input) +```json +{ "kind": "type_text", "text": "user@example.com" } +``` +- For plain text, prefer `type_text` over multiple `key` actions. +- Max 2,048 Unicode scalar values (max 8,192 UTF-8 bytes). +- Non-empty; no NUL, C0 controls, or DEL characters. + +### 3. `key` (Single Key Press) +```json +{ "kind": "key", "key": "Enter" } +``` +- `key`: valid portable key name from the whitelist below. +- *Note*: Focus keys (`Tab`, `Enter`, `Escape`, `Arrow*`) trigger a checkpoint boundary after execution. + +### 4. `chord` (Simultaneous Key Combinations) +```json +{ "kind": "chord", "keys": ["Control", "KeyA"] } +``` +- 2 to 6 distinct keys. +- **Prohibited chords**: `Control+Alt+Delete` and `Meta+KeyL` are strictly rejected. + +### 5. `drag` (Pointer Dragging) +```json +{ + "kind": "drag", + "from": { "x": 100, "y": 100 }, + "to": { "x": 400, "y": 400 }, + "button": "left", + "durationMs": 500 +} +``` +- `from` and `to` points must differ. +- `button` (optional): `"left"` or `"right"`. Default: `"left"`. +- `durationMs` (optional): `100..5000` ms. Default: `500`. +- *Note*: Always triggers a checkpoint boundary after execution. + +### 6. `wheel` (Vertical Scroll) +```json +{ + "kind": "wheel", + "at": { "x": 500, "y": 500 }, + "deltaY": -3 +} +``` +- `deltaY`: non-zero integer in range `-100..100` (positive = scroll up, negative = scroll down). +- *Note*: Always triggers a checkpoint boundary after execution. + +### 7. `pointer_move` (Hover / Reposition) +```json +{ "kind": "pointer_move", "to": { "x": 500, "y": 500 } } +``` + +--- + +## Rare Variant: `sequence` (Modifier-Held Actions) + +Used rarely, only when holding a modifier (such as Ctrl) during a click or drag: +```json +{ + "kind": "sequence", + "steps": [ + { "action": { "kind": "key_down", "key": "Control" } }, + { "action": { "kind": "click", "at": { "x": 500, "y": 500 } } }, + { "action": { "kind": "key_up", "key": "Control" } } + ] +} +``` +- Must contain exactly 1 coordinate action (`click`, `drag`, `pointer_move`, `wheel`). +- Modifiers allowed for `key_down`/`key_up`: `"Control"`, `"Alt"`, `"Shift"`. +- **Strict balance**: Every `key_down` must have a matching `key_up` in reverse order. No keys may remain held. + +--- + +## Key Name Whitelist (Common Keys) + +> **Important**: Never use lowercase letters (`'a'`) or raw characters (`'1'`). Use the canonical names below. + +- **Letters**: `KeyA` through `KeyZ` +- **Digits**: `Digit0` through `Digit9` +- **Common Navigation**: `Enter`, `Tab`, `Escape`, `Space`, `Backspace`, `Delete` +- **Directional**: `ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight` +- **Paging / Bounds**: `Home`, `End`, `PageUp`, `PageDown`, `Insert` +- **Modifiers**: `Shift`, `Control`, `Alt`, `Meta` +- **Function Keys**: `F1` through `F24` +- **Punctuation**: `Semicolon`, `Equal`, `Comma`, `Minus`, `Period`, `Slash`, `Backquote`, `BracketLeft`, `Backslash`, `BracketRight`, `Quote` + +*(Run `cu help action-file` in the workspace for the full machine grammar if encountering unlisted keys).*