Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 128 additions & 0 deletions skills/cu-automation/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
---
name: cu-automation
description: Automate Windows desktop and GUI applications via cu CLI using evidence-bound observations and finite action plans. Use when controlling desktop apps, clicking UI elements, typing into Windows windows, or automating desktop workflows.
---

# cu Automation Skill

Automate Windows GUI tasks safely using the `cu` CLI. `cu` is an evidence-bound computer-use interface that executes finite pointer and keyboard plans against fresh desktop observations.

## Core Rules & Invariants

1. **Strict Evidence-Bound Authority (Single-Use Lifecycle)**:
- Input requires an active, unconsumed `observationId`.
- **One observation = exactly one `act`**: Any `act` command (whether `completed` or `checkpoint`) immediately consumes that `observationId`.
- Never reuse an `observationId`. For the next action, either use the new `checkpoint.observationId` returned in the receipt, or run `cu observe` again.
2. **Normalized Coordinates (`0..999`)**:
- Coordinates (`{"x": 0..999, "y": 0..999}`) are relative to the *observed image crop*, with origin `(0,0)` at the top-left and `(999,999)` at the bottom-right. Never use physical screen pixels in action files.
3. **Always Pass `--json`**:
- All commands must include `--json` for predictable, parseable machine receipts.
4. **Automatic Checkpoint Segmentation**:
- `cu` automatically ends an action segment at visual decision boundaries (before a second coordinate action, or after `drag`, `wheel`, focus keys like `Tab`/`Enter`/arrows, or `chord`).
- When a boundary is hit, `cu` executes only the safe prefix, captures a new observation, and returns `outcome: "checkpoint"`. **The unexecuted trailing actions are discarded and never run automatically.**
5. **No Blind Replay**:
- On exit 4 (`partial`) or exit 5 (`indeterminate`), never replay the old plan. Run `cu observe` anew to inspect the real desktop state before deciding the next step.
6. **Workspace Directory Stickiness & Run IDs**:
- All state is stored locally under `.cu/` in the current working directory. Keep your shell cwd consistent.
- `run_id` must use lowercase ASCII letters, digits, dots, underscores, and hyphens (`[a-z0-9._-]+`), e.g., `work-1`.

---

## Standard Loop (Observe -> Inspect -> Plan -> Act -> Verify)

```
[cu init] -> [cu observe] -> [Inspect PNG] -> [Write action.json] -> [cu act] -> [Check Outcome / Loop]
```

### Step 1: Initialize Workspace (Once per task)
```powershell
cu init --json
```

### Step 2: Observe Target Region
`cu` requires an explicit region selector. Unrestricted full-desktop capture is blocked in preview.
Syntax: `--region pixel:<left>,<top>,<width>,<height>` (L, T, W, H):
```powershell
cu observe work-1 --region pixel:100,100,1200,800 --json
```
*(Optional: Run `cu displays --json` only if multi-monitor topology inspection or `--display <display_id>` binding is required).*

Receipt:
```json
{
"kind": "cu.observe.result/v1",
"runId": "work-1",
"observationId": "obs_0123456789abcdef0123456789abcdef",
"imagePath": ".cu/work-1/captures/obs_0123456789abcdef0123456789abcdef.png",
"coordinateSpace": "normalized_999_top_left",
"expiresAt": "2026-07-23T00:05:00.000Z",
"actionable": true
}
```
*Note*: Observations expire after 300 seconds by default. Act promptly, or re-run `observe` if expired.

### Step 3: Inspect the Screenshot & Locate Coordinates
Inspect `imagePath` using your image inspection tool. Calculate target coordinates directly from this image:
- Top-left: `{"x": 0, "y": 0}`
- Center: `{"x": 500, "y": 500}`
- Bottom-right: `{"x": 999, "y": 999}`

### Step 4: Plan & Execute Actions
Write the plan to a UTF-8 (no BOM) JSON file, then execute:
```json
// action.json
{
"kind": "cu.action/v1",
"observationId": "obs_0123456789abcdef0123456789abcdef",
"coordinateSpace": "normalized_999_top_left",
"actions": [
{ "kind": "click", "at": { "x": 450, "y": 320 } },
{ "kind": "type_text", "text": "Hello World" }
]
}
```
```powershell
cu act work-1 --action-file action.json --json
```

> **Best Practice**: Keep plans micro-batched (1 coordinate action + subsequent text/key). Because a second click or a focus key cuts a checkpoint and discards trailing actions, planning one interaction at a time prevents discarded steps.

### Step 5: Check Outcome & Branch

Read the JSON result:
- **`outcome: "completed"`**: All actions executed. Re-observe if further actions are needed.
- **`outcome: "checkpoint"`**: Truncated at a boundary. Receipt contains:
```json
{
"kind": "cu.act.result/v1",
"outcome": "checkpoint",
"checkpoint": {
"observationId": "obs_fedcba9876543210fedcba9876543210",
"imagePath": ".cu/work-1/captures/obs_fedcba9876543210fedcba9876543210.png",
"actionable": true
}
}
```
**Use `checkpoint.observationId` and inspect `checkpoint.imagePath` for your next plan.** Do not replay the old plan.

---

## Exit Code & Failure Recovery Matrix

| Exit Code | Classification | What Happened | Action to Take |
| :---: | :--- | :--- | :--- |
| **0** | Success / Checkpoint | Execution succeeded or reached a clean checkpoint. | Proceed to next step, or consume `checkpoint.observationId`. |
| **2** | Action File Error | Bad JSON, unrecognized keys, invalid key names, or coordinates out of range. | No desktop effect. Fix `action.json` syntax and retry with the **same** `observationId`. |
| **3** | Blocked | Action blocked before any input was emitted (e.g. `observation_expired`, `observation_consumed`, `run_busy`). | Inspect error receipt `retryable` field. If `retryable: true`, run `cu observe` again and retry with the new `observationId`. If `run_busy`, check `cu status work-1 --json`. |
| **4** | Partial Effect | Some input was emitted, but execution was interrupted or could not be proven. | **DO NOT replay the plan.** Run `cu observe` to inspect current desktop state. |
| **5** | Indeterminate | State could not be proven after input emission. | **DO NOT replay the plan.** Run `cu observe` to inspect current desktop state. |

---

## Diagnosis & Cleanup

- **Diagnose run state**: Run `cu status work-1 --json` to inspect lifecycle, locks, and active observation validity without side effects.
- **Cleanup when task is complete**: Captures consume quota in `.cu/<run_id>/captures/`. Run `cu clearall work-1 --json` only after the entire task is finished (it invalidates the current live observation).

For the action file JSON schema and key whitelist, see:
- [references/action-file-spec.md](references/action-file-spec.md)
130 changes: 130 additions & 0 deletions skills/cu-automation/references/action-file-spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
# cu.action/v1 Specification & Reference

`cu.action/v1` is an exact-key, strictly validated JSON format for emitting pointer and keyboard operations.

## Root Object Format

```json
{
"kind": "cu.action/v1",
"observationId": "obs_0123456789abcdef0123456789abcdef",
"coordinateSpace": "normalized_999_top_left",
"actions": []
}
```

- `kind`: must be literal `"cu.action/v1"`.
- `observationId`: copy verbatim from the latest `observe` or `checkpoint` receipt (`obs_` + 32 lowercase hex characters). Never invent or construct IDs.
- `coordinateSpace`: must be literal `"normalized_999_top_left"`.
- `actions`: array of actions (total leaf actions capped at 32; keep plans to 1–2 actions for predictable micro-batching).

Validation rules:
- **Exact keys only**: Any extra or unknown key at any depth rejects the entire file with no desktop effect.
- No duplicate object keys.
- UTF-8 without BOM, max 64KB, max nesting depth 64.

---

## Primary Action Variants

### 1. `click` (Most Common)
```json
{
"kind": "click",
"at": { "x": 500, "y": 500 },
"button": "left",
"count": 1
}
```
- `at`: required `Point` (`x: 0..999, y: 0..999`).
- `button` (optional): `"left"`, `"middle"`, or `"right"`. Default is `"left"`.
- `count` (optional): `1` (single click) or `2` (double click). Default is `1`.

### 2. `type_text` (Text Input)
```json
{ "kind": "type_text", "text": "user@example.com" }
```
- For plain text, prefer `type_text` over multiple `key` actions.
- Max 2,048 Unicode scalar values (max 8,192 UTF-8 bytes).
- Non-empty; no NUL, C0 controls, or DEL characters.

### 3. `key` (Single Key Press)
```json
{ "kind": "key", "key": "Enter" }
```
- `key`: valid portable key name from the whitelist below.
- *Note*: Focus keys (`Tab`, `Enter`, `Escape`, `Arrow*`) trigger a checkpoint boundary after execution.

### 4. `chord` (Simultaneous Key Combinations)
```json
{ "kind": "chord", "keys": ["Control", "KeyA"] }
```
- 2 to 6 distinct keys.
- **Prohibited chords**: `Control+Alt+Delete` and `Meta+KeyL` are strictly rejected.

### 5. `drag` (Pointer Dragging)
```json
{
"kind": "drag",
"from": { "x": 100, "y": 100 },
"to": { "x": 400, "y": 400 },
"button": "left",
"durationMs": 500
}
```
- `from` and `to` points must differ.
- `button` (optional): `"left"` or `"right"`. Default: `"left"`.
- `durationMs` (optional): `100..5000` ms. Default: `500`.
- *Note*: Always triggers a checkpoint boundary after execution.

### 6. `wheel` (Vertical Scroll)
```json
{
"kind": "wheel",
"at": { "x": 500, "y": 500 },
"deltaY": -3
}
```
- `deltaY`: non-zero integer in range `-100..100` (positive = scroll up, negative = scroll down).
- *Note*: Always triggers a checkpoint boundary after execution.

### 7. `pointer_move` (Hover / Reposition)
```json
{ "kind": "pointer_move", "to": { "x": 500, "y": 500 } }
```

---

## Rare Variant: `sequence` (Modifier-Held Actions)

Used rarely, only when holding a modifier (such as Ctrl) during a click or drag:
```json
{
"kind": "sequence",
"steps": [
{ "action": { "kind": "key_down", "key": "Control" } },
{ "action": { "kind": "click", "at": { "x": 500, "y": 500 } } },
{ "action": { "kind": "key_up", "key": "Control" } }
]
}
```
- Must contain exactly 1 coordinate action (`click`, `drag`, `pointer_move`, `wheel`).
- Modifiers allowed for `key_down`/`key_up`: `"Control"`, `"Alt"`, `"Shift"`.
- **Strict balance**: Every `key_down` must have a matching `key_up` in reverse order. No keys may remain held.

---

## Key Name Whitelist (Common Keys)

> **Important**: Never use lowercase letters (`'a'`) or raw characters (`'1'`). Use the canonical names below.

- **Letters**: `KeyA` through `KeyZ`
- **Digits**: `Digit0` through `Digit9`
- **Common Navigation**: `Enter`, `Tab`, `Escape`, `Space`, `Backspace`, `Delete`
- **Directional**: `ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight`
- **Paging / Bounds**: `Home`, `End`, `PageUp`, `PageDown`, `Insert`
- **Modifiers**: `Shift`, `Control`, `Alt`, `Meta`
- **Function Keys**: `F1` through `F24`
- **Punctuation**: `Semicolon`, `Equal`, `Comma`, `Minus`, `Period`, `Slash`, `Backquote`, `BracketLeft`, `Backslash`, `BracketRight`, `Quote`

*(Run `cu help action-file` in the workspace for the full machine grammar if encountering unlisted keys).*