---
name: snappy-desktop
role: macOS desktop automation primitive via Midscene vision AI. Controls mouse and keyboard from screen vision for any native app.
loaded-by: PreToolUse hook (auto-injected when "snappy-desktop" is mentioned)
Triggers on: macOS automation, native app, Midscene, mouse keyboard, screenshot, Finder
---

# snappy-desktop -- Agent Loader

You are operating the macOS desktop automation primitive. Wraps `npx -y @midscene/computer@1` (Gemini 2.5 Flash via OpenRouter) to give any Snappy skill "hands on the screen" for native apps -- Finder, System Settings, Messages, Notes, Mail, third-party apps. For websites use snappy-browse. For iMessage default to snappy-imessage first.

## API module

```typescript
import { runMidscene, screenshot } from "../snappy-desktop/api.ts";
```

| Function | Purpose |
|----------|---------|
| `runMidscene(instruction)` | Run a Midscene vision instruction via Terminal routing |
| `screenshot()` | Take a screenshot and return the file path |
| `whatsappReadChats()` | Open WhatsApp.app, screenshot chat list, extract chat names + unread badges via Midscene |
| `whatsappReadChat(contactName)` | Click into a specific chat, screenshot, extract recent messages |
| `whatsappSendMessage(contactName, text)` | Open chat with contact, type and send message via clipboard + Enter |

CLI:
```bash
npx tsx ~/.claude/skills/snappy-desktop/api.ts run "click the Settings icon"
npx tsx ~/.claude/skills/snappy-desktop/api.ts screenshot
npx tsx ~/.claude/skills/snappy-desktop/api.ts wa-chats
npx tsx ~/.claude/skills/snappy-desktop/api.ts wa-read "John Smith"
npx tsx ~/.claude/skills/snappy-desktop/api.ts wa-send "John Smith" "Hey, are you free tomorrow?"
```

## Rules

1. **SSH routing required.** Every Midscene command runs from a script file launched via `osascript -e 'tell application "Terminal" to do script "..."'`. Never invoke Midscene directly over SSH -- Screen Recording permission lives on Terminal.app.
2. **Unique RUN_ID.** `RUN_ID=$(date +%s)` in every script + output filename. No reuse of `/tmp/midscene-cmd.sh`.
3. **Poll for done marker.** `while [ ! -f /tmp/midscene-done-${RUN_ID} ]; do sleep 2; done`. Never fixed-sleep.
4. **Screenshot first.** `take_screenshot` before clicking blind.
5. **`--prompt`, not `--action`.** The Midscene flag is `--prompt`. Wrong flag = silent failure.
6. **AppleScript for app launch.** `osascript -e 'tell application "X" to activate'` is instant. Vision is for seeing, not for things you can do without sight.
7. **`act` for multi-step.** Single `act --prompt "step1, step2, step3"` handles transient UI (Spotlight, menus) better than chained atomic calls.
8. **Credentials via `snappy-settings/.env.cache`.** Never hardcode. Load via `env("KEY")` (source `load-env.sh`). Use clipboard hand-off for password fields, never `input --value "password"`.
9. **Charlotte MCP browser tools are forbidden** -- they don't work for desktop OR browsers.
10. **Source env inside scripts.** Each script must `source ~/.midscene-env` -- the osascript Terminal tab is a fresh shell.

## Execution pattern

```bash
RUN_ID=$(date +%s)
cat > /tmp/midscene-cmd-${RUN_ID}.sh << SCRIPT
#!/bin/bash
source ~/.midscene-env
npx -y @midscene/computer@1 take_screenshot > /tmp/midscene-out-${RUN_ID}.txt 2>&1
echo "EXIT_CODE=\$?" >> /tmp/midscene-out-${RUN_ID}.txt
touch /tmp/midscene-done-${RUN_ID}
SCRIPT
chmod +x /tmp/midscene-cmd-${RUN_ID}.sh
osascript -e "tell application \"Terminal\" to do script \"/tmp/midscene-cmd-${RUN_ID}.sh\""
while [ ! -f /tmp/midscene-done-${RUN_ID} ]; do sleep 2; done
cat /tmp/midscene-out-${RUN_ID}.txt
rm -f /tmp/midscene-cmd-${RUN_ID}.sh /tmp/midscene-out-${RUN_ID}.txt /tmp/midscene-done-${RUN_ID}
```

## Key gotchas

1. **Terminal steals focus.** `osascript -e 'tell app "Terminal" to do script ...'` brings Terminal to front. Always re-activate the target app INSIDE the script before Midscene runs: `osascript -e 'tell application "AppName" to activate' && sleep 2`.
2. **Prefer AppleScript keystrokes over Midscene for typing.** Midscene `act --prompt "Type ..."` often targets the wrong field (e.g. search bar instead of note body). Use `osascript -e 'tell application "System Events" to keystroke "text"'` after focusing the right field via Cmd+N or clicking.
3. **Midscene act with long/complex prompts can fail.** XML parse errors occur with long prompts. Keep `act --prompt` short and single-purpose. Break multi-step flows into separate calls or use AppleScript for what you can.
4. **Keyboard shortcuts > vision for standard actions.** Cmd+N (new), Cmd+S (save), Escape (dismiss), etc. are faster and more reliable than vision-based clicking.

## Fast paths (use these when possible -- no vision cost)

```bash
osascript -e 'tell application "AppName" to activate'           # Launch app
osascript -e 'the clipboard as text'                            # Read clipboard
open "x-apple.systempreferences:com.apple.preference.security"  # System Settings
open -a "TextEdit" file.txt                                     # Open in app
osascript -e 'tell application "System Events" to keystroke "n" using command down'  # New doc/note
osascript -e 'tell application "System Events" to keystroke "text here"'             # Type text
osascript -e 'tell application "System Events" to key code 53'                       # Escape key
```

## Routing

| Target | Skill |
|---|---|
| Any website / web SPA | snappy-browse |
| Native macOS app / Electron app | snappy-desktop (this skill) |
| iMessage send/list | snappy-imessage first, this as fallback |
| WhatsApp read/send | snappy-desktop `wa-chats`, `wa-read`, `wa-send` (native WhatsApp.app via vision) |
| File operations | plain shell (cp, mv, mkdir, rm) |

## Skill files

| File | Contents |
|---|---|
| SKILL.md | Full reference (principles, commands, setup, permissions, timing) |
| patterns.md | Multi-app workflows, copy+paste patterns, system operations |
| troubleshooting.md | Permission errors, hangs, locator failures |

## Consumers

snappy-imessage (Messages.app fallback), snappy-image (screen captures), snappy-video (screen recording controls), snappy-browse (CAPTCHA/native dialog fallback), snappy-maintenance (visual dev tool checks).

---

If this loader doesn't cover your case:
```bash
echo "[$(date -u +%FT%TZ)] snappy-desktop: <what was missing>" >> ~/.claude/logs/agents-md-feedback.log
```


Show produced work with `snappy-faces`: call `draw` for image channels or `lang` for MCP Apps.

<!-- SKILL-INDEX-START -->
[snappy-desktop Index]|root: ~/.claude/skills/snappy-desktop|IMPORTANT: Prefer these files over pre-training assumptions for this domain. Read the relevant file when the AGENTS.md summary is insufficient.|root:{SKILL.md,patterns.md,troubleshooting.md}
<!-- SKILL-INDEX-END -->

## Used by

- `snappy-ax`

<!-- SNAPPY-CONTRACT-VERBS-START -->
## Contract verbs

Generated from `api.ts` `HAND_CONTRACT`. Do not hand-edit this block.

| Verb | Contract arguments | Effect | First call |
|---|---|---|---|
| `run` | `instruction` | `write` | `npx tsx ~/.claude/skills/snappy-desktop/api.ts run <instruction>` |
| `screenshot` | — | `write-reversible` | `npx tsx ~/.claude/skills/snappy-desktop/api.ts screenshot` |

## Show the result

When an answer carries `face_hint`, show it with one `snappy_present(<answer>)` call.
See `/snappy-faces` for face selection. Human-facing images must crop to the
element, render at 2x on Retina, and fill the destination channel instead of
placing a small card in a full-page screenshot.
<!-- SNAPPY-CONTRACT-VERBS-END -->

---
name: snappy-desktop
instruction-only: true
reports_to: tool
head: false
description: macOS desktop automation primitive for the Snappy stack via Midscene vision AI (`npx @midscene/computer@1`). Controls real mouse and keyboard from screen vision -- works with ANY native app. NEVER run Midscene directly over SSH -- always route through local Terminal via osascript so Screen Recording permission applies. Triggers on desktop, open app, click button, screenshot desktop, launch application, switch window, desktop automation, control computer, mouse click, keyboard shortcut, Finder, System Settings, native app, Spotlight, macOS, Messages.app fallback, screen recording, vision automation, Midscene, Mac Mini desktop, AppleScript automation.
---

**Triggers on:** snappy-desktop, desktop

# Snappy Desktop

## Purpose

Vision-driven macOS desktop automation primitive. Wraps `@midscene/computer@1` to give Snappy skills "hands on the screen" for any native app -- Finder, System Settings, Messages, Mail, Notes, third-party apps. Works alongside `snappy-browse` (web) and `snappy-imessage` (iMessage CLI) -- those handle their domains; this is the fallback for everything else.

## When to Use This Skill

Activates when:
- Any task targeting a native macOS app (Finder, System Settings, Messages, Notes, Mail, Calendar, third-party apps)
- A web task that `snappy-browse` cannot handle (heavy CAPTCHA, native file dialogs, app-on-Electron quirks)
- A screenshot of the actual screen / a specific app window is needed
- A keyboard shortcut must be sent to a non-active window
- An iMessage UI action that the `imsg` CLI doesn't support (reactions, group chat navigation)
- Screen recording controls (start/stop, region select) via QuickTime / native shortcut

For websites use `snappy-browse`. For iMessage default to `snappy-imessage` first. For pure file ops use plain CLI.

---

## Core Principles

|primitive: `npx -y @midscene/computer@1` (use `-y` always)
|model: `google/gemini-2.5-flash` via OpenRouter (env `~/.midscene-env`)
|ssh-routing: REQUIRED -- every command must run from local Terminal via `osascript`, never directly over SSH
|permission: Screen Recording + Accessibility for **Terminal.app** in System Settings (resets after macOS updates)
|coordinate-mode: 1:1 (Mac Mini display 1920x1080, no Retina scaling)
|fastpath: AppleScript / `open` / CLI for anything Midscene doesn't need to *see*
|locator-style: `--locate '{"prompt": "the [specific element] in the [context]"}'` -- be specific
|race-safety: unique `RUN_ID=$(date +%s)` in script + output filenames
|cleanup: poll for `/tmp/midscene-done-${RUN_ID}` marker, never fixed `sleep`

---

## Quick Start

```bash
# 1. Generate unique ID and write a self-contained script
RUN_ID=$(date +%s)
cat > /tmp/midscene-cmd-${RUN_ID}.sh << SCRIPT
#!/bin/bash
source ~/.midscene-env
npx -y @midscene/computer@1 take_screenshot > /tmp/midscene-out-${RUN_ID}.txt 2>&1
echo "EXIT_CODE=\$?" >> /tmp/midscene-out-${RUN_ID}.txt
touch /tmp/midscene-done-${RUN_ID}
SCRIPT
chmod +x /tmp/midscene-cmd-${RUN_ID}.sh

# 2. Execute via local Terminal (Screen Recording permission lives there)
osascript -e "tell application \"Terminal\" to do script \"/tmp/midscene-cmd-${RUN_ID}.sh\""

# 3. Poll for completion (do NOT fixed-sleep)
while [ ! -f /tmp/midscene-done-${RUN_ID} ]; do sleep 2; done
cat /tmp/midscene-out-${RUN_ID}.txt

# 4. Cleanup
rm -f /tmp/midscene-cmd-${RUN_ID}.sh /tmp/midscene-out-${RUN_ID}.txt /tmp/midscene-done-${RUN_ID}
```

**Why the SSH routing pattern works:** SSH sessions cannot access Screen Recording. The `osascript` call opens a tab in the local Terminal.app which DOES have permission. The script file avoids quote-escaping hell. Unquoted heredoc (`SCRIPT` not `'SCRIPT'`) lets `$RUN_ID` expand at write time while `\$?` stays literal. Polling avoids races.

---

## What AI Agents Get Wrong

### SSH Routing & Permissions

| ❌ WRONG | ✅ CORRECT |
|----------|-----------|
| Run `npx @midscene/computer@1 take_screenshot` directly over SSH | Always wrap in a script file, then `osascript -e 'tell application "Terminal" to do script "..."'`. Screen Recording lives on Terminal.app, not the SSH process. |
| Use Charlotte MCP browser tools to "click around" the desktop | Charlotte MCP browser tools target web pages and DON'T work for the native macOS desktop. Robert's CLAUDE.md explicitly forbids them. Use `npx @midscene/computer@1` for desktop. |
| Pass complex commands inline to `osascript -e` and lose half your quotes | ALWAYS write the command to `/tmp/midscene-cmd-${RUN_ID}.sh` first, then have osascript run the script file. |
| Reuse `/tmp/midscene-cmd.sh` across calls | Use `RUN_ID=$(date +%s)` so concurrent commands don't clobber each other. |
| Fixed-sleep waiting for Midscene output | Poll for the `/tmp/midscene-done-${RUN_ID}` marker file. Midscene timing is variable (5-90s). |
| Skip `chmod +x` on the script | Always `chmod +x` before calling `osascript`. |
| Forget to `source ~/.midscene-env` inside the script | The script must `source ~/.midscene-env` itself -- the `osascript` Terminal tab is a fresh shell. |

### Credentials & Auth

| ❌ WRONG | ✅ CORRECT |
|----------|-----------|
| Hardcode passwords or API keys in the Midscene script | Load credentials from `snappy-settings/.env.cache` via `env("KEY")` (or source `load-env.sh`). Never embed a literal password in the script. |
| Type a password literally with `input --value "password123"` | Have Midscene focus the password field, then paste from clipboard set by `osascript -e 'set the clipboard to "$(env KEY_NAME)"'` (key loaded from `.env.cache`) and `keyboardpress --keyName Command+V`. Clipboard never enters the prompt or output. |
| Prompt the user for the macOS user password | If a sudo prompt appears, bail out and ask Robert directly. Never store macOS user creds in scripts. |

### Tool Selection

| ❌ WRONG | ✅ CORRECT |
|----------|-----------|
| `act --action "do thing"` | `act --prompt "do thing"` -- the flag is `--prompt`, not `--action` |
| Use Midscene to launch an app | `osascript -e 'tell application "AppName" to activate'` is instant and 100% reliable. Midscene only when you must *see* something. |
| Use Midscene to copy a file | `cp src dst` in plain shell. CLI > vision for any non-visual op. |
| Use Midscene to read clipboard | `osascript -e 'the clipboard as text'` |
| Use Midscene to open a URL | `open "https://..."` in local Terminal, or hand off to `snappy-browse`. |
| Use Midscene to open a System Settings pane | `open "x-apple.systempreferences:com.apple.preference.security"` (and similar) |
| Use Midscene for Messages.app messaging | Try `snappy-imessage` first (`imsg` CLI). Only fall back here if the CLI cannot do it (reactions, group nav). |

### Locator Quality

| ❌ WRONG | ✅ CORRECT |
|----------|-----------|
| `--locate '{"prompt": "Save"}'` (ambiguous in dialogs with multiple Saves) | `--locate '{"prompt": "the blue Save button in the export dialog"}'` |
| `--locate '{"prompt": "the button"}'` | Add color, position, parent context: `"the leftmost Delete button in the toolbar"` |
| Click without taking a screenshot first | `take_screenshot` to verify state, THEN target the element |

---

## Core Commands

See the [Cheat Sheet](#core-commands-cheat-sheet) below for the full Midscene command list. For workflow patterns (multi-app, copy/paste, system ops) see [patterns.md](patterns.md).

### Core Commands Cheat Sheet

```bash
# Vision
npx -y @midscene/computer@1 take_screenshot

# Multi-step natural language (best for Spotlight, transient menus)
npx -y @midscene/computer@1 act --prompt "Press Cmd+Space, type Safari, press Enter"

# Click / type
npx -y @midscene/computer@1 tap         --locate '{"prompt": "the Save button"}'
npx -y @midscene/computer@1 doubleclick --locate '{"prompt": "the document icon"}'
npx -y @midscene/computer@1 rightclick  --locate '{"prompt": "the file"}'
npx -y @midscene/computer@1 input       --value "Hello" --locate '{"prompt": "the search field"}'
npx -y @midscene/computer@1 input       --value "more"  --mode append --locate '{"prompt": "the text field"}'
npx -y @midscene/computer@1 clearinput  --locate '{"prompt": "the text field"}'

# Keyboard
npx -y @midscene/computer@1 keyboardpress --keyName "Enter"
npx -y @midscene/computer@1 keyboardpress --keyName "Command+Space"
npx -y @midscene/computer@1 keyboardpress --keyName "Command+Shift+3"
npx -y @midscene/computer@1 keyboardpress --keyName "Enter" --locate '{"prompt": "the search field"}'

# Scroll
npx -y @midscene/computer@1 scroll --direction down
npx -y @midscene/computer@1 scroll --scrollType scrollToBottom
npx -y @midscene/computer@1 scroll --direction down --distance 500
npx -y @midscene/computer@1 scroll --locate '{"prompt": "the list area"}' --direction down

# Mouse / drag
npx -y @midscene/computer@1 mousemove   --locate '{"prompt": "the file icon"}'
npx -y @midscene/computer@1 draganddrop --from '{"prompt": "the file"}' --to '{"prompt": "the trash icon"}'

# Display management
npx -y @midscene/computer@1 list_displays
npx -y @midscene/computer@1 connect --displayId 0
```

### AppleScript / CLI Fast Paths (use these when possible)

```bash
# App lifecycle (instant, no vision cost)
osascript -e 'tell application "AppName" to activate'
osascript -e 'tell application "AppName" to quit'

# Query frontmost app
osascript -e 'tell application "System Events" to get name of first process whose frontmost is true'

# Window control
osascript -e 'tell application "System Events" to set miniaturized of window 1 of process "AppName" to true'

# Clipboard
osascript -e 'the clipboard as text'
osascript -e 'set the clipboard to "text here"'

# File ops
cp / mv / mkdir / rm     # Plain shell
open path/to/file        # Open in OS default app
open -a "TextEdit" file  # Open in specific app

# System Settings deep links
open "x-apple.systempreferences:com.apple.preference.security"
open "x-apple.systempreferences:com.apple.preference.bluetooth"
open "x-apple.systempreferences:com.apple.preference.network"
open "x-apple.systempreferences:com.apple.preference.displays"
```

For copy/paste workflows, multi-app patterns, and complete recipes see [patterns.md](patterns.md).

---

## Setup

### Environment (one-time)

`~/.midscene-env` should contain:

```bash
export MIDSCENE_MODEL_API_KEY="<openrouter-key>"
export MIDSCENE_MODEL_NAME="google/gemini-2.5-flash"
export MIDSCENE_MODEL_BASE_URL="https://openrouter.ai/api/v1"
export MIDSCENE_MODEL_FAMILY="gemini"
```

OpenRouter key lives in `snappy-settings/.env.cache`. Populate `~/.midscene-env` once from the cache:

```bash
# Source the snappy-settings loader, then write ~/.midscene-env
SNAPPY_SETTINGS_QUIET=1 source ~/.claude/skills/snappy-settings/scripts/load-env.sh
cat > ~/.midscene-env <<EOF
export MIDSCENE_MODEL_API_KEY="$OPENROUTER_API_KEY"
export MIDSCENE_MODEL_NAME="google/gemini-2.5-flash"
export MIDSCENE_MODEL_BASE_URL="https://openrouter.ai/api/v1"
export MIDSCENE_MODEL_FAMILY="gemini"
EOF
chmod 600 ~/.midscene-env
```

### Permissions Required

| Permission | Where | Granted To |
|-----------|-------|-----------|
| Screen Recording | System Settings > Privacy & Security > Screen Recording | **Terminal.app** |
| Accessibility | System Settings > Privacy & Security > Accessibility | **Terminal.app** |

After macOS updates, permissions may reset -- re-toggle if commands start failing with "could not create image from display".

---

## Timing

| Op | Typical Duration |
|----|------------------|
| `take_screenshot` | 5-8s |
| `tap` / `keyboardpress` | 10-20s |
| `act` (multi-step) | 30-90s |
| First run after long idle (npx download) | +15s |

Use polling, never fixed `sleep`. Multi-step prompts via `act` are cheaper than multiple atomic commands.

---

## Limitations

| Limitation | Workaround |
|-----------|-----------|
| Pro apps (Xcode, Final Cut, Logic) -- dense custom UIs misclick | Prefer CLI / keyboard shortcuts; only use Midscene for top-level menus |
| Drag-and-drop unreliable on Retina/HiDPI | Prefer copy+paste; use `keyboardpress Command+C` then activate target app then `Command+V` |
| First vision step is slow (~15s) | Cache the npx package by running once at session start |
| Accumulating Terminal tabs from `osascript` | Periodically: `osascript -e 'tell application "Terminal" to close (every tab of every window whose busy is false)'` |
| OpenRouter rate limit / model down | Wait 30s, retry; check `https://openrouter.ai/activity` |

---

## Workflow

**Inputs (skills that feed this one):**
- `snappy-infra` -- provides Mac Mini SSH host (`Roberts-Mac-mini.local` / `10.0.0.199`), `~/.midscene-env` location, `snappy-settings/.env.cache` pattern for OpenRouter key

**Outputs (skills that consume this one):**
- `snappy-imessage` -- fallback when `imsg` CLI cannot do a Messages.app action (reactions, group navigation, viewing threads visually)
- `snappy-image` -- screen captures and app-window screenshots; native screenshot shortcuts
- `snappy-video` -- start/stop screen recording, region-select via `Command+Shift+5`
- `snappy-browse` -- fallback for web tasks where agent-browser CAN'T handle the page (heavy CAPTCHA, native file picker dialogs, OS-level confirmation modals)
- `snappy-ops` -- desktop checks during morning briefing if a screen-only signal is needed
- `snappy-maintenance` -- visual sanity-check of native dev tools running on the Mac Mini (Docker Desktop, Xcode build status)

**Channels (where output is delivered):**
- snappy-desktop is a *capability*, not a channel. Outputs (screenshots, extracted state) flow back to the calling skill, which then delivers via `snappy-slack` / `snappy-telegram` / `snappy-imessage` / `snappy-email` / `snappy-whatsapp`.

**Orchestrator:**
- `snappy-ops` invokes this skill on demand (rare -- most ops route through Xano APIs first); `snappy-imessage`, `snappy-image`, and `snappy-video` invoke it as a fallback when their primary tool can't do the job.

---

## Routing Rule: Browse vs Desktop vs Other

| Target | Use | Why |
|--------|-----|-----|
| Any website / web app / SPA | `snappy-browse` | agent-browser is faster, has cookie auth, no Screen Recording dance |
| Native macOS app (Finder, Messages, Notes, System Settings) | `snappy-desktop` | Midscene sees the actual screen, works with any UI |
| Electron app (VS Code, Slack desktop, Notion desktop) | `snappy-desktop` | Treated as native, not web |
| iMessage send/list | `snappy-imessage` first | `imsg` CLI is faster; fall back to desktop only if CLI lacks the feature |
| Web page agent-browser can't handle (CAPTCHA, native dialogs) | `snappy-desktop` | Open in Safari/Chrome and drive via Midscene vision |
| File operations | plain shell (`cp`, `mv`, `mkdir`, `rm`) | No vision needed |
| App launch / quit | `osascript` AppleScript | 100x faster than vision-clicking the Dock |
| System Settings pane | `open "x-apple.systempreferences:..."` | Direct deep link, no vision |
| Clipboard read/write | `osascript -e 'the clipboard as text'` / `set the clipboard to "..."` | Instant, no vision |

---

## Navigation Guide

| Need to... | Read this |
|------------|-----------|
| Reusable workflow patterns (multi-app, copy+paste, system ops) | [patterns.md](patterns.md) |
| Troubleshooting (permission errors, hangs, locator failures) | [troubleshooting.md](troubleshooting.md) |
| Canonical SSH / Mac Mini setup | [../snappy-infra/auth-reference.md](../snappy-infra/auth-reference.md) Section 5 |

---

## Quick Reference -- Important Rules

1. **SSH routing required.** Every Midscene command runs from a script file launched via `osascript -e 'tell application "Terminal" to do script "..."'`. Never invoke Midscene directly over SSH.
2. **Unique RUN_ID.** `RUN_ID=$(date +%s)` in every script + output file name. No hardcoded `/tmp/midscene-cmd.sh`.
3. **Poll for done marker.** `while [ ! -f /tmp/midscene-done-${RUN_ID} ]; do sleep 2; done`. Never fixed-sleep.
4. **Screenshot first.** `take_screenshot` before clicking blind. Decide based on what you see.
5. **`act` for multi-step.** Single `act --prompt "step1, step2, step3"` handles transient UI like Spotlight better than chained atomic calls.
6. **AppleScript for app launch.** `osascript -e 'tell application "X" to activate'` is instant. Vision is for *seeing*, not for things you can do without sight.
7. **Don't close windows unless told.** Minimize instead -- `Command+M` or AppleScript `set miniaturized of window 1 ... to true`.
8. **Credentials via `snappy-settings/.env.cache`.** Never hardcode. Never paste creds via `input --value`. Use clipboard hand-off.
9. **`--prompt`, not `--action`.** The Midscene flag is `--prompt`. Wrong flag = "No instruction was provided".
10. **Charlotte MCP browser tools are forbidden** -- they don't work for desktop OR for browsers. Use `npx @midscene/computer@1` for desktop, `agent-browser` for web.

---

## Related Skills

| Skill | Why it relates |
|-------|----------------|
| `snappy-browse` | Sister primitive -- agent-browser handles the web; desktop handles native apps and screen capture |
| `snappy-imessage` | Primary path for Messages.app sends; desktop is the UI fallback for reactions/group nav |
| `snappy-image` | Delegates here for full-screen and app-window screenshots |
| `snappy-video` | Delegates here for screen recording controls (`Command+Shift+5`) |
| `snappy-infra` | Mac Mini SSH host, Tailscale networking, `.env.cache` credential source, OpenRouter key |
| `snappy-ops` | Orchestrator -- invokes desktop only when no API or browser path exists |
| `snappy-maintenance` | Visual sanity-check of Mac Mini-resident dev tools |

---

**Skill Status**: COMPLETE

<!-- SNAPPY-NEAR-NEIGHBOURS-START -->
## Near neighbours

A model confuses this hand with `snappy-agent-host`, `snappy-ax`, `snappy-browse`, `snappy-cleanshot`, `snappy-dashboard`, `snappy-docs`, `snappy-dom-cartographer`, `snappy-ffmpeg`, `snappy-gmail`, `snappy-image`, `snappy-imessage`, `snappy-infra`, `snappy-jcode`, `snappy-nightshift`, `snappy-resident`, `snappy-swarm`, `snappy-video`, `snappy-voice-control`. Open one of those when its job is the job.
<!-- SNAPPY-NEAR-NEIGHBOURS-END -->

#!/usr/bin/env npx tsx
/**
 * snappy-desktop/api.ts -- macOS desktop automation via Midscene vision AI.
 *
 * Wraps npx @midscene/computer@1 for native app control.
 * Per AGENTS.md: commands run via osascript Terminal routing for Screen Recording permission.
 *
 * Usage:
 *   npx tsx api.ts run "click the Settings icon"
 *   npx tsx api.ts screenshot
 *
 * Or import as module:
 *   import { runMidscene, screenshot } from "../snappy-desktop/api.ts";
 */

import { execSync } from "child_process";
import { existsSync, readFileSync, realpathSync, unlinkSync } from "fs";
import { env } from "../snappy-settings/load.ts";
import { refusalTable } from "../snappy-settings/refusal-codes.ts";

/** Run a Midscene vision instruction via Terminal routing (required for Screen Recording). */
export function runMidscene(instruction: string): string {
  const runId = Date.now().toString();
  const scriptPath = `/tmp/midscene-cmd-${runId}.sh`;
  const outPath = `/tmp/midscene-out-${runId}.txt`;
  const donePath = `/tmp/midscene-done-${runId}`;

  const escaped = instruction.replace(/"/g, '\\"');
  const script = `#!/bin/bash
source ~/.midscene-env
npx -y @midscene/computer@1 act --prompt "${escaped}" > ${outPath} 2>&1
echo "EXIT_CODE=$?" >> ${outPath}
touch ${donePath}
`;

  execSync(`cat > ${scriptPath} << 'SCRIPT_EOF'\n${script}SCRIPT_EOF`, { encoding: "utf-8" });
  execSync(`chmod +x ${scriptPath}`);
  execSync(`osascript -e 'tell application "Terminal" to do script "${scriptPath}"'`);

  // Poll for completion
  const maxWait = 60_000;
  const start = Date.now();
  while (!existsSync(donePath) && Date.now() - start < maxWait) {
    execSync("sleep 2");
  }

  let output = "";
  if (existsSync(outPath)) {
    output = readFileSync(outPath, "utf-8");
  } else {
    output = "ERROR: Midscene timed out after 60s";
  }

  // Cleanup
  for (const f of [scriptPath, outPath, donePath]) {
    if (existsSync(f)) unlinkSync(f);
  }

  return output.trim();
}

/** Take a screenshot and return the file path. */
export function screenshot(): string {
  const path = `/tmp/screenshot-${Date.now()}.png`;
  execSync(`screencapture -x ${path}`, { timeout: 10_000 });
  return path;
}

// --- CLI ---
/** WHAT THIS HAND ANSWERS, and what each verb does to the world.
 *  Derived from this file's own CLI dispatch by
 *  `snappy-hands/contract-derive.ts` — a verb the code does not implement is
 *  never declared here. Snappy's daemon reads it (`api.ts contract`) to
 *  validate every call, build the argument words in order, decide whether the
 *  act runs now or stages for the owner, and hand the child exactly the
 *  environment keys named in `requires` — never a value, never anything else.
 */
export const HAND_CONTRACT = {
  skill: "snappy-desktop",
  description: "macOS desktop automation primitive for the Snappy stack via Midscene vision AI (`npx @midscene/computer@1`). Controls real mouse and keyboard from screen vision -- works with ANY native app. NEVER run Midscene directly over SSH -- always route through local Terminal via osascript so Screen Recording permission applies. Triggers on desktop, open app, click button, screenshot desktop, launch application, switch window, desktop automation, control computer, mouse click, keyboard shortcut, Finder, System Settings, native app, Spotlight, macOS, Messages.app fallback, screen recording, vision automation, Midscene, Mac Mini desktop, AppleScript automation.",
  managed: false,
  requires: [] as string[],
  refusals: refusalTable("unknown_verb", "missing_argument", "unsupported_platform"),
  verbs: {
    run: {
      args: ["instruction"], effect: "write",
      class: "destructive", execution: "call", openWorld: true,
      annotations: { readOnlyHint: false, destructiveHint: true, idempotentHint: false, openWorldHint: true },
      inputSchema: { properties: { instruction: { type: "string", description: "Plain-language instruction the desktop agent performs on this Mac" } } },
    },
    screenshot: {
      args: [], effect: "write-reversible",
      class: "additive-write", execution: "call", openWorld: true,
      annotations: { readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: true },
    },
  },
} as const;

if (import.meta.url === `file://${realpathSync(process.argv[1])}` && process.argv[2] === "contract") {
  console.log(JSON.stringify(HAND_CONTRACT, null, 2));
  process.exit(0);
}


if (import.meta.url === `file://${realpathSync(process.argv[1])}`) {
  (async () => {
    const [, , cmd, ...args] = process.argv;

    switch (cmd) {
      case "run": {
        const instruction = args.join(" ");
        if (!instruction) { console.error("Usage: api.ts run <instruction>"); process.exit(1); }
        console.log(runMidscene(instruction));
        break;
      }
      case "screenshot": {
        const path = screenshot();
        console.log(path);
        break;
      }
      default:
        console.log("Usage: npx tsx api.ts [run|screenshot] ...");
    }
  })();
}

import { test } from "node:test";
import assert from "node:assert/strict";
import { HAND_CONTRACT } from "./api.ts";
import { REFUSAL_CODES, type RefusalCode } from "../snappy-settings/refusal-codes.ts";

/** RULE 33 COVERAGE, AGAINST THE ONE CLOSED TABLE ⟨lane r30, 2026-09-09⟩.
 *  This file used to restate a refusal row's own properties — that it has a
 *  `contract_slice`, that it has a `fix`, that it leaks no token — once per
 *  hand, 48 times, over rows that all come from the SAME object. Forty-eight
 *  copies of one check is the duplicate road the closed table exists to end:
 *  `snappy-settings/refusal-codes.test.ts` runs those checks ONCE over every
 *  row, and the second test below proves this hand carries THE ROW ITSELF and
 *  not a copy — an identity a drifted duplicate cannot fake. Before this, each
 *  hand hand-wrote its own row, and the wording had already drifted: the
 *  inline `unknown_verb` said "Call one of the verbs named in
 *  HAND_CONTRACT.verbs" while the closed table says "Call one of the verbs the
 *  contract declares; the refusal lists them."
 *
 *  DECLARED stays a literal list, deliberately. It is this hand's coverage
 *  manifest and it is what rule 33's lint reads out of the test SOURCE to ask
 *  "did a person look at this code" — deriving it from Object.keys would make
 *  the test pass for a hand with no refusals at all. `satisfies readonly
 *  RefusalCode[]` makes the compiler refuse a name the one table does not
 *  have. It NAMES codes; it no longer DEFINES them. */
const DECLARED = [
  "unknown_verb",
  "missing_argument",
  "unsupported_platform",
] as const satisfies readonly RefusalCode[];

test("snappy-desktop: the refusal table declares exactly the codes this test names", () => {
  assert.deepEqual(Object.keys(HAND_CONTRACT.refusals ?? {}).sort(), [...DECLARED].sort());
});

test("snappy-desktop: every declared refusal is the one closed table's own row, never a copy", () => {
  const declared = Object.entries(HAND_CONTRACT.refusals ?? {});
  assert.ok(declared.length > 0, "HAND_CONTRACT.refusals is empty");
  for (const [code, row] of declared) {
    assert.ok(code in REFUSAL_CODES, `${code} is not a row of snappy-settings/refusal-codes.ts`);
    assert.equal(row, REFUSAL_CODES[code as RefusalCode], `${code} is a copy of the closed table's row, not the row itself`);
  }
});

# Desktop Patterns

## Table of Contents

- [App Launch Patterns](#app-launch-patterns)
- [File Operations](#file-operations)
- [System Settings](#system-settings)
- [Window Management](#window-management)
- [Text Editing](#text-editing)
- [Multi-App Workflows](#multi-app-workflows)
- [System Operations](#system-operations)
- [Common AppleScript Commands](#common-applescript-commands)
- [Cross-Skill Delegation Patterns](#cross-skill-delegation-patterns)
- [When NOT to Use Midscene](#when-not-to-use-midscene)

---

## App Launch Patterns

### Open app via AppleScript (preferred -- instant)
```bash
osascript -e 'tell application "AppName" to activate'
```

### Open app via Spotlight (Midscene fallback)
```bash
npx -y @midscene/computer@1 act --prompt "Press Cmd+Space, type AppName, press Enter"
```

### Open specific file
```bash
open /path/to/file.txt                     # OS default app
open -a "TextEdit" /path/to/file.txt       # Specific app
```

## File Operations

### Create folder in Finder
```bash
npx -y @midscene/computer@1 act --prompt "Open Finder, go to Documents, press Cmd+Shift+N, type FolderName, press Enter"
```

### Bulk file ops -- use CLI, not Midscene
```bash
mkdir -p ~/Documents/Projects
cp ~/Downloads/file.pdf ~/Documents/
```

## System Settings

### Open specific pane (fast, no vision)
```bash
open "x-apple.systempreferences:com.apple.preference.security"
open "x-apple.systempreferences:com.apple.preference.bluetooth"
open "x-apple.systempreferences:com.apple.preference.network"
open "x-apple.systempreferences:com.apple.preference.displays"
```

### Toggle settings via Midscene
```bash
npx -y @midscene/computer@1 act --prompt "In System Settings, click Displays, change resolution"
```

## Window Management

### Minimize / Close / Full screen
```bash
npx -y @midscene/computer@1 keyboardpress --keyName "Command+M"         # Minimize
npx -y @midscene/computer@1 keyboardpress --keyName "Command+W"         # Close window
npx -y @midscene/computer@1 keyboardpress --keyName "Control+Command+F" # Full screen
```

### Switch between apps
```bash
npx -y @midscene/computer@1 keyboardpress --keyName "Command+Tab"
```

## Text Editing

### Select all and replace
```bash
npx -y @midscene/computer@1 keyboardpress --keyName "Command+A"
npx -y @midscene/computer@1 input --value "new content" --mode replace
```

### Copy/Paste
```bash
npx -y @midscene/computer@1 keyboardpress --keyName "Command+C"
npx -y @midscene/computer@1 keyboardpress --keyName "Command+V"
```

### Undo/Redo
```bash
npx -y @midscene/computer@1 keyboardpress --keyName "Command+Z"
npx -y @midscene/computer@1 keyboardpress --keyName "Command+Shift+Z"
```

## Multi-App Workflows

### Copy from Safari → Paste into Notes
```bash
osascript -e 'tell application "Safari" to activate'
sleep 1
npx -y @midscene/computer@1 keyboardpress --keyName "Command+A"
npx -y @midscene/computer@1 keyboardpress --keyName "Command+C"
osascript -e 'tell application "Notes" to activate'
sleep 1
npx -y @midscene/computer@1 tap --locate '{"prompt": "the note text area"}'
npx -y @midscene/computer@1 keyboardpress --keyName "Command+V"
```

### Search in one app, paste into another
```bash
osascript -e 'tell application "Mail" to activate'
sleep 1
npx -y @midscene/computer@1 keyboardpress --keyName "Command+F"
npx -y @midscene/computer@1 input --value "search term" --locate '{"prompt": "the search field"}'
npx -y @midscene/computer@1 keyboardpress --keyName "Enter"
sleep 2
npx -y @midscene/computer@1 keyboardpress --keyName "Command+C"
osascript -e 'tell application "Notes" to activate'
sleep 1
npx -y @midscene/computer@1 keyboardpress --keyName "Command+V"
```

## System Operations

### Volume
```bash
npx -y @midscene/computer@1 keyboardpress --keyName "VolumeUp"
npx -y @midscene/computer@1 keyboardpress --keyName "VolumeDown"
npx -y @midscene/computer@1 keyboardpress --keyName "VolumeMute"
```

### Native screenshots
```bash
npx -y @midscene/computer@1 keyboardpress --keyName "Command+Shift+3"  # Full screen
npx -y @midscene/computer@1 keyboardpress --keyName "Command+Shift+4"  # Selection
npx -y @midscene/computer@1 keyboardpress --keyName "Command+Shift+5"  # Screen record
```

## Common AppleScript Commands

Use these instead of Midscene when possible -- they're instant.

```bash
# App lifecycle
osascript -e 'tell application "AppName" to activate'
osascript -e 'tell application "AppName" to quit'

# Query state
osascript -e 'tell application "System Events" to get name of first process whose frontmost is true'
osascript -e 'tell application "System Events" to get name of every window of process "Finder"'

# Window control
osascript -e 'tell application "System Events" to set miniaturized of window 1 of process "AppName" to true'
osascript -e 'tell application "System Events" to set miniaturized of window 1 of process "AppName" to false'

# Clipboard
osascript -e 'the clipboard as text'
osascript -e 'set the clipboard to "text here"'
```

## Cross-Skill Delegation Patterns

snappy-desktop is the low-level "hands on the screen" capability. Other skills delegate here when their primary tool can't cover a use case.

### snappy-imessage → snappy-desktop (Messages.app fallback)

When `imsg` CLI fails or a Messages.app UI action is needed (reactions, group chat navigation, viewing threads visually):

```bash
# 1. Open Messages.app
osascript -e 'tell application "Messages" to activate'
sleep 1

# 2. Navigate to conversation
npx -y @midscene/computer@1 tap   --locate '{"prompt": "the search field in Messages"}'
npx -y @midscene/computer@1 input --value "Contact Name" --locate '{"prompt": "the search field"}'
npx -y @midscene/computer@1 tap   --locate '{"prompt": "the first conversation result"}'

# 3. Type and send
npx -y @midscene/computer@1 tap   --locate '{"prompt": "the message input field"}'
npx -y @midscene/computer@1 input --value "Your message" --locate '{"prompt": "the message input field"}'
npx -y @midscene/computer@1 keyboardpress --keyName "Enter"
```

### snappy-image → snappy-desktop (screenshots)

When snappy-image needs a screenshot of the current screen or a specific app window:

```bash
# Full screen capture (saved to temp path printed in output)
npx -y @midscene/computer@1 take_screenshot

# Native macOS screenshot to file (Desktop) -- no clipboard
npx -y @midscene/computer@1 keyboardpress --keyName "Command+Shift+3"

# Specific window: activate app first, then screenshot
osascript -e 'tell application "AppName" to activate'
sleep 1
npx -y @midscene/computer@1 take_screenshot
```

### snappy-video → snappy-desktop (screen recording)

When snappy-video needs to record the screen (demo, walkthrough, tutorial):

```bash
# Open native screen recorder
npx -y @midscene/computer@1 keyboardpress --keyName "Command+Shift+5"
sleep 1
npx -y @midscene/computer@1 tap --locate '{"prompt": "the Record Entire Screen button"}'
npx -y @midscene/computer@1 tap --locate '{"prompt": "the Record button"}'

# ... perform the demo actions ...

# Stop recording (menu bar shortcut)
npx -y @midscene/computer@1 keyboardpress --keyName "Command+Control+Escape"
```

### snappy-browse → snappy-desktop (web fallback)

When agent-browser cannot handle a page (heavy CAPTCHA, native file picker dialog, OS-level confirmation modal):

```bash
# Open the URL in Safari (native), then drive via vision
osascript -e 'tell application "Safari" to activate'
sleep 1
npx -y @midscene/computer@1 tap   --locate '{"prompt": "the address bar"}'
npx -y @midscene/computer@1 input --value "https://example.com" --locate '{"prompt": "the address bar"}'
npx -y @midscene/computer@1 keyboardpress --keyName "Enter"
sleep 3
npx -y @midscene/computer@1 take_screenshot   # Inspect what's on screen
```

---

## When NOT to Use Midscene

| Task | Better tool |
|------|-------------|
| Open/close/switch apps | AppleScript (`osascript -e 'tell application "X" to activate'`) |
| File operations | Plain shell (`cp`, `mv`, `mkdir`, `rm`) |
| Open URLs | `open https://url` or hand off to `snappy-browse` |
| System settings pane | `open "x-apple.systempreferences:..."` deep link |
| Read clipboard | `osascript -e 'the clipboard as text'` |
| Write clipboard | `osascript -e 'set the clipboard to "text"'` |
| Kill processes | `pkill -f "ProcessName"` |
| Send iMessage | `snappy-imessage` (`imsg` CLI) |
| Web automation | `snappy-browse` (agent-browser CLI) |

Use Midscene only when you need to **see and interact with visual UI elements** -- buttons, forms, dialogs, menus -- that have no CLI or AppleScript equivalent.

# Troubleshooting

## "could not create image from display"
**Cause:** Screen Recording permission not granted, or running over SSH without osascript routing.
**Fix:**
1. System Settings > Privacy & Security > Screen Recording > Toggle ON for Terminal
2. Route all commands through `osascript -e 'tell application "Terminal" to do script "..."'`
3. After macOS updates, permissions may reset -- re-check.

## "Unable to connect to computer device"
Same as above -- screencapture can't access the display.

## Command hangs / takes too long
Midscene uses AI inference on every step. Normal times:
- Screenshot: 5-8s
- Single action (tap/press): 10-20s
- Multi-step act: 30-90s
- First run (npx download): add ~15s

If stuck beyond 2 minutes:
```bash
pkill -f "midscene/computer"
pkill -f "Chrome"
```

## "No instruction was provided"
Wrong flag name. Use `--prompt` not `--action`:
```bash
# Wrong
npx -y @midscene/computer@1 act --action "do thing"
# Right
npx -y @midscene/computer@1 act --prompt "do thing"
```

## Midscene clicks wrong element
1. Take screenshot to verify current state
2. Make prompt more specific: `"the Save button in the dialog"` not just `"Save"`
3. Add context: `"In the file export dialog, click the blue Save button"`
4. If UI has duplicates, use position: `"the leftmost Delete button"`

## Permission dialog blocks action
macOS shows "Allow"/"Deny" dialog unexpectedly:
```bash
# Screenshot to confirm
npx -y @midscene/computer@1 take_screenshot
# Click Allow
npx -y @midscene/computer@1 tap --locate '{"prompt": "the Allow button"}'
```

## App becomes unresponsive
```bash
pkill -f "midscene/computer"      # Kill Midscene
pkill -f "AppName"                 # Force quit the app
osascript -e 'tell application "AppName" to activate'  # Restart
```

## Accessibility permission errors
```bash
osascript -e 'tell application "System Events" to get name of first process'
```
If it fails, grant Terminal in System Settings > Privacy & Security > Accessibility.

## Window blocking the action
```bash
# Close blocking app via AppleScript
osascript -e 'tell application "AppName" to quit'
# Or minimize
osascript -e 'tell application "System Events" to set miniaturized of window 1 of process "AppName" to true'
```

## npm/npx issues
Always use `-y` flag:
```bash
npx -y @midscene/computer@1 take_screenshot
```

If npx cache is broken:
```bash
rm -rf ~/.npm/_npx
npx -y @midscene/computer@1 take_screenshot
```

## Quote escaping breaks with osascript inline
Never pass complex commands inline. Always write to a script file:
```bash
cat > /tmp/midscene-cmd.sh << 'SCRIPT'
#!/bin/bash
source ~/.midscene-env
npx -y @midscene/computer@1 act --prompt "your complex command here"
SCRIPT
chmod +x /tmp/midscene-cmd.sh
osascript -e 'tell application "Terminal" to do script "/tmp/midscene-cmd.sh"'
```

## Terminal tab accumulation
Each osascript opens a new Terminal tab. To clean up:
```bash
osascript -e 'tell application "Terminal" to close (every tab of every window whose busy is false)'
```

## OpenRouter API issues
- Rate limiting: wait 30s, retry
- Model unavailable: check https://openrouter.ai/activity
- Key expired: update `~/.midscene-env`

## Multiple displays
If an external monitor is connected/disconnected:
```bash
npx -y @midscene/computer@1 list_displays
npx -y @midscene/computer@1 connect --displayId 0   # Switch to specific display
```

## Concurrent command race condition
Always use unique file names:
```bash
RUN_ID=$(date +%s)
cat > /tmp/midscene-cmd-${RUN_ID}.sh << 'SCRIPT'
...
SCRIPT
```
Never use hardcoded `/tmp/midscene-cmd.sh` if running multiple commands.