@otto-assistant/bridge 0.4.92

package/skills/tuistory/SKILL.md

@@ -0,0 +1,250 @@
+---
+name: tuistory
+description: |
+  Playwright for terminal apps. Like tmux but designed for agents — virtual terminals you can
+  type into, press keys, wait for text, take snapshots, and screenshot as images.
+
+  tuistory has **2 modes**:
+
+  - **CLI** (`tuistory` command) — shell-based. Launch sessions, type, press keys, snapshot, screenshot.
+    Runs a background daemon that persists sessions across commands. Install globally or use `npx`/`bunx`.
+    **You MUST run `tuistory --help` before using the CLI** to see the latest commands and options.
+  - **JS/TS API** (`import { launchTerminal } from 'tuistory'`) — programmatic. Use in vitest/bun:test
+    to write Playwright-style tests for CLIs and TUIs with inline snapshots.
+
+  Use tuistory when you need to:
+  - Write e2e tests for CLI/TUI apps (vitest, bun:test) with inline snapshots
+  - Automate terminal interactions (launch a REPL, debugger, or TUI and drive it)
+  - Screenshot terminal as images to send to users (Discord bots, agent UIs like kimaki/openclaw)
+  - Reproduce bugs in interactive CLIs by scripting the exact steps
+  - Explore TUI apps progressively with observe-act-observe loops
+---
+
+# tuistory
+
+Playwright for terminal user interfaces. Write end-to-end tests for CLI and TUI applications.
+
+tuistory has **2 modes**:
+
+1. **CLI** — the `tuistory` shell command. Launch terminal sessions, type text, press keys, take text snapshots or image screenshots. Sessions run in a background daemon and persist across commands.
+2. **JS/TS API** — `import { launchTerminal } from 'tuistory'`. Use programmatically in test files (vitest, bun:test) to write Playwright-style tests with inline snapshots.
+
+## CLI usage
+
+**REQUIREMENT:** You MUST run `tuistory --help` before using the CLI. The CLI evolves and the help output is the source of truth for available commands, options, and syntax. Always check it first.
+
+Install globally or use npx/bunx:
+
+```bash
+# global
+bun add -g tuistory
+npm install -g tuistory
+
+# or without installing
+npx tuistory --help
+bunx tuistory --help
+```
+
+### CLI quick reference
+
+```bash
+tuistory launch <command> -s <name> [--cols N] [--rows N] [--env KEY=VAL]
+tuistory -s <name> snapshot --trim
+tuistory -s <name> screenshot -o image.jpg --pixel-ratio 2
+tuistory -s <name> type "text"
+tuistory -s <name> press enter
+tuistory -s <name> press ctrl c
+tuistory -s <name> click "Submit"
+tuistory -s <name> wait "pattern" --timeout 10000
+tuistory -s <name> wait "/regex/" --timeout 10000
+tuistory -s <name> scroll down 5
+tuistory -s <name> resize 120 40
+tuistory -s <name> close
+tuistory sessions
+tuistory daemon-stop
+```
+
+Always run `snapshot --trim` after every action to see the current terminal state.
+
+### Screenshot for agent bots
+
+Capture terminal as an image to upload to users (Discord, Slack, web UIs).
+Use `--pixel-ratio 2` for sharp images on social media and messaging apps:
+
+```bash
+tuistory -s myapp screenshot -o /tmp/terminal.jpg --pixel-ratio 2
+# then upload /tmp/terminal.jpg to the user
+```
+
+## JS/TS API (library)
+
+```bash
+bun add tuistory    # or npm install tuistory
+```
+
+```ts
+import { launchTerminal } from 'tuistory'
+
+const session = await launchTerminal({
+  command: 'my-cli',
+  args: ['--flag'],
+  cols: 120,
+  rows: 36,
+  cwd: '/path/to/dir',
+  env: { MY_VAR: 'value' },
+})
+
+// observe
+const text = await session.text()               // full terminal text
+const text = await session.text({ trimEnd: true }) // trimmed
+const bold = await session.text({ only: { bold: true } }) // style filter
+
+// act
+await session.type('hello world')               // type character by character
+await session.press('enter')                    // single key
+await session.press(['ctrl', 'c'])              // key chord
+await session.click('Submit')                   // click on text
+
+// wait
+await session.waitForText('Ready', { timeout: 10000 })
+await session.waitForText(/Loading\.\.\./, { timeout: 5000 })
+
+// screenshot to image
+const data = session.getTerminalData()
+const { renderTerminalToImage } = await import('ghostty-opentui/image')
+const image = await renderTerminalToImage(data, { format: 'jpeg', devicePixelRatio: 2 })
+
+// cleanup
+session.close()
+```
+
+## Writing tests with vitest
+
+tuistory is like Playwright but for CLIs. The workflow is: **observe** (snapshot with inline snapshot), **act** (type/press/click), **observe** again. Build tests progressively.
+
+### Step 1: Launch and observe
+
+Start with an empty inline snapshot. Run with `--update` / `-u` to fill it in.
+
+```ts
+import { test, expect } from 'vitest'
+import { launchTerminal } from 'tuistory'
+
+test('my CLI shows help', async () => {
+  const session = await launchTerminal({
+    command: 'my-cli',
+    args: ['--help'],
+    cols: 120,
+    rows: 36,
+  })
+
+  const text = await session.text({ trimEnd: true })
+  expect(text).toMatchInlineSnapshot()
+  // ^ run `vitest --run -u` to fill this in, then read the file to see what it captured
+
+  session.close()
+}, 10000)
+```
+
+### Step 2: Interact and observe again
+
+Add actions and more snapshots incrementally:
+
+```ts
+test('bash interaction', async () => {
+  const session = await launchTerminal({
+    command: 'bash',
+    args: ['--norc', '--noprofile'],
+    cols: 60,
+    rows: 10,
+    env: { PS1: '$ ', HOME: '/tmp', PATH: process.env.PATH },
+  })
+
+  // observe initial state
+  const initial = await session.text({ trimEnd: true })
+  expect(initial).toMatchInlineSnapshot()
+
+  // act
+  await session.type('echo "hello world"')
+  await session.press('enter')
+
+  // wait + observe
+  const output = await session.waitForText('hello world')
+  expect(output).toMatchInlineSnapshot()
+
+  // cleanup
+  await session.type('exit')
+  await session.press('enter')
+  session.close()
+}, 10000)
+```
+
+### Step 3: Run with -u, read back, iterate
+
+```bash
+# fill in snapshots
+vitest --run -u
+
+# read the test file to see captured terminal output
+# adjust assertions, add more interactions, repeat
+```
+
+This observe-act-observe loop lets you progressively explore any TUI. Each inline snapshot captures the exact terminal state, making tests readable and easy to update.
+
+### Testing a TUI app (e.g. opencode, claude)
+
+```ts
+test('opencode shows welcome', async () => {
+  const session = await launchTerminal({
+    command: 'opencode',
+    cols: 150,
+    rows: 45,
+  })
+
+  await session.waitForText('switch agent', { timeout: 15000 })
+  await session.type('hello from tuistory')
+
+  const text = await session.text({ timeout: 1000 })
+  expect(text).toMatchInlineSnapshot()
+
+  // navigate menus
+  await session.press(['ctrl', 'p'])
+  const commands = await session.waitForText('Commands', { timeout: 5000 })
+  expect(commands).toMatchInlineSnapshot()
+
+  await session.press('esc')
+  session.close()
+}, 30000)
+```
+
+### Testing a Node.js debugger
+
+```ts
+test('node debugger inspect variables', async () => {
+  const session = await launchTerminal({
+    command: 'node',
+    args: ['inspect', 'app.js'],
+    cols: 150,
+    rows: 45,
+  })
+
+  await session.waitForText('Break on start', { timeout: 10000 })
+  await session.type('cont')
+  await session.press('enter')
+  await session.waitForText('break in', { timeout: 5000 })
+
+  const snapshot = await session.text({ trimEnd: true })
+  expect(snapshot).toMatchInlineSnapshot()
+
+  session.close()
+}, 30000)
+```
+
+## Tips
+
+- **Always set a timeout** on `waitForText` for async operations
+- **Use `trimEnd: true`** in `session.text()` to avoid trailing whitespace in snapshots
+- **Set `waitForData: false`** for interactive commands that don't produce output immediately (like `cat`)
+- **Use regex** in `waitForText` for dynamic content: `await session.waitForText(/version \d+/)`
+- **Close sessions** in test teardown to avoid leaked processes
+- **Use `--cols` and `--rows`** to control terminal size — affects TUI layout

package/skills/usecomputer/SKILL.md

@@ -0,0 +1,264 @@
+---
+name: usecomputer
+description: >
+  Desktop automation CLI for AI agents (macOS, Linux, Windows). Screenshot,
+  click, type, scroll, drag with native Zig backend. Use this skill when
+  automating desktop apps with computer use models (GPT-5.4, Claude). Covers
+  the screenshot-action feedback loop, coord-map workflow, window-scoped
+  screenshots, and system prompts for accurate clicking.
+---
+
+# usecomputer
+
+Desktop automation CLI for AI agents. Works on macOS, Linux (X11), and
+Windows. Takes screenshots, clicks, types, scrolls, drags using native
+platform APIs through a Zig binary — no Node.js required at runtime.
+
+## Always start with --help
+
+**Always run `usecomputer --help` before using this tool.** The help output
+is the source of truth for all commands, options, and examples. Never guess
+command syntax — check help first.
+
+When running help commands, read the **full untruncated output**. Never pipe
+help through `head`, `tail`, or `sed` — you will miss critical options.
+
+```bash
+usecomputer --help
+usecomputer screenshot --help
+usecomputer click --help
+usecomputer drag --help
+```
+
+## Install
+
+```bash
+npm install -g usecomputer
+```
+
+Requirements:
+
+- **macOS** — Accessibility permission enabled for your terminal app
+- **Linux** — X11 session with `DISPLAY` set (Wayland via XWayland works too)
+- **Windows** — run in an interactive desktop session
+
+## Core loop: screenshot -> act -> screenshot
+
+Every computer use session follows a feedback loop:
+
+```
+screenshot -> send to model -> model returns action -> execute action -> screenshot again
+     ^                                                                        |
+     |________________________________________________________________________|
+```
+
+1. Take a screenshot with `usecomputer screenshot --json`
+2. Send the screenshot image to the model
+3. Model returns coordinates or an action (click, type, press, scroll)
+4. Execute the action, passing the **exact `--coord-map`** from step 1
+5. Take a fresh screenshot and go back to step 2
+
+### Full cycle example
+
+```bash
+# 1. take screenshot (always use --json to get coordMap)
+usecomputer screenshot ./tmp/screen.png --json
+# output: {"path":"./tmp/screen.png","coordMap":"0,0,3440,1440,1568,657",...}
+
+# 2. send ./tmp/screen.png to the model
+# 3. model says: "click the Save button at x=740 y=320"
+
+# 4. click using the coord-map from the screenshot output
+usecomputer click -x 740 -y 320 --coord-map "0,0,3440,1440,1568,657"
+
+# 5. take a fresh screenshot to see what happened
+usecomputer screenshot ./tmp/screen.png --json
+# ... repeat
+```
+
+**Never skip `--coord-map`.** Screenshots are scaled (longest edge <= 1568px).
+The coord-map maps screenshot-space pixels back to real desktop coordinates.
+Without it, clicks land in wrong positions.
+
+**Always take a fresh screenshot after each action.** The UI changes after
+every click, scroll, or keystroke — menus open, pages scroll, dialogs appear.
+Never reuse a stale screenshot.
+
+## Window-scoped screenshots
+
+Full-desktop screenshots include everything — dock, menu bar, background
+windows. For better accuracy, capture only the target application window.
+This produces a smaller, more focused image the model can reason about.
+
+### Step 1: find the window ID
+
+```bash
+usecomputer window list --json
+```
+
+This returns an array of visible windows with their `id`, `ownerName`,
+`title`, position, and size. Find the window you want to target.
+
+### Step 2: screenshot that window
+
+```bash
+usecomputer screenshot ./tmp/app.png --window 12345 --json
+# output: {"path":"./tmp/app.png","coordMap":"200,100,1200,800,1568,1045",...}
+```
+
+The coord-map in the output is scoped to that window's region on screen.
+
+### Step 3: act using the coord-map
+
+```bash
+# model analyzes ./tmp/app.png and says click at x=400 y=220
+usecomputer click -x 400 -y 220 --coord-map "200,100,1200,800,1568,1045"
+```
+
+The coord-map handles the translation from the window screenshot's pixel
+space back to the correct desktop coordinates. The click lands on the
+right spot even though the screenshot only showed one window.
+
+### Region screenshots
+
+You can also capture an arbitrary rectangle of the screen:
+
+```bash
+usecomputer screenshot ./tmp/region.png --region "100,100,800,600" --json
+```
+
+The coord-map works the same way — pass it to subsequent pointer commands.
+
+## Coord-map explained
+
+The coord-map is 6 comma-separated values emitted by every screenshot:
+
+```
+captureX,captureY,captureWidth,captureHeight,imageWidth,imageHeight
+```
+
+- **captureX, captureY** — top-left corner of the captured region in desktop
+  coordinates
+- **captureWidth, captureHeight** — size of the captured region in desktop
+  pixels
+- **imageWidth, imageHeight** — size of the output PNG (after scaling)
+
+When you pass `--coord-map` to `click`, `hover`, `drag`, or `mouse move`,
+the command maps your screenshot-space x,y coordinates back to the real
+desktop position using these values.
+
+## Validating coordinates with debug-point
+
+Before clicking, you can validate where the click would land:
+
+```bash
+usecomputer debug-point -x 400 -y 220 --coord-map "0,0,1600,900,1568,882"
+```
+
+This captures a screenshot and draws a red marker at the mapped coordinate.
+Send the output image back to the model so it can see if the target is
+correct and adjust if needed.
+
+## Quick examples
+
+```bash
+# screenshot the primary display
+usecomputer screenshot ./tmp/screen.png --json
+
+# screenshot a specific display (0-indexed)
+usecomputer screenshot ./tmp/screen.png --display 1 --json
+
+# click at screenshot coordinates
+usecomputer click -x 600 -y 400 --coord-map "0,0,1600,900,1568,882"
+
+# right-click
+usecomputer click -x 600 -y 400 --button right --coord-map "..."
+
+# double-click
+usecomputer click -x 600 -y 400 --count 2 --coord-map "..."
+
+# click with modifier keys held
+usecomputer click -x 600 -y 400 --modifier option --coord-map "..."
+usecomputer click -x 600 -y 400 --modifier cmd --modifier shift --coord-map "..."
+
+# type text
+usecomputer type "hello from usecomputer"
+
+# type long text from stdin
+cat ./notes.txt | usecomputer type --stdin --chunk-size 4000 --chunk-delay 15
+
+# press a key
+usecomputer press "enter"
+
+# press a shortcut
+usecomputer press "cmd+s"
+usecomputer press "cmd+shift+p"
+
+# press with repeat
+usecomputer press "down" --count 10 --delay 30
+
+# scroll
+usecomputer scroll down 5
+usecomputer scroll up 3
+usecomputer scroll down 5 --at "400,300"
+
+# drag (straight line)
+usecomputer drag 100,200 500,600
+
+# drag (curved path with bezier control point)
+usecomputer drag 100,200 500,600 300,50
+
+# drag with coord-map
+usecomputer drag 100,200 500,600 --coord-map "..."
+
+# mouse position
+usecomputer mouse position --json
+
+# list displays
+usecomputer display list --json
+
+# list windows
+usecomputer window list --json
+
+# list desktops with windows
+usecomputer desktop list --windows --json
+```
+
+## System prompt tips for accurate clicking
+
+When using GPT-5.4 or Claude for computer use, keep the system prompt short
+and task-focused. Verbose system prompts reduce click accuracy.
+
+**GPT-5.4:** Use `detail: "original"` on screenshot inputs. This is the
+single most important setting for click accuracy. Avoid `detail: "high"` or
+`detail: "low"`.
+
+**Claude:** Use the `computer_20251124` tool type with `display_width_px` and
+`display_height_px` matching the screenshot dimensions from the coord-map
+output.
+
+**General rules:**
+
+- Take a fresh screenshot after every action
+- Always pass the coord-map from the screenshot the model analyzed
+- If clicks land in wrong spots, use `debug-point` to diagnose
+- If the model returns coordinates outside screenshot dimensions, re-send
+  the screenshot and remind it of the image size
+
+## Troubleshooting
+
+1. **Clicks land in wrong position** — you probably forgot `--coord-map`,
+   or you are passing a coord-map from a different screenshot than the one
+   the model analyzed. Always use the coord-map from the most recent screenshot.
+
+2. **Retina displays** — usecomputer handles scaling internally via
+   coord-map. Do not try to manually account for display scaling.
+
+3. **Stale screenshots** — the most common source of bugs. Always take a
+   fresh screenshot after each action. The UI changes constantly.
+
+4. **Permission errors on macOS** — enable Accessibility permission for
+   your terminal app in System Settings > Privacy & Security > Accessibility.
+
+5. **X11 errors on Linux** — ensure `DISPLAY` is set. For XWayland, screenshot
+   falls back to XGetImage automatically if XShm fails.