@steipete/peekaboo 3.0.0-beta1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Peter Steinberger
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,120 @@
+# Peekaboo 🫣 - Mac automation that sees the screen and does the clicks.
+
+![Peekaboo Banner](assets/peekaboo.png)
+
+[![npm package](https://img.shields.io/badge/npm_package-3.0.0--beta1-brightgreen?logo=npm&logoColor=white&style=flat-square)](https://www.npmjs.com/package/@steipete/peekaboo)
+[![License: MIT](https://img.shields.io/badge/License-MIT-ffd60a?style=flat-square)](https://opensource.org/licenses/MIT)
+[![macOS 15.0+ (Sequoia)](https://img.shields.io/badge/macOS-15.0%2B_(Sequoia)-0078d7?logo=apple&logoColor=white&style=flat-square)](https://www.apple.com/macos/)
+[![Swift 6.2](https://img.shields.io/badge/Swift-6.2-F05138?logo=swift&logoColor=white&style=flat-square)](https://swift.org/)
+[![node >=22](https://img.shields.io/badge/node-%3E%3D22.0.0-2ea44f?logo=node.js&logoColor=white&style=flat-square)](https://nodejs.org/)
+[![Download macOS](https://img.shields.io/badge/Download-macOS-000000?logo=apple&logoColor=white&style=flat-square)](https://github.com/steipete/peekaboo/releases/latest)
+[![Homebrew](https://img.shields.io/badge/Homebrew-steipete%2Ftap-b28f62?logo=homebrew&logoColor=white&style=flat-square)](https://github.com/steipete/homebrew-tap)
+[![Ask DeepWiki](https://img.shields.io/badge/Ask-DeepWiki-0088cc?style=flat-square)](https://deepwiki.com/steipete/peekaboo)
+
+Peekaboo brings high-fidelity screen capture, AI analysis, and complete GUI automation to macOS. Version 3 adds native agent flows and multi-screen automation across the CLI and MCP server.
+> Note: v3 is currently in beta (3.0.0-beta1) and has a few known issues; see the changelog for details.
+
+## What you get
+- Pixel-accurate captures (windows, screens, menu bar) with optional Retina 2x scaling.
+- Natural-language agent that chains Peekaboo tools (see, click, type, scroll, hotkey, menu, window, app, dock, space).
+- Menu and menubar discovery with structured JSON; no clicks required.
+- Multi-provider AI: GPT-5.1 family, Claude 4.x, Grok 4-fast (vision), Gemini 2.5, and local Ollama models.
+- MCP server for Claude Desktop and Cursor plus a native CLI; the same tools in both.
+- Configurable, testable workflows with reproducible sessions and strict typing.
+- Requires macOS Screen Recording + Accessibility permissions (see [docs/permissions.md](docs/permissions.md)).
+
+## Install
+- macOS app + CLI (Homebrew):
+  ```bash
+  brew install steipete/tap/peekaboo
+  ```
+- MCP server (Node 22+, no global install needed):
+  ```bash
+  npx -y @steipete/peekaboo
+  ```
+
+## Quick start
+```bash
+# Capture full screen at Retina scale and save to Desktop
+peekaboo image --mode screen --retina --path ~/Desktop/screen.png
+
+# Click a button by label (captures, resolves, and clicks in one go)
+peekaboo see --app Safari --json-output | jq -r '.data.session_id' | read SID
+peekaboo click --on "Reload this page" --session "$SID"
+
+# Run a natural-language automation
+peekaboo "Open Notes and create a TODO list with three items"
+
+# Run as an MCP server (Claude/Cursor)
+npx -y @steipete/peekaboo
+
+# Minimal Claude Desktop config snippet (Developer → Edit Config):
+# {
+#   "mcpServers": {
+#     "peekaboo": {
+#       "command": "npx",
+#       "args": ["-y", "@steipete/peekaboo"],
+#       "env": {
+#         "PEEKABOO_AI_PROVIDERS": "openai/gpt-5.1,anthropic/claude-opus-4"
+#       }
+#     }
+#   }
+# }
+```
+
+| Command | Key flags / subcommands | What it does |
+| --- | --- | --- |
+| [see](docs/commands/see.md) | `--app`, `--mode screen/window`, `--retina`, `--json-output` | Capture and annotate UI, return session + element IDs |
+| [click](docs/commands/click.md) | `--on <id/query>`, `--session`, `--wait`, coords | Click by element ID, label, or coordinates |
+| [type](docs/commands/type.md) | `--text`, `--clear`, `--delay-ms` | Enter text with pacing options |
+| [press](docs/commands/press.md) | key names, `--repeat` | Special keys and sequences |
+| [hotkey](docs/commands/hotkey.md) | combos like `cmd,shift,t` | Modifier combos (cmd/ctrl/alt/shift) |
+| [scroll](docs/commands/scroll.md) | `--on <id>`, `--direction up/down`, `--ticks` | Scroll views or elements |
+| [swipe](docs/commands/swipe.md) | `--from/--to`, `--duration`, `--steps` | Smooth gesture-style drags |
+| [drag](docs/commands/drag.md) | `--from/--to`, modifiers, Dock/Trash targets | Drag-and-drop between elements/coords |
+| [move](docs/commands/move.md) | `--to <id/coords>`, `--screen-index` | Position the cursor without clicking |
+| [window](docs/commands/window.md) | `list`, `move`, `resize`, `focus`, `set-bounds` | Move/resize/focus windows and Spaces |
+| [app](docs/commands/app.md) | `launch`, `quit`, `relaunch`, `switch`, `list` | Launch, quit, relaunch, switch apps |
+| [space](docs/commands/space.md) | `list`, `switch`, `move-window` | List or switch macOS Spaces |
+| [menu](docs/commands/menu.md) | `list`, `list-all`, `click`, `click-extra` | List/click app menus and extras |
+| [menubar](docs/commands/menubar.md) | `list`, `click` | Target status-bar items by name/index |
+| [dock](docs/commands/dock.md) | `launch`, `right-click`, `hide`, `show`, `list` | Interact with Dock items |
+| [dialog](docs/commands/dialog.md) | `list`, `click`, `input`, `file`, `dismiss` | Drive system dialogs (open/save/etc.) |
+| [image](docs/commands/image.md) | `--mode screen/window/menu`, `--retina`, `--analyze` | Screenshot screen/window/menu bar (+analyze) |
+| [list](docs/commands/list.md) | `apps`, `windows`, `screens`, `menubar`, `permissions` | Enumerate apps, windows, screens, permissions |
+| [tools](docs/commands/tools.md) | `--source native\|mcp`, `--server <name>` | Inspect native + MCP tools |
+| [config](docs/commands/config.md) | `init`, `show`, `add`, `login`, `models` | Manage credentials/providers/settings |
+| [permissions](docs/commands/permissions.md) | `status`, `grant` | Check/grant required macOS permissions |
+| [run](docs/commands/run.md) | `.peekaboo.json`, `--output`, `--no-fail-fast` | Execute `.peekaboo.json` automation scripts |
+| [sleep](docs/commands/sleep.md) | `--duration` (ms) | Millisecond delays between steps |
+| [clean](docs/commands/clean.md) | `--all-sessions`, `--older-than`, `--session` | Prune sessions and caches |
+| [agent](docs/commands/agent.md) | `--model`, `--dry-run`, `--resume`, `--max-steps`, audio | Natural-language multi-step automation |
+| [mcp](docs/commands/mcp.md) | `serve`, `list`, `add`, `enable/disable`, `test` | Manage external MCP servers and serve Peekaboo |
+
+## Models and providers
+- OpenAI: GPT-5.1 (default) and GPT-4.1/4o vision
+- Anthropic: Claude 4.x
+- xAI: Grok 4-fast reasoning + vision
+- Google: Gemini 2.5 (pro/flash)
+- Local: Ollama (llama3.3, llava, etc.)
+
+Set providers via `PEEKABOO_AI_PROVIDERS` or `peekaboo config add`.
+
+## Learn more
+- Command reference: [docs/commands/](docs/commands/)
+- Architecture: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)
+- Building from source: [docs/building.md](docs/building.md)
+- Testing guide: [docs/testing/tools.md](docs/testing/tools.md)
+- MCP setup: [docs/commands/mcp.md](docs/commands/mcp.md)
+- Permissions: [docs/permissions.md](docs/permissions.md)
+- Ollama/local models: [docs/ollama.md](docs/ollama.md)
+- Agent chat loop: [docs/agent-chat.md](docs/agent-chat.md)
+- Service API reference: [docs/service-api-reference.md](docs/service-api-reference.md)
+
+## Development basics
+- Requirements: macOS 15+, Xcode 16+/Swift 6.2. Node 22+ only if you run the pnpm docs/build helper scripts (core CLI/app/MCP are Swift-only).
+- Install deps: `pnpm install` then `pnpm run build:cli` or `pnpm run test:safe`.
+- Lint/format: `pnpm run lint && pnpm run format`.
+
+## License
+MIT

package/package.json

@@ -0,0 +1,79 @@
+{
+  "name": "@steipete/peekaboo",
+  "version": "3.0.0-beta1",
+  "description": "macOS automation MCP server with screen capture, UI interaction, and AI analysis",
+  "private": false,
+  "type": "module",
+  "main": "peekaboo-mcp.js",
+  "bin": {
+    "peekaboo": "peekaboo",
+    "peekaboo-mcp": "peekaboo-mcp.js"
+  },
+  "files": [
+    "peekaboo",
+    "peekaboo-mcp.js",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "build:cli": "./runner swift build --package-path Apps/CLI",
+    "build:cli:release": "./runner swift build --configuration release --package-path Apps/CLI",
+    "build:swift": "./runner ./scripts/build-swift-arm.sh",
+    "build:swift:all": "./runner ./scripts/build-swift-universal.sh",
+    "build:polter": "./runner polter peekaboo --version",
+    "build": "pnpm run build:cli",
+    "test:safe": "./runner swift test --package-path Apps/CLI -Xswiftc -DPEEKABOO_SKIP_AUTOMATION --no-parallel",
+    "test:automation": "PEEKABOO_INCLUDE_AUTOMATION_TESTS=true ./runner swift test --package-path Apps/CLI --no-parallel",
+    "test:automation:read": "RUN_AUTOMATION_READ=true PEEKABOO_INCLUDE_AUTOMATION_TESTS=true ./runner swift test --package-path Apps/CLI --no-parallel",
+    "test:automation:local": "./runner bash -lc 'cd Apps/CLI && BIN_PATH=$(swift build --show-bin-path) && RUN_LOCAL_TESTS=true PEEKABOO_INCLUDE_AUTOMATION_TESTS=true PEEKABOO_CLI_PATH=\"$BIN_PATH/peekaboo\" swift test --no-parallel'",
+    "test:all": "./runner bash -lc 'set -euo pipefail; cd Apps/CLI && swift test -Xswiftc -DPEEKABOO_SKIP_AUTOMATION --no-parallel && PEEKABOO_INCLUDE_AUTOMATION_TESTS=true swift test --no-parallel'",
+    "test": "pnpm run test:safe",
+    "tachikoma:test:integration": "./runner bash -lc 'cd Tachikoma && source ~/.profile && INTEGRATION_TESTS=1 swift test --parallel -Xswiftc -DLIVE_PROVIDER_TESTS'",
+    "lint:swift": "./runner swiftlint lint --config .swiftlint.yml",
+    "lint": "pnpm run lint:swift",
+    "format:swift": "./runner swiftformat .",
+    "format": "pnpm run format:swift",
+    "prepare-release": "./runner node scripts/prepare-release.js",
+    "docs:list": "./runner node scripts/docs-list.mjs",
+    "lint:docs": "./runner node scripts/docs-lint.mjs",
+    "polter": "FORCE_COLOR=1 CLICOLOR_FORCE=1 NODE_PATH=../poltergeist/node_modules script -q /dev/null ./runner node ../poltergeist/dist/polter.js",
+    "polter:dev": "cd /Users/steipete/Projects/Peekaboo && FORCE_COLOR=1 CLICOLOR_FORCE=1 NODE_PATH=../poltergeist/node_modules ./runner pnpm --dir ../poltergeist exec tsx ../poltergeist/src/polter.ts",
+    "peekaboo": "FORCE_COLOR=1 CLICOLOR_FORCE=1 NODE_PATH=../poltergeist/node_modules script -q /dev/null ./scripts/poltergeist-wrapper.sh peekaboo",
+    "peekaboo:dev": "pnpm run polter:dev -- peekaboo",
+    "poltergeist:start": "./runner ./scripts/poltergeist-wrapper.sh start",
+    "poltergeist:haunt": "./runner ./scripts/poltergeist-wrapper.sh haunt",
+    "poltergeist:stop": "./runner ./scripts/poltergeist-wrapper.sh stop",
+    "poltergeist:rest": "./runner ./scripts/poltergeist-wrapper.sh rest",
+    "poltergeist:status": "./runner ./scripts/poltergeist-wrapper.sh status",
+    "poltergeist:logs": "./runner ./scripts/poltergeist-wrapper.sh logs",
+    "oracle": "./runner pnpm -C ../oracle oracle",
+    "postinstall": "chmod +x peekaboo peekaboo-mcp.js 2>/dev/null || true"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/steipete/peekaboo.git"
+  },
+  "author": "Peter Steinberger <steipete@gmail.com>",
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/steipete/peekaboo/issues"
+  },
+  "homepage": "https://github.com/steipete/peekaboo#readme",
+  "keywords": [
+    "mcp",
+    "model-context-protocol",
+    "macos",
+    "automation",
+    "screen-capture",
+    "ai"
+  ],
+  "engines": {
+    "node": ">=22.0.0"
+  },
+  "os": ["darwin"],
+  "cpu": ["x64", "arm64"],
+  "dependencies": {},
+  "devDependencies": {
+    "chrome-devtools-mcp": "0.10.1"
+  }
+}

package/peekaboo-mcp.js

@@ -0,0 +1,111 @@
+#!/usr/bin/env node
+// Peekaboo MCP wrapper that restarts the Swift server on crash
+
+import { spawn, execSync } from 'child_process';
+import { fileURLToPath } from 'url';
+import { dirname, join } from 'path';
+import { existsSync } from 'fs';
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const binaryPath = join(__dirname, 'peekaboo');
+
+const MAX_RESTARTS = 5;
+const RESTART_WINDOW_MS = 60_000;
+const INITIAL_DELAY_MS = 1000;
+const MAX_DELAY_MS = 30_000;
+
+class PeekabooMCPWrapper {
+  constructor() {
+    this.restartTimestamps = [];
+    this.delay = INITIAL_DELAY_MS;
+    this.child = null;
+    this.shuttingDown = false;
+  }
+
+  start() {
+    if (!existsSync(binaryPath)) {
+      console.error(`[Peekaboo MCP] Binary not found at ${binaryPath}`);
+      process.exit(1);
+    }
+
+    const now = Date.now();
+    this.restartTimestamps = this.restartTimestamps.filter(ts => now - ts < RESTART_WINDOW_MS);
+    if (this.restartTimestamps.length >= MAX_RESTARTS) {
+      console.error(`[Peekaboo MCP] Aborting: restarted ${MAX_RESTARTS} times within a minute.`);
+      process.exit(1);
+    }
+
+    console.error('[Peekaboo MCP] Starting Swift server...');
+    this.child = spawn(binaryPath, ['mcp', 'serve'], {
+      stdio: 'inherit',
+      env: {
+        ...process.env,
+        PEEKABOO_MCP_WRAPPER: 'true'
+      }
+    });
+
+    this.child.on('exit', (code, signal) => {
+      if (this.shuttingDown) return process.exit(code || 0);
+
+      if (code === 0 || signal === 'SIGINT' || signal === 'SIGTERM') {
+        console.error('[Peekaboo MCP] Server exited cleanly');
+        process.exit(code || 0);
+      }
+
+      this.handleCrash(code, signal);
+    });
+
+    this.child.on('error', (err) => {
+      console.error('[Peekaboo MCP] Failed to launch:', err.message);
+      if (err.code === 'EACCES') {
+        try {
+          execSync(`chmod +x "${binaryPath}"`);
+          console.error('[Peekaboo MCP] Fixed executable bit, retrying...');
+          this.handleCrash(1);
+          return;
+        } catch (_) {
+          console.error('[Peekaboo MCP] Could not make binary executable.');
+        }
+      }
+      this.handleCrash(err.code || 1);
+    });
+  }
+
+  handleCrash(code, signal) {
+    console.error(`[Peekaboo MCP] Server crashed (code ${code}${signal ? `, signal ${signal}` : ''}).`);
+    this.restartTimestamps.push(Date.now());
+    setTimeout(() => {
+      this.delay = Math.min(this.delay * 2, MAX_DELAY_MS);
+      this.start();
+    }, this.delay);
+  }
+
+  shutdown() {
+    this.shuttingDown = true;
+    if (this.child && !this.child.killed) {
+      this.child.kill('SIGTERM');
+    }
+  }
+}
+
+const wrapper = new PeekabooMCPWrapper();
+wrapper.start();
+
+process.on('SIGINT', () => {
+  console.error('\n[Peekaboo MCP] SIGINT received, shutting down...');
+  wrapper.shutdown();
+});
+
+process.on('SIGTERM', () => {
+  console.error('[Peekaboo MCP] SIGTERM received, shutting down...');
+  wrapper.shutdown();
+});
+
+process.on('uncaughtException', (err) => {
+  console.error('[Peekaboo MCP] Uncaught exception:', err);
+  wrapper.shutdown();
+});
+
+process.on('unhandledRejection', (reason) => {
+  console.error('[Peekaboo MCP] Unhandled rejection:', reason);
+});