screenhand 0.1.1 → 0.3.0

package/README.md

@@ -2,86 +2,62 @@
 
 # ScreenHand
 
-**Give AI eyes and hands on your desktop.**
+**Let AI control your desktop — click buttons, fill forms, automate workflows in ~50ms with zero extra AI calls.**
 
-ScreenHand is an [MCP server](https://modelcontextprotocol.io/) that lets AI agents see your screen, click buttons, type text, and control any app on macOS and Windows.
+An open-source [MCP server](https://modelcontextprotocol.io/) for macOS and Windows. Works with Claude, Cursor, Codex CLI, and any MCP-compatible client.
 
 [![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-blue.svg)](LICENSE)
 [![npm: screenhand](https://img.shields.io/npm/v/screenhand)](https://www.npmjs.com/package/screenhand)
+[![CI](https://github.com/manushi4/screenhand/actions/workflows/ci.yml/badge.svg)](https://github.com/manushi4/screenhand/actions/workflows/ci.yml)
 [![Platform: macOS & Windows](https://img.shields.io/badge/Platform-macOS%20%7C%20Windows-green)]()
 [![MCP Compatible](https://img.shields.io/badge/MCP-Compatible-purple)]()
 
-[Website](https://screenhand.com) | [Quick Start](#quick-start) | [Use Cases](#use-cases) | [FAQ](#faq)
+[Quick Start](#quick-start) | [What It Does](#what-it-does) | [Example](#example) | [All 111 Tools](docs/tools.md) | [Architecture](docs/architecture.md) | [Website](https://screenhand.com)
 
 </div>
 
 ---
 
+<!-- TODO: Add demo GIF here — 15 sec showing Claude controlling a real app -->
+
 ## The Problem
 
-AI assistants are powerful — but they're blind. They can't see what's on your screen, click a button, or type into an app. If you want Claude to help you automate a workflow, debug a UI, or fill out a form, you're stuck copy-pasting screenshots and describing what you see.
+AI assistants can write code but can't use your computer. Every click requires a screenshot → LLM interpretation → coordinate guess — **3-5 seconds and an API call per action**.
 
-**ScreenHand fixes that.** It gives any AI agent direct access to your desktop through native OS APIs — not slow screenshot-and-guess loops.
+ScreenHand gives AI direct access to native OS APIs. No screenshots needed for clicks. No AI calls for button presses.
 
-## How It Works
+| | Without ScreenHand | With ScreenHand |
+|---|---|---|
+| Click a button | Screenshot → LLM → coordinate click (~3-5s) | Native Accessibility API (~50ms) |
+| Cost per action | 1 LLM API call | 0 LLM calls |
+| Accuracy | Coordinate guessing — misses on layout shift | Exact element targeting by role/name |
+| Browser control | Needs focus, screenshot per action | CDP in background (~10ms), no focus needed |
+| Works across apps | One app at a time | Cross-app workflows, multi-agent coordination |
 
-You connect ScreenHand to your AI client (Claude, Cursor, Codex CLI, etc.) via the [Model Context Protocol](https://modelcontextprotocol.io/). Once connected, your AI can:
+## Quick Start
 
-- **See** your screen via screenshots and OCR
-- **Read** UI elements directly via native Accessibility APIs
-- **Click** buttons, menus, and links
-- **Type** text into any input field
-- **Control** Chrome tabs via DevTools Protocol
-- **Automate** cross-app workflows
+### 1. Add to your AI client (one step)
 
-```
-Your AI Client (Claude, Cursor, etc.)
-    |  MCP protocol (stdio)
-ScreenHand
-    |  Native OS APIs
-Your Desktop (any app, any browser)
-```
-
-## Quick Start
+<details open>
+<summary><b>Claude Code</b> (recommended)</summary>
 
 ```bash
-git clone https://github.com/manushi4/screenhand.git
-cd screenhand
-npm install
-npm run build:native   # macOS — builds Swift bridge
-# npm run build:native:windows   # Windows — builds .NET bridge
+claude mcp add screenhand -- npx -y screenhand
 ```
 
-### Connect to Your AI Client
-
-<details>
-<summary><strong>Claude Desktop</strong></summary>
-
-Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
-
-```json
-{
-  "mcpServers": {
-    "screenhand": {
-      "command": "npx",
-      "args": ["tsx", "/path/to/screenhand/mcp-desktop.ts"]
-    }
-  }
-}
-```
+Done. That's it.
 </details>
 
 <details>
-<summary><strong>Claude Code</strong></summary>
-
-Add to your project `.mcp.json` or `~/.claude/settings.json`:
+<summary><b>Claude Desktop</b></summary>
 
+Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
 ```json
 {
   "mcpServers": {
     "screenhand": {
       "command": "npx",
-      "args": ["tsx", "/path/to/screenhand/mcp-desktop.ts"]
+      "args": ["-y", "screenhand"]
     }
   }
 }
@@ -89,16 +65,15 @@ Add to your project `.mcp.json` or `~/.claude/settings.json`:
 </details>
 
 <details>
-<summary><strong>Cursor</strong></summary>
-
-Add to `.cursor/mcp.json` in your project (or `~/.cursor/mcp.json` globally):
+<summary><b>Cursor</b></summary>
 
+Add to `.cursor/mcp.json`:
 ```json
 {
   "mcpServers": {
     "screenhand": {
       "command": "npx",
-      "args": ["tsx", "/path/to/screenhand/mcp-desktop.ts"]
+      "args": ["-y", "screenhand"]
     }
   }
 }
@@ -106,127 +81,236 @@ Add to `.cursor/mcp.json` in your project (or `~/.cursor/mcp.json` globally):
 </details>
 
 <details>
-<summary><strong>OpenAI Codex CLI</strong></summary>
+<summary><b>OpenAI Codex CLI</b></summary>
 
 Add to `~/.codex/config.toml`:
-
 ```toml
 [mcp.screenhand]
 command = "npx"
-args = ["tsx", "/path/to/screenhand/mcp-desktop.ts"]
+args = ["-y", "screenhand"]
 transport = "stdio"
 ```
 </details>
 
 <details>
-<summary><strong>Any MCP Client</strong></summary>
+<summary><b>Any MCP Client</b></summary>
 
-ScreenHand is a standard MCP server over stdio. Point any MCP-compatible client at `mcp-desktop.ts`.
+ScreenHand is a standard MCP server over stdio. Run with `npx -y screenhand`.
 </details>
 
-Replace `/path/to/screenhand` with the actual path where you cloned the repo.
+### 2. Grant permissions
 
-## Use Cases
+**macOS**: System Settings > Privacy & Security > Accessibility > enable your terminal app.
 
-### Automate Repetitive Workflows
-Tell your AI "submit this form on 10 websites" or "export all these reports as PDFs" — and it does it. ScreenHand handles the clicking, typing, and navigating across any app.
+**Windows**: No special permissions needed.
 
-### Debug UIs Faster
-Instead of clicking through your app manually, let Claude inspect the full UI element tree, check states, and walk through flows — all from your terminal.
+### 3. Browser control (optional)
 
-### Browser Automation Without Selenium
-Fill forms, scrape data, run JavaScript, and navigate pages through Chrome DevTools Protocol. Works with sites that block traditional automation.
+Launch Chrome with remote debugging to enable browser tools:
+```bash
+open -a "Google Chrome" --args --remote-debugging-port=9222
+```
 
-### Cross-App Workflows
-Read data from a spreadsheet, search it in Chrome, paste results into Notes — chain actions across your entire desktop.
+That's it. Your AI client now has 111 tools for desktop automation.
 
-### AI-Powered UI Testing
-Click buttons, verify text appears, check element states, and catch regressions — all driven by your AI agent.
+<details>
+<summary><b>Building from source</b> (contributors only)</summary>
 
-## What's Included
+```bash
+git clone https://github.com/manushi4/screenhand.git
+cd screenhand && npm install && npm run build:native
+```
 
-ScreenHand exposes **70+ tools** organized by what you need to do:
+On Windows, use `npm run build:native:windows` instead.
+</details>
 
-| Category | Examples | What For |
-|----------|----------|----------|
-| **Screen** | `screenshot`, `ocr` | See what's on screen, read all visible text |
-| **App Control** | `ui_tree`, `ui_press`, `menu_click` | Read and interact with any native app |
-| **Keyboard & Mouse** | `click`, `type_text`, `key`, `drag` | Direct input control |
-| **Chrome Browser** | `browser_navigate`, `browser_js`, `browser_dom` | Full browser automation via CDP |
-| **Memory** | `memory_recall`, `memory_save` | ScreenHand learns from past sessions |
-| **AppleScript** | `applescript` | Run AppleScript on macOS |
+---
 
-For the full tool reference, see the [tool documentation](DESKTOP_MCP_GUIDE.md).
+## What It Does
 
-## Requirements
+ScreenHand gives AI agents seven capabilities:
 
-| | macOS | Windows |
-|---|---|---|
-| **OS** | macOS 12+ | Windows 10 (1809+) |
-| **Runtime** | Node.js 18+ | Node.js 18+ |
-| **Permissions** | Accessibility (System Settings) | None (no admin needed) |
-| **Browser tools** | Chrome with `--remote-debugging-port=9222` | Same |
-| **Build** | `npm run build:native` | `npm run build:native:windows` |
+### Desktop Control — 19 tools
+Click buttons, type text, read UI trees, navigate menus, drag, scroll — all via native Accessibility APIs in ~50ms. Works with any app: Finder, Notes, VS Code, Xcode, System Settings, etc.
+
+### Browser Automation — 15 tools
+Full Chrome control via DevTools Protocol. Navigate, click, type, run JavaScript, fill forms — all in the background at ~10ms. Built-in anti-detection (`browser_stealth`, `browser_human_click`) for sites with bot protection.
+
+### Smart Fallbacks — 8 tools
+`click_with_fallback`, `type_with_fallback`, etc. automatically try Accessibility → CDP → OCR → coordinates. You don't have to pick the right method — ScreenHand figures it out.
+
+### Memory & Learning — 14 tools
+Gets smarter every session. Logs tool calls, saves winning strategies, tracks error patterns with fixes. Zero config, zero latency overhead (in-memory cache, async disk writes). Ships with 12 seed strategies for common macOS workflows. 6 learning policies: locator stability, sensor effectiveness, recovery ranking, pattern recognition, adaptive timing, and topology (navigation edge reliability).
+
+### App Mastery Map — automatic per-app spatial understanding
+Builds a persistent reverse-engineered blueprint of every app from normal tool usage. 8 features record automatically: page zones, navigation graph (BFS pathfinding), hierarchy, I/O contracts, state machine, element visibility, timing profiles, and ready signals. Mastery levels (beginner → pro → expert → grandmaster) honestly reflect how well ScreenHand knows each app. Maps stored at `~/.screenhand/app-maps/`.
+
+### Jobs & Orchestration — 34 tools
+Queue multi-step jobs, run them via background worker daemon, coordinate multiple AI agents with session leases, detect stalls, auto-recover. Survives client restarts.
 
-## Development
+### Perception & Planning — 17 tools
+Continuous screen awareness (3-rate perception loop at 100ms/300ms/1000ms), real-time world model with entity tracking, goal-oriented planning with auto-decomposition, recovery engine with self-healing. The system always knows what's on screen and feeds observations into the App Mastery Map.
+
+> **Full reference**: See all [111 tools with descriptions](docs/tools.md).
+
+---
+
+## Example
+
+**Browser** — Claude controls Chrome in the background while you work:
+
+```
+You: Search for "screenhand" on Instagram
+
+→ browser_tabs()                                        # ~10ms
+  [34DF5DE1] Instagram — https://www.instagram.com/
+
+→ browser_js({ code: "/* click Search icon */" })       # ~10ms
+→ browser_fill_form({ selector: "input", text: "screenhand" })  # ~50ms (human-like)
+→ browser_js({ code: "/* extract results */" })         # ~10ms
+
+Found @screenhand_ as the top result.
+```
+
+**Desktop** — native app control without screenshots:
+
+```
+→ apps()                     # List running apps           ~10ms
+→ focus("com.apple.Notes")   # Bring Notes to front        ~10ms
+→ ui_tree()                  # Read full UI element tree    ~50ms
+→ ui_press("New Note")       # Click "New Note" button     ~50ms
+→ type_text("Hello world")   # Type text                   ~30ms
+```
+
+**Cross-app** — chain actions across your whole desktop:
+
+```
+→ browser_js(...)            # Extract data from Chrome
+→ focus("com.apple.Notes")   # Switch to Notes
+→ type_text(extractedData)   # Paste it in
+→ key("cmd+s")               # Save
+```
+
+---
+
+## Claude Code Plugin
+
+If you use Claude Code, ScreenHand includes a plugin with **13 skills and 5 agents** that wrap all 111 tools into intent-oriented workflows.
 
 ```bash
-npm run check              # type-check
-npm test                   # run test suite
-npm run build              # compile TypeScript
-npm run build:native       # build native bridge
+./install-plugin.sh   # after npm install && npm run build:native
 ```
 
+| Skill | What it does |
+|-------|-------------|
+| `/automate` | Control any desktop app |
+| `/post-social` | Post to X, LinkedIn, Instagram, Reddit, Threads, Discord |
+| `/run-campaign` | Multi-platform marketing campaigns |
+| `/edit-video` | DaVinci Resolve automation |
+| `/design-figma` | Figma design via Plugin API + browser |
+| `/edit-canva` | Canva template editing |
+| `/scrape-web` | Data extraction with anti-detection |
+| `/fill-form` | Human-like form filling |
+| `/qa-smoke-test` | Automated UI testing |
+| `/record-workflow` | Record into reusable playbooks |
+| `/learn-platform` | Discover how to automate a new app/site |
+| `/run-jobs` | Job queues, background workers |
+| `/manage-system` | Supervisor, memory, diagnostics |
+
+5 specialized agents: **marketing**, **design**, **QA**, **scraper**, **orchestrator**.
+
+---
+
+## How It Works
+
+```
+AI Client (Claude, Cursor, Codex CLI)
+    ↓ MCP protocol (stdio)
+ScreenHand MCP Server (TypeScript)
+    ↓ JSON-RPC (stdio)
+Native Bridge (Swift on macOS / C# on Windows)
+    ↓ OS APIs
+Accessibility, CoreGraphics, Vision, UI Automation, SendInput
+```
+
+ScreenHand reads the UI tree and DOM directly — no screenshots needed for most operations. When screenshots are needed (canvas apps, visual verification), OCR runs in ~600ms via the native Vision framework.
+
+---
+
+## Requirements
+
+| | macOS | Windows |
+|---|---|---|
+| OS | macOS 12+ | Windows 10 (1809+) |
+| Runtime | Node.js 18+ | Node.js 18+ |
+| Native | Swift (included) | [.NET 8 SDK](https://dotnet.microsoft.com/download/dotnet/8.0) |
+| Permissions | Accessibility access for terminal | None (UI Automation works without admin) |
+| Browser | Chrome with `--remote-debugging-port=9222` | Same |
+
+## Docs
+
+| Document | What's in it |
+|----------|-------------|
+| [All 111 Tools](docs/tools.md) | Complete tool reference with descriptions and speeds |
+| [Architecture](docs/architecture.md) | 7-layer design, app tiers, performance targets |
+| [App Mastery Map](docs/app-mastery-map.md) | Layer 7: persistent spatial understanding, 8 auto-recording features |
+| [Bug Tracker](docs/l2-bug-tracker.md) | 103 bugs found and fixed, 80-scenario validation results |
+| [Testing Plan](docs/testing-plan.md) | L1/L2 test methodology and gate criteria |
+
 ## FAQ
 
 <details>
-<summary><strong>What is ScreenHand?</strong></summary>
+<summary><b>How is this different from Anthropic's Computer Use?</b></summary>
 
-An MCP server that gives AI agents the ability to see and control your desktop. It uses native OS APIs (Accessibility on macOS, UI Automation on Windows) for fast, reliable automation — not slow screenshot-based guessing.
+Computer Use is cloud-based and screenshot-driven. ScreenHand is local-first, uses native OS APIs (50ms vs 3-5s per action), costs zero API calls for clicks/typing, and runs entirely on your machine.
 </details>
 
 <details>
-<summary><strong>How is this different from Anthropic's Computer Use?</strong></summary>
+<summary><b>What apps can it control?</b></summary>
 
-Computer Use is cloud-based and built into Claude. ScreenHand is open-source, runs locally on your machine, and uses native OS APIs which are faster and more reliable than screenshot-based approaches. It also works with any MCP-compatible client, not just Claude.
+Any app with Accessibility support (most macOS/Windows apps). Chrome and Electron apps get full DOM access via CDP. Canvas-heavy apps (games, Photoshop viewport) use OCR as fallback.
 </details>
 
 <details>
-<summary><strong>Is it safe?</strong></summary>
+<summary><b>Is it safe?</b></summary>
 
-ScreenHand runs entirely on your machine — no screen data is sent to external servers. All tool calls are audit-logged. See our [Security Policy](SECURITY.md) for details on permissions and boundaries.
+Runs locally, never sends screen data externally. PII is redacted from all persisted data (memory, playbooks, strategies). Dangerous protocols (`javascript:`, `data:`) are blocked. AppleScript and browser JS execution are audit-logged.
 </details>
 
 <details>
-<summary><strong>What AI clients work with it?</strong></summary>
+<summary><b>Does it work with multiple AI agents at once?</b></summary>
 
-Any MCP-compatible client: Claude Desktop, Claude Code, Cursor, Windsurf, OpenAI Codex CLI, and more.
+Yes. Session leases with heartbeat prevent conflicts. The supervisor daemon detects stalls and recovers. Each agent claims its own app window.
 </details>
 
 <details>
-<summary><strong>Can it control any app?</strong></summary>
+<summary><b>How fast is it?</b></summary>
 
-On macOS, any app that exposes Accessibility elements (most do). On Windows, any app supporting UI Automation. For apps with custom rendering (games, some Electron apps), OCR is available as a fallback.
+Accessibility: ~50ms. Chrome CDP: ~10ms (background, no focus needed). OCR: ~600ms. Memory lookups: ~0ms (in-memory cache). All disk writes are async and non-blocking.
 </details>
 
 ## Contributing
 
-Contributions welcome! Please open an issue first to discuss what you'd like to change.
-
 ```bash
 git clone https://github.com/manushi4/screenhand.git
-cd screenhand
-npm install && npm run build:native && npm test
+cd screenhand && npm install && npm run build:native
+npm test   # 1306 tests, 53 files
 ```
 
+## Contact
+
+- **Email**: [khushi@clazro.com](mailto:khushi@clazro.com)
+- **Issues**: [github.com/manushi4/screenhand/issues](https://github.com/manushi4/screenhand/issues)
+- **Website**: [screenhand.com](https://screenhand.com)
+
 ## License
 
-[AGPL-3.0](LICENSE) — Copyright (C) 2025 Clazro Technology Private Limited
+AGPL-3.0-only — Copyright (C) 2025-2026 Clazro Technology Private Limited
 
 ---
 
 <div align="center">
 
-**[screenhand.com](https://screenhand.com)** | Built by **[Clazro Technology Private Limited](https://github.com/manushi4)**
+**[screenhand.com](https://screenhand.com)** | [khushi@clazro.com](mailto:khushi@clazro.com) | A product of **Clazro Technology Private Limited**
 
 </div>