@hybridlabor-api/aos 4.3.0 → 4.4.0

package/mcps/windows-computer-use-mcp/docs/MEMOPS_CUA.md

@@ -1,147 +0,0 @@
----
-title: windows-computer-use-mcp — doctrine (2026-06-16)
-status: active
-tags: [cua, windows-computer-use-mcp, automation, gui, fleet]
-related:
-  - docs/ROADMAP.md
-  - docs/TOOLS.md
-  - docs/ARCHITECTURE.md
-  - docs/cua-nsis-certification.md
----
-
-# windows-computer-use-mcp — MemOps note
-
-## One-line thesis
-
-**A Windows UI automation MCP server that is both a tool for AI agents and an agent itself** — 22 portmanteau tools, autonomous mission engine, macros, watchers, self-healing, screen recording, dual OCR providers. Replaces pywinauto-mcp (v0.5.x) with full CUA capability.
-
-## Naming
-
-| Public | GitHub | Port |
-|--------|--------|------|
-| **windows-computer-use-mcp** | `sandraschi/windows-computer-use-mcp` | 10989 |
-
-Old `pywinauto-mcp` redirects automatically. Local directory still `pywinauto-mcp` (locked by open handles).
-
-## Current state (v0.7.0 — released 2026-06-15)
-
-### Tool surface (22 portmanteau tools)
-
-| Tool | Operations | Phase |
-|------|-----------|-------|
-| `automation_windows` | list, focus, maximize, minimize, close, get_state, wait_window | core |
-| `automation_elements` | find, click, set_text, get_text, select_option, verify, state, tree | core |
-| `automation_mouse` | click, double_click, right_click, move_to, drag, scroll, get_pos | core |
-| `automation_keyboard` | type_text, press_key, hotkey, paste, get_state | core |
-| `automation_visual` | screenshot, ocr, highlight, get_window_state, record, record_to_gif | core |
-| `automation_shortcut` | semantic shortcuts per app, multi-shortcut workflows | core |
-| `automation_dialog` | list, wait, click_button, set_field, handle_file_dialog | core |
-| `automation_assert` | verify_state, verify_text, verify_file, verify_process | core |
-| `automation_smart` | discover, click (by intent), set_text (by intent), wait (by intent) | P4 |
-| `automation_mission` | run, status, cancel, resume, run_preset | P1 |
-| `automation_macro` | record, stop, replay, replay_with_verify, replay_video, list | P3 |
-| `automation_watch` | start, stop, status, list | P5 |
-| `automation_system` | telemetry, strategies, health, env, settings | P2 |
-| `automation_faces` | detect, save (opt-in, off by default) | core |
-| `automation_keylogger` | start, stop, status (opt-in, off by default) | core |
-| `analyze_winapp` | crawl, discover, portfolio | core |
-
-### Key capabilities
-
-- **Autonomous mission engine**: `automation_mission(run="open Notepad, type hello, save")` — decomposes via LLM sampling, executes each step with retry (exponential backoff), outcome verification, self-healing. Returns per-step pass/fail with evidence.
-- **Outcome verification**: post-condition checks after each action — text appeared, element state changed. Verification failure returns `status=blocked` with detail.
-- **Retry policy**: unified `RetryPolicy` with strategy chain: refocus → wait_stable → fallback_selector → escalate. Each strategy uses exponential backoff.
-- **Adaptive element location**: cascades through title → auto_id → control_id → class+type → OCR region scan when element not found.
-- **Self-healing**: mission engine checks window alive before each step, re-launches app if dead, aborts after 5 consecutive failures.
-- **Cross-app data flow**: steps with `store_as` save to shared context; `$ref:key` reads from context.
-- **Event-driven watchers**: background thread polling — window_appears, window_closes, text_appears, element_appears.
-- **Telemetry store**: SQLite-backed at `~/.windows-computer-use-mcp/telemetry.db`. Every action logged with strategy, success, duration.
-- **Telemetry-driven strategy**: mission engine queries `get_best_strategy()` from telemetry before each step.
-- **Mission presets**: `presets/*.json` files loaded by `automation_mission(run_preset="notepad_demo")`.
-- **YAML/JSON workflow runner**: `scripts/run-workflow.py` with `just run-workflow <file>` recipe.
-
-### OCR (dual provider)
-
-| Provider | Default | Install | Speed |
-|----------|---------|---------|-------|
-| **Windows Media OCR** | **Yes** (auto-probed first) | Zero-install (WinRT built-in) | Fast |
-| **Tesseract 5.5** | Fallback | `just install-tesseract` or manual | Moderate |
-
-Env: `WINDOWS_COMPUTER_USE_MCP_OCR_PROVIDER=auto|tesseract|windowsmedia`
-
-### Screen recording
-
-- `automation_visual(record, duration=10, fps=10)` — MP4 via ffmpeg pipe (no intermediate files)
-- `record_to_gif` — animated GIF output
-- `automation_macro(replay, record_video=True)` — records screen during macro replay
-- ffmpeg 8.1.1 at `C:\Users\sandr\scoop\shims\ffmpeg.exe`
-
-### System tray controller
-
-- `scripts/tray-control.py` — standalone tray app
-- Health icon (green/grey), HITL toggle, approve 5min
-- Recording indicator (red R icon) during capture
-- Quick macro replay from saved files
-- View log, server status, telemetry query
-- Ctrl+Shift+R hotkey for recording
-- 5-second polling
-
-### CUA-NSIS certification
-
-12 phases pass: kill → install → launch → health → window → screenshot → diagnostics → uninstall → Tesseract → 3 desktop tests → cert. Config at `scripts/cua-nsis-config.json`.
-
-### Safety
-
-- HITL (human-in-the-loop) blocks all mutating actions by default
-- Bypass via `WINDOWS_COMPUTER_USE_MCP_BYPASS_HITL=1`
-- Blocked actions return `status="error"` with `"Blocked by HITL safety setting: ..."`
-- Keylogger and face detection opt-in only (off by default)
-
-## What it is NOT
-
-- Not a vision model — screenshots go to host LLM
-- Not reliable on Unity/GPU apps without foreground + calibration
-- Not a replacement for `autohotkey-mcp` (AHK has faster hotkey dispatch for power users)
-
-## Fleet role
-
-```
-windows-computer-use-mcp (10989)  ← hands: click, type, mission, record
-  → autohotkey-mcp (10747)         ← alternative: faster hotkey dispatch for power users
-  → system-admin-mcp (10861)      ← preflight: process find, RAM/disk depletion
-  → browser-mcp (10781)           ← cross-ref: Playwright + CUA composing
-```
-
-## Design principles
-
-1. **Mission-first** — complex goals should be decomposed and verified, not scripted
-2. **Assert everything** — every action must verify its outcome
-3. **Evidence on failure** — before/after/diff for host LLM review
-4. **Telemetry-driven** — learn from past failures to choose better strategies
-5. **Shortcut where possible** — clicks only as fallback
-
-## Roadmap
-
-See [docs/ROADMAP.md](./ROADMAP.md) — 14 items across short/medium/long term:
-
-- **Short (done)**: Win Media OCR default, screen recording, replay-to-video, tray polish, mission presets, YAML workflow runner
-- **Medium**: multi-monitor coordinates, app-specific automation profiles, YAML/JSON portable runner
-- **Long**: computer vision, voice control, remote desktop, self-improving telemetry, Docker sandbox
-
-## Env cheatsheet
-
-```powershell
-$env:WINDOWS_COMPUTER_USE_MCP_URL = "http://127.0.0.1:10989"
-$env:WINDOWS_COMPUTER_USE_MCP_BYPASS_HITL = "1"       # disable HITL
-$env:WINDOWS_COMPUTER_USE_MCP_OCR_PROVIDER = "tesseract"  # force Tesseract
-$env:WINDOWS_COMPUTER_USE_MCP_DISPATCH = "foreground"  # VRoid / Unity
-$env:WINDOWS_COMPUTER_USE_MCP_KEYBOARD = "win32"       # raw Win32 input
-$env:SYSTEM_ADMIN_MCP_URL = "http://127.0.0.1:10861"
-```
-
-## Relations
-
-- implements [[FLEET_AUTOMATION_PIPELINE]] Windows GUI automation leg
-- blocks [[vroidstudio-mcp]] reliability until fully calibrated
-- pairs with [[virtualization-mcp]] for sandbox isolation (not wired)
-- cross-ref [[autohotkey-mcp]] for complementary AHK-based automation

package/mcps/windows-computer-use-mcp/docs/PRD.md

@@ -1,93 +0,0 @@
-# PRD — windows-computer-use-mcp (Computer Use Agent)
-
-## 1. Overview
-
-**windows-computer-use-mcp** is the fleet's Windows **computer use agent** — a [Model Context Protocol](https://modelcontextprotocol.io/) server that exposes **portmanteau tools** so AI clients can operate the **real desktop**: discover windows, interact with UI Automation elements, inject mouse/keyboard, capture screenshots/OCR, run shortcuts, handle dialogs, and **verify outcomes** (assert).
-
-**Positioning:** *Hands, not brain.* The host LLM plans; this server executes and reports structured evidence. Closed-loop task running is roadmap ([CUA_ROADMAP.md](CUA_ROADMAP.md)); today operators combine MCP tools + HITL.
-
-**Non-goals:** Not a browser DOM agent (use Playwright MCP). Not an isolation boundary — pair with **`virtualization-mcp`** for Sandbox/VM. See [SAFETY.md](SAFETY.md).
-
-**Package name:** `windows-computer-use-mcp` / `windows_computer_use_mcp`. Public fleet label: **Windows Computer Use MCP**.
-
-## 2. Goals
-
-| Goal | Detail |
-|------|--------|
-| **Computer use** | Reliable Windows GUI actuation with structured tool results for agent loops |
-| **Tool efficiency** | Portmanteau tools limit tool explosion and token load |
-| **Framework** | **FastMCP 3.2+** — stdio + HTTP `/mcp`; async tools; docstring-first descriptions |
-| **Safety** | HITL via `approve_automation`, kill switch, rate limits, dry-run; invasive tools opt-in only |
-| **Verification** | `automation_assert`, set-of-mark state, evidence on failure |
-| **Operator UX** | Optional **`web_sota`** + Tauri **operator app** — tools hub, HITL, local LLM chat |
-| **Testability** | Environment-aware pytest per **mcp-central-docs** — [TESTING.md](TESTING.md) |
-| **Distribution** | PyPI (`uvx`), MCPB bundle, desktop installer |
-
-## 3. Key product surfaces
-
-### 3.1 MCP tools (core)
-
-| Tool | Purpose |
-|------|---------|
-| `automation_windows` | Window lifecycle — find, focus, move, resize, close |
-| `automation_elements` | UIA element ops — click, text, tree |
-| `automation_mouse` | Pointer — click, drag, scroll, telemetry HUD |
-| `automation_keyboard` | Keys, hotkeys, typing |
-| `automation_visual` | Screenshot, OCR, image match |
-| `automation_assert` | Outcome verification (CUA loops) |
-| `automation_dialog` | Native dialog handling |
-| `automation_shortcut` | App-specific shortcut tables (e.g. VRoid) |
-| `automation_task` | Multi-step task helpers |
-| `automation_system` | Help, clipboard, processes, `start_app` |
-| `get_desktop_state` / `get_window_state` | Discovery, set-of-mark |
-| `approve_automation`, `automation_safety` | HITL and counters |
-
-**Opt-in only (not registered in default / desktop bundle):**
-
-- `automation_face` — implemented local webcam tool; `WINDOWS_COMPUTER_USE_MCP_ENABLE_FACE=1` + `face` extra ([SAFETY.md](SAFETY.md) §5)
-- `global_keylogger` — implemented session debug hook; non-stealth, HITL-gated; `WINDOWS_COMPUTER_USE_MCP_ENABLE_KEYLOGGER=1` ([SAFETY.md](SAFETY.md) §6)
-
-### 3.2 HTTP / ASGI
-
-- FastAPI **`/api/v1/*`** — health, tools, windows, LLM proxy, cameras, safety
-- FastMCP **`http_app()`** mounted at **`/mcp`**
-- CORS for local `web_sota` dev ports and Tauri webview (`PYWINAUTO_TAURI=1`)
-
-### 3.3 Web operator (`web_sota`)
-
-- Vite UI; **`start.ps1`** — frontend **10788**, backend **10789**
-- Routes: Overview, Windows, Elements, Tools Hub, Local LLM (`/chat`), Help, Biometrics, Settings
-- Local LLM: Ollama / LM Studio via OpenAI-compatible proxy (`PYWINAUTO_LLM_BASE_URL`)
-
-### 3.4 Desktop operator app
-
-- Tauri 2 + PyInstaller sidecar — see [DESKTOP_APP.md](DESKTOP_APP.md), [INSTALL.md](../INSTALL.md)
-- Bundled backend; optional Cursor/Claude registration on first launch
-
-## 4. Technical standards
-
-- **Python 3.12+**, **Windows 10/11** for production automation
-- **Ruff** lint/format; `win32_mouse` for DPI-aware pointer injection
-- **Env:** `CUA_MCP_KEYBOARD=win32`, `CUA_MCP_RETRY_ATTEMPTS`, standard safety keys (see README / SAFETY)
-- **Testing:** `requires_hardware`, `destructive` markers; CI skips hardware probes
-
-## 5. Success metrics (directional)
-
-- Tool calls return structured **`success` / `error`** with recoverable guidance
-- First connection documented for MCP clients ([INSTALL.md](../INSTALL.md), [mcpb/README.md](../mcpb/README.md))
-- Docs stay aligned with implemented tools and env vars
-- Assert/evidence paths support agent retry loops
-
-## 6. References
-
-| Doc | Purpose |
-|-----|---------|
-| [SAFETY.md](SAFETY.md) | Isolation, opt-in invasive tools |
-| [MEMOPS_CUA.md](MEMOPS_CUA.md) | Fleet CUA doctrine |
-| [OPERATOR_PROTOCOL.md](OPERATOR_PROTOCOL.md) | Focus during automation |
-| [TESTING.md](TESTING.md) | CI vs local |
-| **mcp-central-docs** | `patterns/WINDOWS_COMPUTER_USE_MCP_SAFETY.md`, packaging standards |
-
----
-
-*PRD v0.5.x — computer use agent positioning, assert/dialog/shortcut/task tools, operator app — 2026.*

package/mcps/windows-computer-use-mcp/docs/README.md

@@ -1,34 +0,0 @@
-# Documentation index
-
-Hub for **windows-computer-use-mcp**. Root overview: [README.md](../README.md). Releases: [CHANGELOG.md](../CHANGELOG.md).
-
-## Core
-
-| Document | Description |
-|----------|-------------|
-| [PRD.md](PRD.md) | Product requirements |
-| [ARCHITECTURE.md](ARCHITECTURE.md) | System diagram and component map |
-| [TOOLS.md](TOOLS.md) | Full 22-tool reference with phase annotations and examples |
-| [py-stack.md](py-stack.md) | Python dependency deep dive |
-| [SAFETY.md](SAFETY.md) | HITL, kill switch, isolation, opt-in invasive tools |
-| [ROADMAP.md](ROADMAP.md) | Improvement plan — short, medium, long term |
-| [TESTING.md](TESTING.md) | CI vs local pytest, hardware markers |
-| [composing-with-playwright.md](composing-with-playwright.md) | Browser automation via Playwright MCP |
-| [ocr.md](ocr.md) | OCR system — Tesseract setup, limitations, competition |
-| [DEVELOPMENT.md](DEVELOPMENT.md) | Layout, justfile, CI parity |
-
-## CUA
-
-| Document | Description |
-|----------|-------------|
-| [MEMOPS_CUA.md](MEMOPS_CUA.md) | Fleet CUA doctrine |
-| [CUA_PARITY.md](CUA_PARITY.md) | Parity vs other computer-use stacks |
-| [AUTOMATION_ASSERT_SPEC.md](AUTOMATION_ASSERT_SPEC.md) | Assert tool spec |
-| [ASSESSMENT_BY_CURSOR_2026-06-14_CUA_NSIS.md](ASSESSMENT_BY_CURSOR_2026-06-14_CUA_NSIS.md) | CUA-NSIS certification canary |
-
-## Fleet cross-links
-
-- **mcp-central-docs** — patterns/WINDOWS_COMPUTER_USE_MCP_SAFETY.md, fleet standards
-- **Siblings:** [autohotkey-mcp](https://github.com/sandraschi/autohotkey-mcp), [virtualization-mcp](https://github.com/sandraschi/virtualization-mcp)
-
-**Scope:** Native **desktop** UI only. **Website** automation belongs on a browser MCP (typically Playwright).

package/mcps/windows-computer-use-mcp/docs/ROADMAP.md

@@ -1,142 +0,0 @@
-# Roadmap
-
-## Short-term (highest impact)
-
-### 1. Windows Media OCR as default
-Windows Media OCR (via PyWinRT) is zero-install on every Windows 10/11 system and produces comparable results to Tesseract. Currently Tesseract is the default. Switch the default to Windows Media OCR when available, with Tesseract as fallback.
-
-- [ ] Modify `automation_visual` to probe Windows Media OCR first
-- [ ] Remove Tesseract PATH requirement for basic OCR
-- [ ] Add `WINDOWS_COMPUTER_USE_MCP_OCR_PROVIDER` env var override
-
-### 2. Screen recording / GIF capture
-Add ffmpeg-based screen recording to `automation_visual`. A sequence of screenshots can be composited into a GIF or MP4 for demo videos, verification, or debugging failed missions.
-
-- [ ] `automation_visual(record, duration=30, output="demo.mp4")`
-- [ ] `automation_visual(record_to_gif, duration=10, output="demo.gif")`
-- [ ] Auto-attach recording to mission results
-
-### 3. Session replay to video
-When replaying a macro with `automation_macro(replay)`, optionally record the screen simultaneously. Produces a demo video of any automated workflow.
-
-- [ ] `automation_macro(replay_with_video, name="my_macro")`
-- [ ] Records screen during replay, saves as MP4
-- [ ] Useful for documentation, demos, debugging
-
-### 4. HITL tray app polish
-The system tray controller exists (`scripts/tray-control.py`) but is bare. Add:
-
-- [ ] Real-time HITL approval notifications (balloon popup)
-- [ ] Quick-record button (macro record/stop with hotkey shown)
-- [ ] Server status indicator (connection health, tool count)
-- [ ] Macro list with one-click replay
-
-## Medium-term
-
-### 5. Multi-monitor coordinate support
-Window management across multiple monitors. Currently coordinates assume primary display. Add display-aware coordinate translation.
-
-- [x] Virtual screen bounds detection
-- [x] Per-monitor DPI scaling
-- [x] `automation_mouse(position)` returns which monitor cursor is on
-- [x] `monitor_index` on mouse/visual/window tools
-
-### 6. App-specific automation profiles
-Structured profiles for common applications defining element selectors, window states, recovery sequences, and semantic shortcuts. Extends `automation_shortcut`.
-
-- [x] Profile format: YAML with selectors, states, expected behaviors
-- [x] Built-in profiles: Notepad, Paint
-- [x] Auto-detect app and load profile
-
-### 7. Mission presets
-Pre-built workflows for common tasks as JSON files. Loadable via `automation_mission(run_preset="install_app")`.
-
-- [x] Preset format with parameter substitution
-- [x] Registry: `~/.windows-computer-use-mcp/presets/`
-- [x] Initial presets: "screenshot + OCR" (screenshot_demo.json)
-
-### 8. YAML/JSON portable workflow runner
-Define multi-step automations as portable config files. No Python needed to define a workflow.
-
-- [x] `just run-workflow path/to/workflow.yaml`
-- [x] Step format: {tool, params, verify, retry}
-- [x] Cross-repo compatible (same format usable by fleet)
-
-## Long-term
-
-### 9. Computer vision for element finding
-When UIA fails (e.g., custom controls, games, embedded web views), use lightweight object detection to find buttons and controls by appearance.
-
-- [x] Multi-scale template matching across scale range 0.5-2.0
-- [x] Rotation-invariant ORB feature matching
-- [x] Screen region description (contour detection) for agent vision loops
-- [ ] YOLO model for common UI elements (planned)
-
-### 10. Voice control integration
-Integrate with `speech-mcp` to allow voice commands: "click Save", "type hello world", "screenshot". The tray app becomes a voice-controlled automation assistant.
-
-- [x] Speech-to-text via speech-mcp API
-- [x] Natural language → tool call mapping (+ parse_command module)
-- [x] Background VoiceListener for tray app integration
-- [ ] Voice initiator button in tray app (planned)
-
-### 11. Remote desktop automation
-Drive automation on a remote machine via RDP/VNC integration. Two approaches:
-
-**Via rustdesk-mcp:** rustdesk-mcp already provides remote desktop access. Extend it with RDP/VNC protocol support, or create a dedicated `remote-control-mcp` that bundles RDP + VNC + RustDesk under one interface.
-
-- [ ] RDP client automation (send keystrokes/clicks to RDP session)
-- [ ] VNC client automation (same)
-- [ ] `remote-control-mcp` umbrella repo, or extend `rustdesk-mcp`
-- [ ] Transparent: `automation_windows(find, remote="hostname:port")`
-
-### 12. Self-improving telemetry
-The telemetry SQLite store already logs every action. Extend it to:
-
-- [x] Auto-detect failure patterns (specific app + operation + error)
-- [x] Generate GitHub issue drafts from failure clusters
-- [x] Suggest config changes based on error patterns
-- [x] Weekly improvement report
-
-### 13. Docker/sandbox execution mode
-Run automation inside a Windows Sandbox or Docker container instead of on the host. Pairs with `virtualization-mcp`.
-
-- [ ] `automation_system(sandbox=True)` — provision sandbox, run automation there
-- [ ] Guest-side agent: lightweight pywinauto runner in the sandbox
-- [ ] Result extraction: screenshots, logs, evidence back to host
-
-## AI Model Notes
-
-### Kimi K2.7 Code (Moonshot AI, June 2026)
-
-Open-weight coding agent model, very relevant to this repo's mission engine:
-
-| Metric | K2.6 | K2.7 Code |
-|--------|------|-----------|
-| MCP Atlas | 69.4 | **76.0** |
-| MCP Mark Verified | 72.8 | **81.1** |
-| Kimi Claw 24/7 | 42.9 | **46.9** |
-| Thinking tokens | baseline | **-30%** |
-
-- **Architecture:** 1T total / 32B activated MoE, 256K context, MLA attention
-- **Multi-step tool call:** Native interleaved thinking + tool call support — matches `automation_mission(run=...)` pattern exactly
-- **License:** Modified MIT (open weight)
-- **Deployment:** INT4 quantization available, fits RTX 4090 at 4-bit
-- **Vision:** MoonViT encoder (400M params) — can process screenshots for visual grounding
-
-Could serve as the LLM backend for `ctx.sample()` in the mission engine, either via API or local inference.
-
-### Other notable Chinese open-weight models
-
-| Model | Company | Notes |
-|-------|---------|-------|
-| DeepSeek V4 / R2 | DeepSeek | Strong general reasoning, used by opencode |
-| Qwen 3.5 | Alibaba | Good code + agentic, 235B MoE |
-| GLM-5 | Zhipu | Strong on Chinese language tasks |
-| Yi-Lightning | 01.AI | Fast inference, good for real-time |
-
-## Cross-cutting
-
-- **RDP/VNC consolidation:** Consider creating `remote-control-mcp` that unifies RustDesk, RDP, and VNC under a single tool surface. This avoids splitting focus across three repos. rustdesk-mcp can be the reference implementation.
-- **Benchmark suite:** Automate comparison of Tesseract vs Windows Media OCR accuracy across font sizes, UI types, and languages.
-- **User documentation:** Video tutorials for each major feature, linked from README.

package/mcps/windows-computer-use-mcp/docs/SAFETY.md

@@ -1,138 +0,0 @@
-# windows-computer-use-mcp — Safety & isolation
-
-**Read this before enabling this server in an IDE.** Desktop automation is **host-powerful**; “sandbox safety” for **untrusted UI** requires a **second fleet server**, not pywinauto alone.
-
-**Foreground / focus:** If you automate “start app X and test it,” read **[OPERATOR_PROTOCOL.md](OPERATOR_PROTOCOL.md)** — the user should **step back from the terminal/IDE** so automation does not fight for focus.
-
----
-
-## Dual-use tooling: research, forensics, and guardrails
-
-### Why this looks “malware-adjacent”
-
-Anything that does **session-wide capture**, **low-level hooks**, or **unattended control of the desktop** shares **capabilities** with a lot of offensive tooling. That is an observation about **surface area**, not a moral verdict on this repository.
-
-### Why hosted assistants sometimes refuse to help
-
-A model vendor may decline to **generate** keyloggers or similar hooks because they optimize for **reducing harm from misuse**. That policy stance is about **mass-market generation**, not a technical claim that your use case is wrong.
-
-### Legitimate uses
-
-The same mechanisms support **bug reproduction**, **test harnesses**, **workflow instrumentation**, and **forensics or research on machines you own or are explicitly authorized to assess**. Capability overlaps with malware; **intent and authorization** do not.
-
-### What this repo does about it
-
-Optional, high-risk features (for example **`global_keylogger`** — see **§6**) are **off by default**, gated by **environment opt-in**, **human-in-the-loop** where applicable, **kill switch** / **dry-run**, and this document—because **failure modes** (misconfigured MCP server, lost laptop, accidental exposure) are serious, not because ethical security work is suspect.
-
-### Capability vs. intent
-
-**“Malware-adjacent” in capability** is fair; **“malware” in intent** is **not** implied. Use this project only under **law, policy, and consent** appropriate to your environment.
-
----
-
-## 1. Two-server model (recommended for new users)
-
-| Goal | Install |
-|------|---------|
-| **Drive the real Windows desktop** (your session) | **windows-computer-use-mcp** (this repo) + **`approve_automation`**, env limits below |
-| **Disposable desktop** (Windows Sandbox / VM) for installs + UI tests | **Also install `virtualization-mcp`** (fleet repo) — launches Sandbox, maps `assets`, optional dev stack / AIRGAP |
-
-**windows-computer-use-mcp does not spin up Windows Sandbox.** For **full** isolation, add **virtualization-mcp** to your MCP client and use it to **provision** the box; run **automation inside the guest** (script with pywinauto) or a test runner — see fleet doc below.
-
----
-
-## 2. Built into this server (host session)
-
-- **HITL (human-in-the-loop):** `approve_automation` — required for mutating **mouse** / **keyboard** (except `automation_mouse("position")` and `automation_mouse("telemetry")`).
-- **Env:** `WINDOWS_COMPUTER_USE_MCP_KILL_SWITCH`, `WINDOWS_COMPUTER_USE_MCP_MAX_ACTIONS_PER_MINUTE`, `WINDOWS_COMPUTER_USE_MCP_DRY_RUN`.
-- **HITL (human-in-the-loop) bypass:** `WINDOWS_COMPUTER_USE_MCP_BYPASS_HITL=1` allows zero-friction automation in trusted demo environments (disables security prompts).
-- **Pointer injection:** Mutating **mouse** paths use **`win32_mouse`** (`SetCursorPos` / `mouse_event`, DPI-aware), not PyAutoGUI alone. **Failsafe:** moving the cursor to the **upper-left screen corner** aborts injected pointer ops (same idea as PyAutoGUI). **`WINDOWS_COMPUTER_USE_MCP_BYPASS_HITL=1`** disables both **`pyautogui.FAILSAFE`** and that **`win32_mouse`** corner check for trusted demos/CI.
-- **Tool:** `automation_safety(status | reset_counters)`.
-- **Telemetry HUD:** The `automation_mouse("telemetry")` operation provides visible, on-screen verification of inputs. It is **non-persistent** and **localized** (volatile memory only), ensuring auditing without the security risks of background logging.
-
----
-
-## 3. Fleet documentation (canonical)
-
-Clone **mcp-central-docs** (sibling repo) and read:
-
-- **`patterns/WINDOWS_COMPUTER_USE_MCP_SAFETY.md`** — sampling amplification, IDE chain rules, **sandboxed execution** (guest-side pywinauto + virtualization-mcp).
-- **`patterns/FLEET_COMPUTER_USE_MCP.md`** — computer-use architecture, mitigations.
-
----
-
-## 4. Do not
-
-- Put **windows-computer-use-mcp** in the **default IDE chain** for **web-only** work — use a **browser MCP** (typically **Playwright**-based) for Vite/webapps and **HTML DOM** / network / console analysis. **pywinauto** targets **native Windows UI**, not the browser’s DOM; keep the split explicit.
-- Rely on **host** pywinauto to “click inside” the Windows Sandbox window — automate **inside** the guest session instead.
-- Treat **OpenManus** + **openmanus-mcp** + **pywinauto** + **OpenClaw / Manus-class** autonomy as **low-friction** — that combination **multiplies** tool loops and sampling against **OS-wide** input. See **mcp-central-docs** `patterns/WINDOWS_COMPUTER_USE_MCP_SAFETY.md` § *OpenManus, openmanus-mcp, OpenClaw, Manus-class* and **integrations/openmanus.md** (Caution block).
-
----
-
-## 5. Face recognition (`automation_face`) — opt-in, implemented, not default
-
-**Not a roadmap-only idea** — the tool is **implemented** but **excluded from default installs** (no `face` extra, env flag unset). It is for **local, operator-initiated** webcam capture on a machine you control — not ambient surveillance.
-
-The **`automation_face`** tool is **off by default**. Two steps are required to expose it:
-
-1. **Operator opt-in (runtime):** set **`WINDOWS_COMPUTER_USE_MCP_ENABLE_FACE=1`** (or `true` / `yes` / `on`) in the server environment (see **`automation_safety(status)`** — `face_tool_opt_in` / `WINDOWS_COMPUTER_USE_MCP_ENABLE_FACE`).
-2. **Dependencies:** install the **`face`** extra (see `pyproject.toml`) so the face-recognition stack is available.
-
-Without both, the server does **not** register **`automation_face`**; other portmanteau tools are unchanged.
-
-### Why it exists
-
-It supports **explicit, local** workflows for **operator presence** at the machine: e.g. a coarse “is the intended person still at this PC?” signal to **reduce casual misuse** if someone else sits down at an unlocked session while **desktop automation** is in play. That is **not** strong authentication, not a compliance control, and not a substitute for locking the screen — but it can **raise the bar** together with **HITL (human-in-the-loop)**, **env limits**, and normal session hygiene.
-
-### Communication
-
-Describe it as **optional**, **consent-based**, **local**, and **for operator presence** — not for covert or remote monitoring. Demos and README should state that clearly.
-
-### Treat face data as sensitive
-
-- Stored encodings and images under **`data/known_faces/`** (or paths your deployment uses) are **biometric-adjacent** — protect them like **secrets** (disk encryption, backups policy, who can read the repo).
-- **Do not** imply that face match **prevents** misuse; it only **narrows** risk when integrated thoughtfully.
-
-### If you do not need it
-
-Leave **`WINDOWS_COMPUTER_USE_MCP_ENABLE_FACE`** unset (default). Do not install the **`face`** extra. Window/element automation does not require face features.
-
-### Camera hardware (`capture`)
-
-**Supported:** Any camera that Windows exposes to **OpenCV** via `cv2.VideoCapture(index)` — typically:
-
-- **Integrated / built-in** laptop camera (often listed as *Integrated Camera* in Device Manager).
-- **Standard USB webcams** (UVC / “USB Video Device” class).
-
-Use the **`camera_index`** argument (`0`, `1`, …) to pick the device. Order is **OS enumeration order**, not a fixed name: the built-in cam is often **`0`** when it is the default; an external USB cam may be **`0`** or **`1`** depending on what Windows assigns. There is **no** special product name “usbcam” in the API — that is just informal shorthand some people use for **USB UVC** webcams.
-
-**Out of scope (for now):** **TP-Link Tapo**, other **IP / RTSP / cloud** cameras, and stacks that need vendor apps or RTSP URLs. Those are unnecessarily complicated for local **`capture`**; use a **built-in or USB UVC** camera instead.
-
----
-
-## 6. Global keyboard capture (`global_keylogger`) — opt-in, high-risk, non-stealth
-
-This project is **not** a hacker or spyware tool. **`global_keylogger`** exists only as an **explicit, operator-controlled** debugging aid — never enabled in default installs, never silent background exfiltration.
-
-The **`global_keylogger`** tool is **off by default**. The server does **not** register it unless you set:
-
-- **`WINDOWS_COMPUTER_USE_MCP_ENABLE_KEYLOGGER=1`** (or `true` / `yes` / `on`) in the server environment.
-
-See **`automation_safety(status)`** — `keylogger_tool_opt_in` / `WINDOWS_COMPUTER_USE_MCP_ENABLE_KEYLOGGER`.
-
-### Why it exists
-
-Session-wide keyboard hooks support **authorized** debugging and analysis (for example reproducing focus/shortcut issues, or **instrumenting** a session under explicit policy). This is **dual-use**: the same API surface is sensitive if misused.
-
-### How it is gated
-
-- **Opt-in** at registration time (env above).
-- **`approve_automation`** / **HITL (human-in-the-loop)** applies to **`start`** the same way as other sensitive input tools (unless **`WINDOWS_COMPUTER_USE_MCP_BYPASS_HITL`** is set — see §2).
-- **`WINDOWS_COMPUTER_USE_MCP_KILL_SWITCH=1`** blocks starting capture; **`WINDOWS_COMPUTER_USE_MCP_DRY_RUN=1`** simulates **`start`** without installing the hook.
-- The MCP server attempts to **stop** the listener on shutdown.
-- Events live in a **bounded in-memory ring buffer** exposed only via MCP `read` — not written to hidden log files.
-- **`start`** requires **HITL approval** (same as other sensitive input) unless bypass is explicitly set.
-
-### If you do not need it
-
-Leave **`WINDOWS_COMPUTER_USE_MCP_ENABLE_KEYLOGGER`** unset. Window, element, mouse, and normal keyboard **simulation** do not require it.

package/mcps/windows-computer-use-mcp/docs/TESTING.md

@@ -1,90 +0,0 @@
-# Testing (CI vs local)
-
-Aligned with **mcp-central-docs** → `standards/testing-environment-aware.md` (clone sibling repo **mcp-central-docs**).
-
-## Fleet context
-
-**windows-computer-use-mcp** is the **implemented** variant for **local Windows + OpenCV / USB cameras** (hardware-marked tests skip in CI). Central docs describe the same pattern and **fleet intent**: roll the drop-in scaffold into **all other repos** that touch **hardware** — IP or UVC **cameras**, **robots**, **scanners**, **vacuums**, **smart home / IoT**, and similar — shared **markers** and **hooks**, with **device-specific fixtures** per repo.
-
-## What this repo does
-
-| Location | Behavior |
-|----------|----------|
-| **GitHub Actions / `CI=true`** | No real Windows desktop, no OpenCV cameras. Tests marked `requires_hardware` or `destructive` are **skipped**. Prefer **mocks** and **HTTP** (`TestClient`) tests. |
-| **Local Windows** | You can run real desktop / camera probes. Opt in to heavy suites with env vars (e.g. `PYWINAUTO_TEST_REAL_WINDOWS=1` for Notepad window tests). |
-
-## Markers
-
-Defined in `tests/conftest_env.py` and `[tool.pytest.ini_options]` in `pyproject.toml`.
-
-- **`requires_hardware`** — Opens real OpenCV devices or host-only code paths. **Skipped in CI.**
-- **`destructive`** — Drives real UI (e.g. Notepad tests). **Skipped in CI**; locally also needs `PYWINAUTO_TEST_REAL_WINDOWS=1` where documented.
-- **`requires_network`** — LAN-dependent; skipped in CI.
-- **`ci_only`** — Runs when `CI=true`; skipped on typical local runs.
-
-## Commands
-
-```powershell
-# Same as CI: hardware tests skipped
-$env:CI = "true"; uv run pytest tests/ -q
-
-# Local: include hardware-marked tests (needs Windows + devices as applicable)
-uv run pytest tests/ -q
-
-# Only hardware-marked tests
-uv run pytest tests/ -m requires_hardware -q
-```
-
-## Camera API
-
-- **`tests/test_cameras_api.py`** — `test_cameras_get_returns_json` mocks `enumerate_cameras` (CI-safe).
-- **`test_enumerate_cameras_runs_on_local_host`** is marked **`requires_hardware`** (skipped in CI).
-
-## Web operator (`web_sota`)
-
-| Layer | Tool | Scope |
-|-------|------|--------|
-| Unit | **Vitest** | `src/**/*.test.ts` (e.g. `toolResult` parser) |
-| Browser e2e | **Playwright** | DOM for dashboard, `/targets`, HITL bar; hits live API via Vite proxy |
-| Native GUI | **pytest `-m e2e`** | LibreOffice / pywinauto loop (existing) |
-
-### Commands
-
-```powershell
-cd web_sota
-npm install
-npm run test              # Vitest
-npm run build             # required before e2e preview
-npm run test:e2e:install  # Chromium for Playwright
-npm run test:e2e          # starts backend + vite preview (10788/10789)
-```
-
-Playwright config: `web_sota/playwright.config.ts`. Stack launcher: `web_sota/scripts/start-e2e-stack.ps1`.
-
-**Do not** conflate Playwright (browser operator UI) with pytest LibreOffice e2e (native Windows automation).
-
-## Desktop (Tauri)
-
-Sidecar + installer build: `docs/DESKTOP_APP.md`.
-
-```powershell
-cd web_sota
-npm run sidecar:build
-npm run tauri:build
-```
-
-## CUA-NSIS smoke (installed app)
-
-Tests the actual NSIS installer artifact — catches backend unreachable, CSP/CORS, wrong paths, and orphan processes.
-
-`powershell
-just build-native
-just cua-nsis-test
-`
-
-The smoke script (scripts/cua-smoke.py) runs 9 phases: kill → install → launch → window verify → screenshot → feature route → diagnostics → WebView bridge OCR → uninstall.
-
-All phases are non-fatal except install, launch, and WebView bridge (release gate). The config file scripts/cua-nsis-config.json enables fleet reuse.
-
-See docs/ASSESSMENT_BY_CURSOR_2026-06-14_CUA_NSIS.md for full gap analysis.
-