macos-harness 0.1.0__py3-none-any.whl

macos_harness/SKILL.md

@@ -0,0 +1,96 @@
+---
+name: macos-harness
+description: Control a whole Mac from one persistent Python session with screenshots, PID-targeted input, an animated virtual pointer, targeted Apple Accessibility, Apple Events, Browser Harness CDP, and filesystem access. Use for native, Electron, browser, dialog, file, or cross-app tasks without moving the physical cursor or forcing apps into the foreground.
+---
+
+# macOS Harness
+
+Use one CLI call per decision point, not per primitive:
+
+```bash
+macos-harness <<'PY'
+app = "Spotify"
+mac.see(app)
+mac.key("cmd+k", app=app)
+mac.type("Alessia Cara", app=app)
+print(mac.see(app))
+PY
+```
+
+The CLI preloads `mac`, `browser`, `Path`, and `subprocess`. Prefer bounded stdin
+programs; reserve `macos-harness repl` for manual exploration and always exit it.
+
+## Minimize round trips
+
+- Bundle deterministic, reversible steps into one program, then verify once. Opening
+  search, typing a query, and capturing the results is one burst—not three calls.
+- Stop at a genuine decision boundary: ambiguous identity, new coordinates, an
+  irreversible action, or unexpected state. Inspect once, then run the next burst.
+- Do not screenshot merely to confirm that a known shortcut opened a text field
+  before typing. Let the final screenshot verify the whole sequence.
+- Poll exact AX or Apple Events state inside the same Python program when possible;
+  do not make the LLM repeatedly ask whether a transition finished.
+- Use the cheapest strong end-state check. Prefer one screenshot for visible state
+  or one exact API/AX query for semantic state; use both only when they prove
+  different things.
+
+## Use the small surface
+
+Think in six verbs: `see`, `key`, `type`, `click`, `ax`, `script`.
+
+```python
+frame = mac.see("Spotify")
+mac.key("cmd+k", app="Spotify")
+mac.type("Alessia Cara", app="Spotify")
+mac.click(640, 420, app="Spotify")
+
+item = mac.ax.at(640, 420, app="Spotify")
+mac.ax.perform(item["element_index"], "AXPress")
+
+mac.script('tell application "Spotify" to play')
+```
+
+Use ordinary Python for local context and one-off logic. Do not add app-specific
+helpers when a short program can resolve the task.
+
+## Choose the lowest useful mode
+
+1. When identity depends on local context (`my`, `friend`, or prior activity),
+   inspect that context and correlate stable fields; a loose text hit is not enough.
+2. Use `mac.script()` for a known exact, focus-safe app command.
+3. Otherwise use `mac.see(app)` and vision.
+4. Prefer a known keyboard route; use a verified coordinate for a visible,
+   low-risk target.
+5. Use targeted `mac.ax` only when semantic identity or state matters. Do not dump
+   a full AX tree before trying the direct route.
+
+After a failed verified burst, switch mode or stop. Never repair uncertainty with
+repeated keys, clicks, deletion loops, or bulk input.
+
+## Keep the invariants
+
+- Input targets an already-running app PID and never requests activation or raise.
+- A background target becoming frontmost raises `FocusChangedError`; never
+  manipulate focus to restore it.
+- `mac.click()` is raw PID-targeted input. It never guesses an AX action.
+- The animated pointer is click-through and never moves the physical cursor.
+- `mac.move()` moves only that pointer; it cannot produce native hover.
+- Inactive apps may reject raw clicks. After one verified failure, switch mode.
+- Never launch a closed app or use a custom URL scheme when focus is forbidden.
+- Screenshot coordinates come from the latest `mac.see()` and preserve window
+  bounds and Retina scaling.
+
+Secondary primitives are `mac.move`, `drag`, `scroll`, `show_pointer`, and
+`hide_pointer`. `mac.ax.query()` returns compact matches and bounds fallback
+traversal; lower `max_nodes` for especially large apps.
+
+## Browser and permissions
+
+Use `browser` for DOM, tabs, network, downloads, and uploads. Do not substitute AX
+for CDP inside a web page. While Browser Harness connects, macOS Harness accepts
+Chrome's exact `Allow remote debugging?` sheet through system-wide AX. It never
+activates Chrome or emits a mouse event.
+
+Run `macos-harness doctor` to inspect permissions without prompting. Run
+`macos-harness doctor --request` only with user approval. Accessibility, screen
+recording, and event posting are global; Apple Events Automation is per target.

macos_harness/__init__.py

@@ -0,0 +1,12 @@
+"""Public Python interface for macOS Harness."""
+
+from .browser import BrowserHarness
+from .macos import AccessibilityPermissionError, FocusChangedError, MacOS, MacOSError
+
+__all__ = [
+    "AccessibilityPermissionError",
+    "BrowserHarness",
+    "FocusChangedError",
+    "MacOS",
+    "MacOSError",
+]

macos_harness/browser.py

@@ -0,0 +1,168 @@
+"""Lazy bridge to the Browser Harness Python SDK."""
+
+from __future__ import annotations
+
+import os
+import platform
+import subprocess
+import sys
+import threading
+import time
+from types import ModuleType
+from typing import Any
+
+
+def _approve_chrome_prompt(stop: threading.Event, timeout: float) -> None:
+    """Press Chrome's exact remote-debugging Allow button through system-wide AX."""
+    if platform.system() != "Darwin":
+        return
+    try:
+        import ApplicationServices as AS
+    except ImportError:  # pragma: no cover - macOS dependency failure
+        return
+
+    deadline = time.monotonic() + timeout
+    while not stop.is_set() and time.monotonic() < deadline:
+        root = AS.AXUIElementCreateSystemWide()
+        pressed = False
+        windows = AS.CGWindowListCopyWindowInfo(
+            AS.kCGWindowListOptionOnScreenOnly, AS.kCGNullWindowID
+        )
+        for window in windows:
+            if window.get("kCGWindowOwnerName") != "Google Chrome":
+                continue
+            if window.get("kCGWindowName") != "Allow remote debugging?":
+                continue
+            bounds = window.get("kCGWindowBounds") or {}
+            left = float(bounds.get("X", 0))
+            top = float(bounds.get("Y", 0))
+            width = float(bounds.get("Width", 0))
+            height = float(bounds.get("Height", 0))
+            points = [(left + width - 58, top + height - 38)]
+            points.extend(
+                (x, y)
+                for y in range(int(top + height - 16), int(top + height - 96), -16)
+                for x in range(int(left + width - 16), int(left), -16)
+            )
+            for x, y in points:
+                error, element = AS.AXUIElementCopyElementAtPosition(root, x, y, None)
+                if error != 0 or element is None:
+                    continue
+                role_error, role = AS.AXUIElementCopyAttributeValue(
+                    element, "AXRole", None
+                )
+                description_error, description = AS.AXUIElementCopyAttributeValue(
+                    element, "AXDescription", None
+                )
+                if (
+                    role_error == 0
+                    and description_error == 0
+                    and str(role) == "AXButton"
+                    and str(description) == "Allow"
+                ):
+                    AS.AXUIElementPerformAction(element, "AXPress")
+                    pressed = True
+                    break
+            if pressed:
+                break
+        stop.wait(0.2 if pressed else 0.05)
+
+
+def _watch_chrome_prompts(timeout: float) -> None:
+    """Start AX only after Chrome creates its protected permission window."""
+    if platform.system() != "Darwin":
+        return
+    try:
+        import ApplicationServices as AS
+    except ImportError:  # pragma: no cover - macOS dependency failure
+        return
+
+    source = (
+        "import threading; "
+        "from macos_harness.browser import _approve_chrome_prompt; "
+        "_approve_chrome_prompt(threading.Event(), 0.8)"
+    )
+    deadline = time.monotonic() + timeout
+    while time.monotonic() < deadline:
+        windows = AS.CGWindowListCopyWindowInfo(
+            AS.kCGWindowListOptionOnScreenOnly, AS.kCGNullWindowID
+        )
+        prompt_exists = any(
+            str(window.get("kCGWindowOwnerName") or "") == "Google Chrome"
+            and str(window.get("kCGWindowName") or "") == "Allow remote debugging?"
+            for window in windows
+        )
+        if prompt_exists:
+            subprocess.run(
+                [sys.executable, "-c", source],
+                stdout=subprocess.DEVNULL,
+                stderr=subprocess.DEVNULL,
+                timeout=2,
+                check=False,
+            )
+        time.sleep(0.05)
+
+
+class BrowserHarness:
+    """Expose Browser Harness helpers without connecting until first use."""
+
+    def __init__(self, *, name: str | None = None, wait: float = 60.0) -> None:
+        self.name = name
+        self.wait = wait
+        self._helpers: ModuleType | None = None
+
+    def connect(self) -> BrowserHarness:
+        if self._helpers is not None:
+            return self
+
+        if self.name:
+            os.environ["BU_NAME"] = self.name
+
+        try:
+            from browser_harness import admin, helpers
+        except ImportError as exc:  # pragma: no cover - packaging failure
+            raise RuntimeError(
+                "Browser Harness is unavailable. Install the project dependencies "
+                "with `uv sync`."
+            ) from exc
+
+        approver = subprocess.Popen(
+            [
+                sys.executable,
+                "-c",
+                (
+                    "from macos_harness.browser import _watch_chrome_prompts; "
+                    f"_watch_chrome_prompts({self.wait!r})"
+                ),
+            ],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+        )
+        try:
+            admin.ensure_daemon(wait=self.wait, name=self.name)
+        finally:
+            if approver.poll() is None:
+                approver.terminate()
+                try:
+                    approver.wait(timeout=0.2)
+                except subprocess.TimeoutExpired:
+                    approver.kill()
+        if self.name:
+            helpers.NAME = self.name
+        self._helpers = helpers
+        return self
+
+    def __getattr__(self, name: str) -> Any:
+        if name.startswith("_"):
+            raise AttributeError(name)
+        self.connect()
+        assert self._helpers is not None
+        try:
+            return getattr(self._helpers, name)
+        except AttributeError as exc:
+            raise AttributeError(f"Browser Harness has no helper {name!r}") from exc
+
+    def __repr__(self) -> str:
+        status = "connected" if self._helpers is not None else "lazy"
+        suffix = f", name={self.name!r}" if self.name else ""
+        return f"BrowserHarness({status}{suffix})"

macos_harness/cli.py

@@ -0,0 +1,158 @@
+"""One Python execution surface for the browser, macOS, and local files."""
+
+from __future__ import annotations
+
+import argparse
+import code
+import json
+import subprocess
+import sys
+import time
+from importlib import resources
+from pathlib import Path
+from typing import Any
+
+from .browser import BrowserHarness
+from .macos import MacOS, MacOSError
+from .telemetry import capture_cli
+from .telemetry import run_cli as run_telemetry_cli
+
+
+def _namespace() -> dict[str, Any]:
+    return {
+        "__name__": "__macos_harness__",
+        "browser": BrowserHarness(),
+        "mac": MacOS(),
+        "Path": Path,
+        "subprocess": subprocess,
+    }
+
+
+def _execute(code: str) -> int:
+    if not code.strip():
+        print("No Python code received on stdin", file=sys.stderr)
+        return 2
+    namespace = _namespace()
+    exec(compile(code, "<macos-harness>", "exec"), namespace, namespace)  # noqa: S102
+    return 0
+
+
+def _skill_text() -> str:
+    bundled = resources.files("macos_harness").joinpath("SKILL.md")
+    if bundled.is_file():
+        return bundled.read_text(encoding="utf-8")
+    checkout = Path(__file__).resolve().parents[2] / "skills/macos-harness/SKILL.md"
+    return checkout.read_text(encoding="utf-8")
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="macos-harness",
+        description="Execute Python with browser, macOS, and filesystem access.",
+        epilog=(
+            "Typical usage:\n  macos-harness <<'PY'\n  print(mac.see('Spotify'))\n  PY"
+        ),
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    subparsers = parser.add_subparsers(dest="command")
+    doctor = subparsers.add_parser("doctor", help="check macOS permissions and runtime")
+    doctor.add_argument(
+        "--request", action="store_true", help="request missing global permissions"
+    )
+    subparsers.add_parser("apps", help="list running macOS applications")
+    subparsers.add_parser("repl", help="start a persistent interactive Python session")
+    subparsers.add_parser("skill", help="print the macOS Harness skill")
+    telemetry = subparsers.add_parser(
+        "telemetry", help="inspect or change anonymous telemetry"
+    )
+    telemetry.add_argument("action", nargs="?", choices=("status", "enable", "disable"))
+    see = subparsers.add_parser("see", help="capture a bounded application window")
+    see.add_argument("app")
+    see.add_argument("--max-width", type=int, default=1280)
+    see.add_argument("--max-height", type=int, default=1280)
+    see.add_argument("--no-pointer", action="store_true")
+    state = subparsers.add_parser(
+        "state", help="print an application's AX state as JSON"
+    )
+    state.add_argument("app")
+    state.add_argument("--screenshot", action="store_true")
+    state.add_argument("--max-depth", type=int, default=25)
+    state.add_argument("--max-nodes", type=int, default=5000)
+    state.add_argument("--include-menu-bar", action="store_true")
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = _build_parser()
+    args = parser.parse_args(argv)
+    if args.command == "telemetry":
+        return run_telemetry_cli([args.action] if args.action else [])
+
+    started = time.monotonic()
+    result: int | None = None
+    try:
+        if args.command == "doctor":
+            mac = MacOS()
+            if args.request:
+                mac.request_permissions()
+            print(json.dumps(mac.doctor(), indent=2))
+            result = 0
+            return result
+        if args.command == "apps":
+            print(json.dumps(MacOS().list_apps(), indent=2))
+            result = 0
+            return result
+        if args.command == "skill":
+            print(_skill_text(), end="")
+            result = 0
+            return result
+        if args.command == "repl":
+            code.interact(
+                banner=(
+                    "macos-harness: mac.see/key/type/click/ax/script, browser, "
+                    "Path, and subprocess are ready"
+                ),
+                local=_namespace(),
+                exitmsg="",
+            )
+            result = 0
+            return result
+        if args.command == "see":
+            result = MacOS().see(
+                args.app,
+                max_width=args.max_width,
+                max_height=args.max_height,
+                show_pointer=not args.no_pointer,
+            )
+            print(json.dumps(result, indent=2, ensure_ascii=False))
+            result = 0
+            return result
+        if args.command == "state":
+            state = MacOS().get_app_state(
+                args.app,
+                screenshot=args.screenshot,
+                max_depth=args.max_depth,
+                max_nodes=args.max_nodes,
+                include_menu_bar=args.include_menu_bar,
+            )
+            print(json.dumps(state, indent=2, ensure_ascii=False))
+            result = 0
+            return result
+        if args.command is None:
+            if sys.stdin.isatty():
+                parser.print_help()
+                result = 2
+                return result
+            result = _execute(sys.stdin.read())
+            return result
+        parser.error(f"unknown command: {args.command}")
+    except (MacOSError, RuntimeError) as exc:
+        print(f"macos-harness: {exc}", file=sys.stderr)
+        result = 1
+        return result
+    finally:
+        capture_cli(args.command or "python", result == 0, time.monotonic() - started)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

macos_harness/controls.py

@@ -0,0 +1,119 @@
+"""The explicit Accessibility escape hatch for the macOS harness."""
+
+from __future__ import annotations
+
+from collections.abc import Iterable
+from typing import TYPE_CHECKING, Any
+
+import ApplicationServices as AS
+
+if TYPE_CHECKING:
+    from .macos import MacOS
+
+
+class Accessibility:
+    """Thin, opt-in access to Apple's AXUIElement operations."""
+
+    _COMPACT_ATTRIBUTES = (
+        "AXRole",
+        "AXTitle",
+        "AXDescription",
+        "AXValue",
+        "AXFrame",
+    )
+
+    def __init__(self, host: MacOS) -> None:
+        self._host = host
+
+    def at(
+        self,
+        x: float,
+        y: float,
+        *,
+        app: str | None = None,
+        coordinate_space: str = "screenshot",
+        attributes: Iterable[str] = _COMPACT_ATTRIBUTES,
+        include_actions: bool = True,
+    ) -> dict[str, Any]:
+        self._host._ensure_accessibility()
+        point = self._host._screen_point(x, y, coordinate_space)
+        pid = self._host._pid(app)
+        root = (
+            self._host._application_element(pid)
+            if pid is not None
+            else AS.AXUIElementCreateSystemWide()
+        )
+        error, element = AS.AXUIElementCopyElementAtPosition(
+            root, point[0], point[1], None
+        )
+        if error != 0 or element is None:
+            raise RuntimeError(f"AX hit test failed with AXError {error}")
+        index = self._host._remember_element(element)
+        return self._host._describe_element(
+            element,
+            index,
+            attributes=attributes,
+            include_actions=include_actions,
+        )
+
+    def query(
+        self,
+        *,
+        app: str | None = None,
+        element_index: int | None = None,
+        search_key: str = "AXAnyTypeSearchKey",
+        text: str | None = None,
+        visible_only: bool = True,
+        limit: int = 20,
+        direction: str = "next",
+        immediate_descendants_only: bool = False,
+        attributes: Iterable[str] = _COMPACT_ATTRIBUTES,
+        include_actions: bool = False,
+        max_nodes: int = 500,
+    ) -> list[dict[str, Any]]:
+        return self._host.ax_search(
+            element_index=element_index,
+            app=app,
+            search_key=search_key,
+            text=text,
+            visible_only=visible_only,
+            limit=limit,
+            direction=direction,
+            immediate_descendants_only=immediate_descendants_only,
+            attributes=attributes,
+            include_actions=include_actions,
+            max_nodes=max_nodes,
+        )
+
+    def get(
+        self, element_index: int, attributes: str | Iterable[str] = "AXValue"
+    ) -> Any:
+        if isinstance(attributes, str):
+            return self._host.get(element_index, attributes)
+        return self._host.get_attributes(element_index, attributes)
+
+    def set(self, element_index: int, attribute: str, value: Any) -> None:
+        self._host.set(element_index, value, attribute)
+
+    def actions(self, element_index: int) -> list[str]:
+        return self._host._actions(self._host._element(element_index))
+
+    def perform(self, element_index: int, action: str = "AXPress") -> None:
+        self._host.perform_action(element_index, action)
+
+    def parameterized(self, element_index: int, attribute: str, parameter: Any) -> Any:
+        error, value = AS.AXUIElementCopyParameterizedAttributeValue(
+            self._host._element(element_index), attribute, parameter, None
+        )
+        if error != 0:
+            raise RuntimeError(
+                f"Parameterized AX read {attribute!r} failed with AXError {error}"
+            )
+        return self._host._jsonable(value)
+
+    def raw(self, element_index: int) -> Any:
+        return self._host._element(element_index)
+
+    def dump(self, app: str, **kwargs: Any) -> dict[str, Any]:
+        kwargs.setdefault("screenshot", False)
+        return self._host.get_app_state(app, **kwargs)