use-computer-cli 0.1.1__tar.gz → 0.2.1__tar.gz

{use_computer_cli-0.1.1 → use_computer_cli-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: use-computer-cli
-Version: 0.1.1
+Version: 0.2.1
 Summary: Execute input on a screen for computer-use agents: move, click, drag, scroll, type, key, screenshot.
 Project-URL: Homepage, https://github.com/applica-software-guru/use-computer
 Author: Bruno Fortunato
@@ -50,19 +50,32 @@ Requires-Dist: ruff>=0.6; extra == 'dev'
 Provides-Extra: local
 Requires-Dist: mss>=10; extra == 'local'
 Requires-Dist: pynput>=1.8; extra == 'local'
+Provides-Extra: tree
+Requires-Dist: pyobjc-framework-applicationservices>=10; (sys_platform == 'darwin') and extra == 'tree'
+Requires-Dist: uiautomation>=2.0; (sys_platform == 'win32') and extra == 'tree'
 Provides-Extra: vnc
 Requires-Dist: vncdotool>=1.3; extra == 'vnc'
 Description-Content-Type: text/markdown
 
 # use-computer
 
-Executes input on a screen for computer-use agents: move, click, double-click, right-click,
-drag, scroll, type text, press key combinations, and capture a screenshot.
+Reads and acts on a screen for computer-use agents. It reads the **accessibility tree** the
+operating system already maintains -- roles, names, states and clickable boxes -- and it moves the
+mouse, clicks, drags, scrolls, types text, presses key combinations, and captures screenshots.
 
-`use-computer` is the **acting** half of a pair. [ui-locator](https://github.com/applica-software-guru/ui-locator)
-answers *where* the Invia button is and returns pixel coordinates; `use-computer` performs the
-click there. Both are driven by another AI agent through a CLI that emits JSON on stdout and
-diagnostics on stderr, with a Python API underneath.
+Every interaction takes the highest rung it can reach:
+
+1. **Element, through the platform API** -- the OS presses the button itself. No coordinates, so
+   nothing to aim and no scale to get wrong.
+2. **Element, by coordinate** -- the tree can see the control but exposes no way to operate it, so
+   `use-computer` clicks its centre and tells you it did.
+3. **Pixel, from vision** -- the tree cannot see it. Screenshot,
+   [ui-locator](https://github.com/applica-software-guru/ui-locator), click those pixels.
+
+Each rung is cheaper, faster and more accurate than the one below. Rung three is the floor the
+whole ladder stands on and is not going anywhere; it is simply no longer the only rung. Everything
+is driven by another AI agent through a CLI that emits JSON on stdout and diagnostics on stderr,
+with a Python API underneath.
 
 ## Install
 
@@ -70,9 +83,26 @@ diagnostics on stderr, with a Python API underneath.
 pip install use-computer-cli            # no backend
 pip install "use-computer-cli[local]"   # drive this machine's display (pynput + mss)
 pip install "use-computer-cli[vnc]"     # drive a remote framebuffer over RFB (vncdotool)
+pip install "use-computer-cli[tree]"    # read the accessibility tree (Windows, macOS)
+```
+
+Backends and the accessibility bindings are optional extras, imported lazily, so the package
+installs without them.
+
+**On Linux, do not use the `tree` extra.** PyGObject has no Linux wheel, so pip would build it from
+source and fail. The bindings are already on almost every desktop; the virtualenv just has to see
+them:
+
+```bash
+sudo apt install python3-gi gir1.2-atspi-2.0
+python3 -m venv --system-site-packages .venv
+.venv/bin/pip install "use-computer-cli[local]"
 ```
 
-Backends are optional extras, imported lazily, so the package installs without them.
+`gi` is a compiled extension built for one Python minor version -- Ubuntu 22.04 ships it for 3.10 --
+so the virtualenv has to be the distro's `python3`, not another minor version. Both packages report
+as installed either way, which is why the error message checks and says which case you are in.
+With uv: `uv venv --python /usr/bin/python3 --system-site-packages`.
 
 The distribution is `use-computer-cli` because `use-computer` is taken on PyPI by an unrelated
 project. The command it installs is `use-computer`, and the package it imports is `use_computer`.
@@ -80,16 +110,28 @@ project. The command it installs is `use-computer`, and the package it imports i
 ## Use
 
 ```bash
-use-computer click --x 120 --y 340 --use staging
+use-computer tree --use laptop                          # what is on screen, structurally
+use-computer click --role button --name "Invia"         # act on it by name
+use-computer set-value --role text --name "Email" --value "mario@example.com"
+use-computer click --x 120 --y 340 --use staging        # or by pixel, when the tree cannot see it
 use-computer type --text "hello" --use staging
 use-computer key ctrl+s --use staging
-use-computer screenshot --use laptop            # writes a file, returns its path
+use-computer screenshot --use laptop                    # writes a file, returns its path
 
 # a batch runs over one connection -- the default command, so `batch` may be omitted
-echo '[{"action":"click","x":120,"y":340},{"action":"key","combo":"enter"}]' \
-  | use-computer - --use staging --verify
+echo '[
+  {"action":"focus","role":"text","name":"Destinatario"},
+  {"action":"type","text":"mario@example.com"},
+  {"action":"click","role":"button","name":"Invia"},
+  {"action":"tree"}
+]' | use-computer - --use laptop
 ```
 
+An action names its target by coordinate **or** by element, never both. A selector that matches
+nothing comes back with a screenshot -- the signal to switch to vision. A selector that matches
+several comes back with the candidates, because two buttons named "OK" in two dialogs is the
+ordinary case and picking one silently fails a hundred runs later.
+
 stdout is one JSON object per run; every diagnostic goes to stderr. Exit codes: `0` success,
 `1` failure, `2` bad usage.
 
@@ -104,7 +146,8 @@ action and reports the path, so a verified action does not need a `screenshot` c
 - **Setup cost.** Opening a VNC connection dominates a single action, so one run performs a
   batch of actions over one connection.
 - **Blind actuation.** A click that lands on nothing looks exactly like a click that worked, so
-  `--verify` compares the screen before and after and reports whether it changed.
+  `--verify` compares the screen before and after and reports whether it changed -- and an action
+  that went through the accessibility API reports the element it actually operated.
 
 ## Configure
 

use_computer_cli-0.2.1/README.md

@@ -0,0 +1,137 @@
+# use-computer
+
+Reads and acts on a screen for computer-use agents. It reads the **accessibility tree** the
+operating system already maintains -- roles, names, states and clickable boxes -- and it moves the
+mouse, clicks, drags, scrolls, types text, presses key combinations, and captures screenshots.
+
+Every interaction takes the highest rung it can reach:
+
+1. **Element, through the platform API** -- the OS presses the button itself. No coordinates, so
+   nothing to aim and no scale to get wrong.
+2. **Element, by coordinate** -- the tree can see the control but exposes no way to operate it, so
+   `use-computer` clicks its centre and tells you it did.
+3. **Pixel, from vision** -- the tree cannot see it. Screenshot,
+   [ui-locator](https://github.com/applica-software-guru/ui-locator), click those pixels.
+
+Each rung is cheaper, faster and more accurate than the one below. Rung three is the floor the
+whole ladder stands on and is not going anywhere; it is simply no longer the only rung. Everything
+is driven by another AI agent through a CLI that emits JSON on stdout and diagnostics on stderr,
+with a Python API underneath.
+
+## Install
+
+```bash
+pip install use-computer-cli            # no backend
+pip install "use-computer-cli[local]"   # drive this machine's display (pynput + mss)
+pip install "use-computer-cli[vnc]"     # drive a remote framebuffer over RFB (vncdotool)
+pip install "use-computer-cli[tree]"    # read the accessibility tree (Windows, macOS)
+```
+
+Backends and the accessibility bindings are optional extras, imported lazily, so the package
+installs without them.
+
+**On Linux, do not use the `tree` extra.** PyGObject has no Linux wheel, so pip would build it from
+source and fail. The bindings are already on almost every desktop; the virtualenv just has to see
+them:
+
+```bash
+sudo apt install python3-gi gir1.2-atspi-2.0
+python3 -m venv --system-site-packages .venv
+.venv/bin/pip install "use-computer-cli[local]"
+```
+
+`gi` is a compiled extension built for one Python minor version -- Ubuntu 22.04 ships it for 3.10 --
+so the virtualenv has to be the distro's `python3`, not another minor version. Both packages report
+as installed either way, which is why the error message checks and says which case you are in.
+With uv: `uv venv --python /usr/bin/python3 --system-site-packages`.
+
+The distribution is `use-computer-cli` because `use-computer` is taken on PyPI by an unrelated
+project. The command it installs is `use-computer`, and the package it imports is `use_computer`.
+
+## Use
+
+```bash
+use-computer tree --use laptop                          # what is on screen, structurally
+use-computer click --role button --name "Invia"         # act on it by name
+use-computer set-value --role text --name "Email" --value "mario@example.com"
+use-computer click --x 120 --y 340 --use staging        # or by pixel, when the tree cannot see it
+use-computer type --text "hello" --use staging
+use-computer key ctrl+s --use staging
+use-computer screenshot --use laptop                    # writes a file, returns its path
+
+# a batch runs over one connection -- the default command, so `batch` may be omitted
+echo '[
+  {"action":"focus","role":"text","name":"Destinatario"},
+  {"action":"type","text":"mario@example.com"},
+  {"action":"click","role":"button","name":"Invia"},
+  {"action":"tree"}
+]' | use-computer - --use laptop
+```
+
+An action names its target by coordinate **or** by element, never both. A selector that matches
+nothing comes back with a screenshot -- the signal to switch to vision. A selector that matches
+several comes back with the candidates, because two buttons named "OK" in two dialogs is the
+ordinary case and picking one silently fails a hundred runs later.
+
+stdout is one JSON object per run; every diagnostic goes to stderr. Exit codes: `0` success,
+`1` failure, `2` bad usage.
+
+Screenshots are files, never bytes in the JSON. `--verify` writes the screen it captured after the
+action and reports the path, so a verified action does not need a `screenshot` call after it.
+
+## Three problems it solves
+
+- **Coordinate spaces.** A screenshot on a HiDPI display is larger than the space the OS clicks
+  in. Every coordinate carries its space, `use-computer` scales between them, and it refuses to
+  guess when the ratio is unknown.
+- **Setup cost.** Opening a VNC connection dominates a single action, so one run performs a
+  batch of actions over one connection.
+- **Blind actuation.** A click that lands on nothing looks exactly like a click that worked, so
+  `--verify` compares the screen before and after and reports whether it changed -- and an action
+  that went through the accessibility API reports the element it actually operated.
+
+## Configure
+
+```bash
+use-computer config init                                  # asks, then proves it works
+use-computer config init --backend vnc --host 10.0.0.5    # doesn't ask
+use-computer config init --backend local --allow-local
+```
+
+`config init` writes the file below, then opens the backend it just configured and reports the
+screen geometry and scale — so a coordinate space whose ratio cannot be derived surfaces at setup
+rather than at the first click that lands in the wrong place.
+
+It writes `.use-computer/config.toml` at the project root (found by walking up, the way git finds
+its own):
+
+```toml
+default-profile = "laptop"
+delay = 0.1
+
+[profiles.laptop]
+backend = "local"
+allow-local = true
+
+[profiles.staging]
+backend = "vnc"
+host = "10.0.0.5"
+port = 5900
+```
+
+Secrets go in `.use-computer/.env`, which is not committed. `use-computer config show` prints
+every resolved value, the layer it came from and the variable that would override it.
+
+## The agent skill
+
+Instructions for the calling agent ship inside the package and are installed from it, so they
+always match the installed version:
+
+```bash
+use-computer skill install --scope project
+```
+
+## Documentation
+
+This package is developed with [SDD](https://github.com/applica-software-guru/sdd). The specs
+it implements live in `product/` and `system/` at the repository root.

{use_computer_cli-0.1.1 → use_computer_cli-0.2.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "use-computer-cli"
-version = "0.1.1"
+version = "0.2.1"
 description = "Execute input on a screen for computer-use agents: move, click, drag, scroll, type, key, screenshot."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -38,6 +38,23 @@ dependencies = [
 # pyautogui is deliberately not used: its last release (0.9.54) dates from 2023.
 local = ["pynput>=1.8", "mss>=10"]
 vnc = ["vncdotool>=1.3"]
+# Reading and operating the UI tree. One binding per platform, behind a marker.
+#
+# Linux needs a system package as well: there is no pyatspi on PyPI, so the AT-SPI bindings come
+# from gir1.2-atspi-2.0 (plus python3-pyatspi on Debian/Ubuntu) and PyGObject reaches them through
+# GObject Introspection. UITreeUnavailableError names both halves.
+#
+# Not dogtail (GPLv2 -- fine to run, wrong to import from a permissively licensed library) and not
+# atomacos (last release 3.3.0, May 2021 -- the same staleness argument that ruled out pyautogui).
+# Linux is deliberately absent. PyGObject has no Linux wheel, so listing it here made
+# `pip install "use-computer-cli[tree]"` build from source, need pycairo and system headers, and
+# fail -- leaving the user with no CLI at all rather than a CLI missing one capability. The AT-SPI
+# bindings come from the distro (python3-gi, gir1.2-atspi-2.0) and are reached with a
+# --system-site-packages virtualenv; the error message says so when they are absent.
+tree = [
+    "uiautomation>=2.0; sys_platform == 'win32'",
+    "pyobjc-framework-ApplicationServices>=10; sys_platform == 'darwin'",
+]
 dev = [
     "pytest>=8",
     "pytest-mock>=3.12",
@@ -78,7 +95,16 @@ strict = true
 files = ["src", "tests"]
 
 [[tool.mypy.overrides]]
-module = ["pynput.*", "mss.*", "vncdotool.*", "ApplicationServices.*", "Quartz.*", "tomli"]
+module = [
+    "pynput.*",
+    "mss.*",
+    "vncdotool.*",
+    "ApplicationServices.*",
+    "Quartz.*",
+    "gi.*",
+    "uiautomation.*",
+    "tomli",
+]
 ignore_missing_imports = true
 
 [tool.pytest.ini_options]

{use_computer_cli-0.1.1 → use_computer_cli-0.2.1}/src/use_computer/__init__.py

@@ -7,11 +7,15 @@ from __future__ import annotations
 
 from importlib.metadata import PackageNotFoundError, version
 
+from use_computer.accessibility import AccessibilityProvider, create_provider
 from use_computer.actions import (
     Action,
     ClickAction,
+    CollapseAction,
     DoubleClickAction,
     DragAction,
+    ExpandAction,
+    FocusAction,
     KeyAction,
     MouseButton,
     MoveAction,
@@ -19,7 +23,13 @@ from use_computer.actions import (
     ScreenshotAction,
     ScrollAction,
     ScrollDirection,
+    SelectAction,
+    SetValueAction,
+    ShowMenuAction,
+    ToggleAction,
+    TreeAction,
     TypeAction,
+    WindowsAction,
 )
 from use_computer.backends import Backend, create_backend
 from use_computer.compare import ChangeReport, Screenshot, compare
@@ -28,15 +38,31 @@ from use_computer.config import load as load_config
 from use_computer.coordinates import Coordinate, CoordinateSpace, ScreenInfo, convert
 from use_computer.errors import (
     ActionFailedError,
+    ActionNotSupportedError,
+    AmbiguousNodeError,
     BackendNotAvailableError,
     ConfigError,
     CoordinateSpaceError,
     KeySyntaxError,
+    NodeNotFoundError,
     PermissionDeniedError,
+    UITreeUnavailableError,
     UseComputerError,
 )
 from use_computer.keys import KeyCombo, parse_combo
 from use_computer.runner import ActionResult, ErrorInfo, RunResult, Session, run_actions
+from use_computer.tree import (
+    Box,
+    NodeSelector,
+    OutputFormat,
+    TreeReason,
+    TreeResult,
+    TreeScope,
+    UINode,
+    Via,
+    WindowInfo,
+    WindowsResult,
+)
 
 #: The distribution name on PyPI, which differs from the import package: `use-computer` was
 #: already taken there. importlib.metadata is keyed by the distribution, so this is the name
@@ -50,14 +76,19 @@ except PackageNotFoundError:  # pragma: no cover - source checkout without an in
 
 __all__ = [
     "DISTRIBUTION",
+    "AccessibilityProvider",
     "Action",
     "ActionFailedError",
+    "ActionNotSupportedError",
     "ActionResult",
+    "AmbiguousNodeError",
     "Backend",
     "BackendNotAvailableError",
     "BackendProfile",
+    "Box",
     "ChangeReport",
     "ClickAction",
+    "CollapseAction",
     "ConfigError",
     "Coordinate",
     "CoordinateSpace",
@@ -65,11 +96,16 @@ __all__ = [
     "DoubleClickAction",
     "DragAction",
     "ErrorInfo",
+    "ExpandAction",
+    "FocusAction",
     "KeyAction",
     "KeyCombo",
     "KeySyntaxError",
     "MouseButton",
     "MoveAction",
+    "NodeNotFoundError",
+    "NodeSelector",
+    "OutputFormat",
     "PermissionDeniedError",
     "ResolvedConfig",
     "RightClickAction",
@@ -79,14 +115,29 @@ __all__ = [
     "ScreenshotAction",
     "ScrollAction",
     "ScrollDirection",
+    "SelectAction",
     "Session",
+    "SetValueAction",
     "Settings",
+    "ShowMenuAction",
+    "ToggleAction",
+    "TreeAction",
+    "TreeReason",
+    "TreeResult",
+    "TreeScope",
     "TypeAction",
+    "UINode",
+    "UITreeUnavailableError",
     "UseComputerError",
+    "Via",
+    "WindowInfo",
+    "WindowsAction",
+    "WindowsResult",
     "__version__",
     "compare",
     "convert",
     "create_backend",
+    "create_provider",
     "load_config",
     "parse_combo",
     "run_actions",

use_computer_cli-0.2.1/src/use_computer/accessibility/__init__.py

@@ -0,0 +1,51 @@
+"""Choosing an accessibility provider.
+
+By the *running platform*, never by configuration: a profile does not get to claim macOS
+accessibility on Linux. And only for a backend that can have one at all -- RFB carries pixels,
+so a vnc profile gets a clear error and the calling agent uses coordinates.
+"""
+
+from __future__ import annotations
+
+import sys
+
+from use_computer.accessibility.base import AccessibilityProvider, require
+from use_computer.errors import UITreeUnavailableError
+
+#: Backends that drive a display this machine can also inspect. Everything else is remote
+#: pixels, and no accessibility API reaches across RFB.
+LOCAL_BACKENDS = frozenset({"local"})
+
+
+def create_provider(backend: str) -> AccessibilityProvider:
+    """Build the provider for this platform, or say why there cannot be one.
+
+    Raises:
+        UITreeUnavailableError: wrong backend, unsupported platform, or a missing binding.
+        PermissionDeniedError: the OS refused the accessibility permission.
+    """
+    if backend not in LOCAL_BACKENDS:
+        raise UITreeUnavailableError(
+            f"the {backend!r} backend drives a remote framebuffer, which carries pixels and no "
+            "accessibility information. Use a screenshot and coordinates instead."
+        )
+
+    if sys.platform.startswith("linux"):
+        from use_computer.accessibility.atspi import AtspiProvider  # noqa: PLC0415
+
+        return AtspiProvider()
+    if sys.platform == "win32":
+        from use_computer.accessibility.uia import UiaProvider  # noqa: PLC0415
+
+        return UiaProvider()
+    if sys.platform == "darwin":
+        from use_computer.accessibility.ax import AxProvider  # noqa: PLC0415
+
+        return AxProvider()
+
+    raise UITreeUnavailableError(
+        f"no accessibility provider for platform {sys.platform!r}."
+    )
+
+
+__all__ = ["AccessibilityProvider", "UITreeUnavailableError", "create_provider", "require"]