use-computer-cli 0.1.1__tar.gz → 0.2.0__tar.gz

{use_computer_cli-0.1.1 → use_computer_cli-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: use-computer-cli
-Version: 0.1.1
+Version: 0.2.0
 Summary: Execute input on a screen for computer-use agents: move, click, drag, scroll, type, key, screenshot.
 Project-URL: Homepage, https://github.com/applica-software-guru/use-computer
 Author: Bruno Fortunato
@@ -50,19 +50,33 @@ Requires-Dist: ruff>=0.6; extra == 'dev'
 Provides-Extra: local
 Requires-Dist: mss>=10; extra == 'local'
 Requires-Dist: pynput>=1.8; extra == 'local'
+Provides-Extra: tree
+Requires-Dist: pygobject>=3.46; (sys_platform == 'linux') and extra == 'tree'
+Requires-Dist: pyobjc-framework-applicationservices>=10; (sys_platform == 'darwin') and extra == 'tree'
+Requires-Dist: uiautomation>=2.0; (sys_platform == 'win32') and extra == 'tree'
 Provides-Extra: vnc
 Requires-Dist: vncdotool>=1.3; extra == 'vnc'
 Description-Content-Type: text/markdown
 
 # use-computer
 
-Executes input on a screen for computer-use agents: move, click, double-click, right-click,
-drag, scroll, type text, press key combinations, and capture a screenshot.
+Reads and acts on a screen for computer-use agents. It reads the **accessibility tree** the
+operating system already maintains -- roles, names, states and clickable boxes -- and it moves the
+mouse, clicks, drags, scrolls, types text, presses key combinations, and captures screenshots.
 
-`use-computer` is the **acting** half of a pair. [ui-locator](https://github.com/applica-software-guru/ui-locator)
-answers *where* the Invia button is and returns pixel coordinates; `use-computer` performs the
-click there. Both are driven by another AI agent through a CLI that emits JSON on stdout and
-diagnostics on stderr, with a Python API underneath.
+Every interaction takes the highest rung it can reach:
+
+1. **Element, through the platform API** -- the OS presses the button itself. No coordinates, so
+   nothing to aim and no scale to get wrong.
+2. **Element, by coordinate** -- the tree can see the control but exposes no way to operate it, so
+   `use-computer` clicks its centre and tells you it did.
+3. **Pixel, from vision** -- the tree cannot see it. Screenshot,
+   [ui-locator](https://github.com/applica-software-guru/ui-locator), click those pixels.
+
+Each rung is cheaper, faster and more accurate than the one below. Rung three is the floor the
+whole ladder stands on and is not going anywhere; it is simply no longer the only rung. Everything
+is driven by another AI agent through a CLI that emits JSON on stdout and diagnostics on stderr,
+with a Python API underneath.
 
 ## Install
 
@@ -70,9 +84,16 @@ diagnostics on stderr, with a Python API underneath.
 pip install use-computer-cli            # no backend
 pip install "use-computer-cli[local]"   # drive this machine's display (pynput + mss)
 pip install "use-computer-cli[vnc]"     # drive a remote framebuffer over RFB (vncdotool)
+pip install "use-computer-cli[tree]"    # read the accessibility tree
 ```
 
-Backends are optional extras, imported lazily, so the package installs without them.
+Backends and the accessibility bindings are optional extras, imported lazily, so the package
+installs without them.
+
+The `tree` extra installs one binding per platform: `uiautomation` on Windows,
+`pyobjc-framework-ApplicationServices` on macOS, PyGObject on Linux. **Linux needs a system package
+as well** -- there is no `pyatspi` on PyPI, so the AT-SPI bindings come from `gir1.2-atspi-2.0`
+(plus `python3-pyatspi` on Debian and Ubuntu). The error names both halves when one is missing.
 
 The distribution is `use-computer-cli` because `use-computer` is taken on PyPI by an unrelated
 project. The command it installs is `use-computer`, and the package it imports is `use_computer`.
@@ -80,16 +101,28 @@ project. The command it installs is `use-computer`, and the package it imports i
 ## Use
 
 ```bash
-use-computer click --x 120 --y 340 --use staging
+use-computer tree --use laptop                          # what is on screen, structurally
+use-computer click --role button --name "Invia"         # act on it by name
+use-computer set-value --role text --name "Email" --value "mario@example.com"
+use-computer click --x 120 --y 340 --use staging        # or by pixel, when the tree cannot see it
 use-computer type --text "hello" --use staging
 use-computer key ctrl+s --use staging
-use-computer screenshot --use laptop            # writes a file, returns its path
+use-computer screenshot --use laptop                    # writes a file, returns its path
 
 # a batch runs over one connection -- the default command, so `batch` may be omitted
-echo '[{"action":"click","x":120,"y":340},{"action":"key","combo":"enter"}]' \
-  | use-computer - --use staging --verify
+echo '[
+  {"action":"focus","role":"text","name":"Destinatario"},
+  {"action":"type","text":"mario@example.com"},
+  {"action":"click","role":"button","name":"Invia"},
+  {"action":"tree"}
+]' | use-computer - --use laptop
 ```
 
+An action names its target by coordinate **or** by element, never both. A selector that matches
+nothing comes back with a screenshot -- the signal to switch to vision. A selector that matches
+several comes back with the candidates, because two buttons named "OK" in two dialogs is the
+ordinary case and picking one silently fails a hundred runs later.
+
 stdout is one JSON object per run; every diagnostic goes to stderr. Exit codes: `0` success,
 `1` failure, `2` bad usage.
 
@@ -104,7 +137,8 @@ action and reports the path, so a verified action does not need a `screenshot` c
 - **Setup cost.** Opening a VNC connection dominates a single action, so one run performs a
   batch of actions over one connection.
 - **Blind actuation.** A click that lands on nothing looks exactly like a click that worked, so
-  `--verify` compares the screen before and after and reports whether it changed.
+  `--verify` compares the screen before and after and reports whether it changed -- and an action
+  that went through the accessibility API reports the element it actually operated.
 
 ## Configure
 

{use_computer_cli-0.1.1 → use_computer_cli-0.2.0}/README.md

@@ -1,12 +1,22 @@
 # use-computer
 
-Executes input on a screen for computer-use agents: move, click, double-click, right-click,
-drag, scroll, type text, press key combinations, and capture a screenshot.
+Reads and acts on a screen for computer-use agents. It reads the **accessibility tree** the
+operating system already maintains -- roles, names, states and clickable boxes -- and it moves the
+mouse, clicks, drags, scrolls, types text, presses key combinations, and captures screenshots.
 
-`use-computer` is the **acting** half of a pair. [ui-locator](https://github.com/applica-software-guru/ui-locator)
-answers *where* the Invia button is and returns pixel coordinates; `use-computer` performs the
-click there. Both are driven by another AI agent through a CLI that emits JSON on stdout and
-diagnostics on stderr, with a Python API underneath.
+Every interaction takes the highest rung it can reach:
+
+1. **Element, through the platform API** -- the OS presses the button itself. No coordinates, so
+   nothing to aim and no scale to get wrong.
+2. **Element, by coordinate** -- the tree can see the control but exposes no way to operate it, so
+   `use-computer` clicks its centre and tells you it did.
+3. **Pixel, from vision** -- the tree cannot see it. Screenshot,
+   [ui-locator](https://github.com/applica-software-guru/ui-locator), click those pixels.
+
+Each rung is cheaper, faster and more accurate than the one below. Rung three is the floor the
+whole ladder stands on and is not going anywhere; it is simply no longer the only rung. Everything
+is driven by another AI agent through a CLI that emits JSON on stdout and diagnostics on stderr,
+with a Python API underneath.
 
 ## Install
 
@@ -14,9 +24,16 @@ diagnostics on stderr, with a Python API underneath.
 pip install use-computer-cli            # no backend
 pip install "use-computer-cli[local]"   # drive this machine's display (pynput + mss)
 pip install "use-computer-cli[vnc]"     # drive a remote framebuffer over RFB (vncdotool)
+pip install "use-computer-cli[tree]"    # read the accessibility tree
 ```
 
-Backends are optional extras, imported lazily, so the package installs without them.
+Backends and the accessibility bindings are optional extras, imported lazily, so the package
+installs without them.
+
+The `tree` extra installs one binding per platform: `uiautomation` on Windows,
+`pyobjc-framework-ApplicationServices` on macOS, PyGObject on Linux. **Linux needs a system package
+as well** -- there is no `pyatspi` on PyPI, so the AT-SPI bindings come from `gir1.2-atspi-2.0`
+(plus `python3-pyatspi` on Debian and Ubuntu). The error names both halves when one is missing.
 
 The distribution is `use-computer-cli` because `use-computer` is taken on PyPI by an unrelated
 project. The command it installs is `use-computer`, and the package it imports is `use_computer`.
@@ -24,16 +41,28 @@ project. The command it installs is `use-computer`, and the package it imports i
 ## Use
 
 ```bash
-use-computer click --x 120 --y 340 --use staging
+use-computer tree --use laptop                          # what is on screen, structurally
+use-computer click --role button --name "Invia"         # act on it by name
+use-computer set-value --role text --name "Email" --value "mario@example.com"
+use-computer click --x 120 --y 340 --use staging        # or by pixel, when the tree cannot see it
 use-computer type --text "hello" --use staging
 use-computer key ctrl+s --use staging
-use-computer screenshot --use laptop            # writes a file, returns its path
+use-computer screenshot --use laptop                    # writes a file, returns its path
 
 # a batch runs over one connection -- the default command, so `batch` may be omitted
-echo '[{"action":"click","x":120,"y":340},{"action":"key","combo":"enter"}]' \
-  | use-computer - --use staging --verify
+echo '[
+  {"action":"focus","role":"text","name":"Destinatario"},
+  {"action":"type","text":"mario@example.com"},
+  {"action":"click","role":"button","name":"Invia"},
+  {"action":"tree"}
+]' | use-computer - --use laptop
 ```
 
+An action names its target by coordinate **or** by element, never both. A selector that matches
+nothing comes back with a screenshot -- the signal to switch to vision. A selector that matches
+several comes back with the candidates, because two buttons named "OK" in two dialogs is the
+ordinary case and picking one silently fails a hundred runs later.
+
 stdout is one JSON object per run; every diagnostic goes to stderr. Exit codes: `0` success,
 `1` failure, `2` bad usage.
 
@@ -48,7 +77,8 @@ action and reports the path, so a verified action does not need a `screenshot` c
 - **Setup cost.** Opening a VNC connection dominates a single action, so one run performs a
   batch of actions over one connection.
 - **Blind actuation.** A click that lands on nothing looks exactly like a click that worked, so
-  `--verify` compares the screen before and after and reports whether it changed.
+  `--verify` compares the screen before and after and reports whether it changed -- and an action
+  that went through the accessibility API reports the element it actually operated.
 
 ## Configure
 

{use_computer_cli-0.1.1 → use_computer_cli-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "use-computer-cli"
-version = "0.1.1"
+version = "0.2.0"
 description = "Execute input on a screen for computer-use agents: move, click, drag, scroll, type, key, screenshot."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -38,6 +38,19 @@ dependencies = [
 # pyautogui is deliberately not used: its last release (0.9.54) dates from 2023.
 local = ["pynput>=1.8", "mss>=10"]
 vnc = ["vncdotool>=1.3"]
+# Reading and operating the UI tree. One binding per platform, behind a marker.
+#
+# Linux needs a system package as well: there is no pyatspi on PyPI, so the AT-SPI bindings come
+# from gir1.2-atspi-2.0 (plus python3-pyatspi on Debian/Ubuntu) and PyGObject reaches them through
+# GObject Introspection. UITreeUnavailableError names both halves.
+#
+# Not dogtail (GPLv2 -- fine to run, wrong to import from a permissively licensed library) and not
+# atomacos (last release 3.3.0, May 2021 -- the same staleness argument that ruled out pyautogui).
+tree = [
+    "uiautomation>=2.0; sys_platform == 'win32'",
+    "pyobjc-framework-ApplicationServices>=10; sys_platform == 'darwin'",
+    "PyGObject>=3.46; sys_platform == 'linux'",
+]
 dev = [
     "pytest>=8",
     "pytest-mock>=3.12",
@@ -78,7 +91,16 @@ strict = true
 files = ["src", "tests"]
 
 [[tool.mypy.overrides]]
-module = ["pynput.*", "mss.*", "vncdotool.*", "ApplicationServices.*", "Quartz.*", "tomli"]
+module = [
+    "pynput.*",
+    "mss.*",
+    "vncdotool.*",
+    "ApplicationServices.*",
+    "Quartz.*",
+    "gi.*",
+    "uiautomation.*",
+    "tomli",
+]
 ignore_missing_imports = true
 
 [tool.pytest.ini_options]

{use_computer_cli-0.1.1 → use_computer_cli-0.2.0}/src/use_computer/__init__.py

@@ -7,11 +7,15 @@ from __future__ import annotations
 
 from importlib.metadata import PackageNotFoundError, version
 
+from use_computer.accessibility import AccessibilityProvider, create_provider
 from use_computer.actions import (
     Action,
     ClickAction,
+    CollapseAction,
     DoubleClickAction,
     DragAction,
+    ExpandAction,
+    FocusAction,
     KeyAction,
     MouseButton,
     MoveAction,
@@ -19,7 +23,13 @@ from use_computer.actions import (
     ScreenshotAction,
     ScrollAction,
     ScrollDirection,
+    SelectAction,
+    SetValueAction,
+    ShowMenuAction,
+    ToggleAction,
+    TreeAction,
     TypeAction,
+    WindowsAction,
 )
 from use_computer.backends import Backend, create_backend
 from use_computer.compare import ChangeReport, Screenshot, compare
@@ -28,15 +38,31 @@ from use_computer.config import load as load_config
 from use_computer.coordinates import Coordinate, CoordinateSpace, ScreenInfo, convert
 from use_computer.errors import (
     ActionFailedError,
+    ActionNotSupportedError,
+    AmbiguousNodeError,
     BackendNotAvailableError,
     ConfigError,
     CoordinateSpaceError,
     KeySyntaxError,
+    NodeNotFoundError,
     PermissionDeniedError,
+    UITreeUnavailableError,
     UseComputerError,
 )
 from use_computer.keys import KeyCombo, parse_combo
 from use_computer.runner import ActionResult, ErrorInfo, RunResult, Session, run_actions
+from use_computer.tree import (
+    Box,
+    NodeSelector,
+    OutputFormat,
+    TreeReason,
+    TreeResult,
+    TreeScope,
+    UINode,
+    Via,
+    WindowInfo,
+    WindowsResult,
+)
 
 #: The distribution name on PyPI, which differs from the import package: `use-computer` was
 #: already taken there. importlib.metadata is keyed by the distribution, so this is the name
@@ -50,14 +76,19 @@ except PackageNotFoundError:  # pragma: no cover - source checkout without an in
 
 __all__ = [
     "DISTRIBUTION",
+    "AccessibilityProvider",
     "Action",
     "ActionFailedError",
+    "ActionNotSupportedError",
     "ActionResult",
+    "AmbiguousNodeError",
     "Backend",
     "BackendNotAvailableError",
     "BackendProfile",
+    "Box",
     "ChangeReport",
     "ClickAction",
+    "CollapseAction",
     "ConfigError",
     "Coordinate",
     "CoordinateSpace",
@@ -65,11 +96,16 @@ __all__ = [
     "DoubleClickAction",
     "DragAction",
     "ErrorInfo",
+    "ExpandAction",
+    "FocusAction",
     "KeyAction",
     "KeyCombo",
     "KeySyntaxError",
     "MouseButton",
     "MoveAction",
+    "NodeNotFoundError",
+    "NodeSelector",
+    "OutputFormat",
     "PermissionDeniedError",
     "ResolvedConfig",
     "RightClickAction",
@@ -79,14 +115,29 @@ __all__ = [
     "ScreenshotAction",
     "ScrollAction",
     "ScrollDirection",
+    "SelectAction",
     "Session",
+    "SetValueAction",
     "Settings",
+    "ShowMenuAction",
+    "ToggleAction",
+    "TreeAction",
+    "TreeReason",
+    "TreeResult",
+    "TreeScope",
     "TypeAction",
+    "UINode",
+    "UITreeUnavailableError",
     "UseComputerError",
+    "Via",
+    "WindowInfo",
+    "WindowsAction",
+    "WindowsResult",
     "__version__",
     "compare",
     "convert",
     "create_backend",
+    "create_provider",
     "load_config",
     "parse_combo",
     "run_actions",

use_computer_cli-0.2.0/src/use_computer/accessibility/__init__.py

@@ -0,0 +1,51 @@
+"""Choosing an accessibility provider.
+
+By the *running platform*, never by configuration: a profile does not get to claim macOS
+accessibility on Linux. And only for a backend that can have one at all -- RFB carries pixels,
+so a vnc profile gets a clear error and the calling agent uses coordinates.
+"""
+
+from __future__ import annotations
+
+import sys
+
+from use_computer.accessibility.base import AccessibilityProvider, require
+from use_computer.errors import UITreeUnavailableError
+
+#: Backends that drive a display this machine can also inspect. Everything else is remote
+#: pixels, and no accessibility API reaches across RFB.
+LOCAL_BACKENDS = frozenset({"local"})
+
+
+def create_provider(backend: str) -> AccessibilityProvider:
+    """Build the provider for this platform, or say why there cannot be one.
+
+    Raises:
+        UITreeUnavailableError: wrong backend, unsupported platform, or a missing binding.
+        PermissionDeniedError: the OS refused the accessibility permission.
+    """
+    if backend not in LOCAL_BACKENDS:
+        raise UITreeUnavailableError(
+            f"the {backend!r} backend drives a remote framebuffer, which carries pixels and no "
+            "accessibility information. Use a screenshot and coordinates instead."
+        )
+
+    if sys.platform.startswith("linux"):
+        from use_computer.accessibility.atspi import AtspiProvider  # noqa: PLC0415
+
+        return AtspiProvider()
+    if sys.platform == "win32":
+        from use_computer.accessibility.uia import UiaProvider  # noqa: PLC0415
+
+        return UiaProvider()
+    if sys.platform == "darwin":
+        from use_computer.accessibility.ax import AxProvider  # noqa: PLC0415
+
+        return AxProvider()
+
+    raise UITreeUnavailableError(
+        f"no accessibility provider for platform {sys.platform!r}."
+    )
+
+
+__all__ = ["AccessibilityProvider", "UITreeUnavailableError", "create_provider", "require"]