agentenv-framework-protocol 0.1.506__tar.gz → 0.1.508__tar.gz

{agentenv_framework_protocol-0.1.506 → agentenv_framework_protocol-0.1.508}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentenv-framework-protocol
-Version: 0.1.506
+Version: 0.1.508
 Summary: Open data-plane protocol and server SDK for agent environments.
 Project-URL: Homepage, https://www.agentenvframework.com
 Project-URL: Documentation, https://www.agentenvframework.com/docs
@@ -58,6 +58,8 @@ Requires-Dist: websockets>=13; extra == 'dev'
 Provides-Extra: macos
 Requires-Dist: pyobjc-framework-quartz>=11; (sys_platform == 'darwin') and extra == 'macos'
 Requires-Dist: pyobjc-framework-screencapturekit>=11; (sys_platform == 'darwin') and extra == 'macos'
+Provides-Extra: windows
+Requires-Dist: mss<11,>=9; extra == 'windows'
 Provides-Extra: x11
 Requires-Dist: mss<11,>=9; extra == 'x11'
 Requires-Dist: python-xlib<0.34,>=0.33; extra == 'x11'
@@ -555,6 +557,50 @@ called with the caller's role in `ToolContext`. The site's `/mcp` is one of the
 browsing route (`/website/<name>/`) never reaches. `python -m agentenv_protocol.website.conformance` checks a declared
 endpoint (`mcp`): it answers `initialize` and lists the tools the card does.
 
+## Live websites
+
+A live site is a real one, such as github.com, that a task puts in a multi env beside synthetic sites. Its server serves
+none of its pages: the real site does, and the computer reaches it over its own network. The server holds what agentenv
+knows of the site: its card, a data plane that seeds and reads the task's account through the real site's API, `ready`
+saying the account works, and `session`, the account's cookies and local storage, which `browser_sign_in` loads into the
+browser.
+
+```python
+from pydantic import BaseModel, SecretStr
+from agentenv_protocol.website import WebsiteEnvironment, ready_check, session_state, website
+
+
+@website(domains=["github.com"], live=True, allow_hosts=["*.githubassets.com", "api.github.com"])
+class GitHub(WebsiteEnvironment):
+    class Settings(BaseModel):
+        token: SecretStr                  # in env.toml as a secret:KEY or env:NAME reference
+
+    @ready_check
+    def account_works(self) -> bool:
+        ...                               # the API answers for self.settings.token
+
+    @session_state
+    def account(self, user: str | None) -> dict:
+        ...                               # {"cookies": [...], "local_storage": [...]} that sign a browser in
+```
+
+- **Its names:** `domains` are the real site's, where start tabs may open and `browser_sign_in` sets the session's
+  cookies. `allow_hosts` are the other hosts its pages reach, such as its assets and API. A computer whose network is an
+  allowlist may reach both; one that allows every host is left as it is. A name a synthetic site of the run also claims
+  is that site's.
+- **Its card:** `website/v1` says `live` (`WEBSITE_LIVE_PARAM`) and `allow_hosts` (`WEBSITE_ALLOW_HOSTS_PARAM`) beside
+  `domains`, and names no routes. The server serves no `/api/session` page route. `@website` refuses a live site
+  with no domains or a static folder, and `allow_hosts` on a site that isn't live.
+- **Its session:** the method marked `@session_state` answers `session {user}`. A live site must mark one, since only
+  the real site's cookies sign a browser in to it. Each cookie carries no domain and a `max_age`, as a synthetic
+  site's does.
+- **Settings:** any site may declare `Settings`. A deploy hands them to its server in `WEBSITE_SETTINGS_ENV`, a JSON
+  object, and the site reads them as `self.settings`, unless it has a member of that name of its own. A `SecretStr`
+  field's value in env.toml is a `secret:` or `env:` reference, which agentenv resolves only as it writes the server's
+  container, and every record shows it as withheld.
+- **The run:** a run that deploys a live site is marked live. The deploy record says `live`, and the run's metadata lists
+  its live sites with `reproducible: false`, because the real site may change between runs.
+
 ## Computer drivers
 
 A computer driver is what runs inside a computer an agent uses through screenshots and input. Write

{agentenv_framework_protocol-0.1.506 → agentenv_framework_protocol-0.1.508}/README.md

@@ -490,6 +490,50 @@ called with the caller's role in `ToolContext`. The site's `/mcp` is one of the
 browsing route (`/website/<name>/`) never reaches. `python -m agentenv_protocol.website.conformance` checks a declared
 endpoint (`mcp`): it answers `initialize` and lists the tools the card does.
 
+## Live websites
+
+A live site is a real one, such as github.com, that a task puts in a multi env beside synthetic sites. Its server serves
+none of its pages: the real site does, and the computer reaches it over its own network. The server holds what agentenv
+knows of the site: its card, a data plane that seeds and reads the task's account through the real site's API, `ready`
+saying the account works, and `session`, the account's cookies and local storage, which `browser_sign_in` loads into the
+browser.
+
+```python
+from pydantic import BaseModel, SecretStr
+from agentenv_protocol.website import WebsiteEnvironment, ready_check, session_state, website
+
+
+@website(domains=["github.com"], live=True, allow_hosts=["*.githubassets.com", "api.github.com"])
+class GitHub(WebsiteEnvironment):
+    class Settings(BaseModel):
+        token: SecretStr                  # in env.toml as a secret:KEY or env:NAME reference
+
+    @ready_check
+    def account_works(self) -> bool:
+        ...                               # the API answers for self.settings.token
+
+    @session_state
+    def account(self, user: str | None) -> dict:
+        ...                               # {"cookies": [...], "local_storage": [...]} that sign a browser in
+```
+
+- **Its names:** `domains` are the real site's, where start tabs may open and `browser_sign_in` sets the session's
+  cookies. `allow_hosts` are the other hosts its pages reach, such as its assets and API. A computer whose network is an
+  allowlist may reach both; one that allows every host is left as it is. A name a synthetic site of the run also claims
+  is that site's.
+- **Its card:** `website/v1` says `live` (`WEBSITE_LIVE_PARAM`) and `allow_hosts` (`WEBSITE_ALLOW_HOSTS_PARAM`) beside
+  `domains`, and names no routes. The server serves no `/api/session` page route. `@website` refuses a live site
+  with no domains or a static folder, and `allow_hosts` on a site that isn't live.
+- **Its session:** the method marked `@session_state` answers `session {user}`. A live site must mark one, since only
+  the real site's cookies sign a browser in to it. Each cookie carries no domain and a `max_age`, as a synthetic
+  site's does.
+- **Settings:** any site may declare `Settings`. A deploy hands them to its server in `WEBSITE_SETTINGS_ENV`, a JSON
+  object, and the site reads them as `self.settings`, unless it has a member of that name of its own. A `SecretStr`
+  field's value in env.toml is a `secret:` or `env:` reference, which agentenv resolves only as it writes the server's
+  container, and every record shows it as withheld.
+- **The run:** a run that deploys a live site is marked live. The deploy record says `live`, and the run's metadata lists
+  its live sites with `reproducible: false`, because the real site may change between runs.
+
 ## Computer drivers
 
 A computer driver is what runs inside a computer an agent uses through screenshots and input. Write

{agentenv_framework_protocol-0.1.506 → agentenv_framework_protocol-0.1.508}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "agentenv-framework-protocol"
-version = "0.1.506"
+version = "0.1.508"
 description = "Open data-plane protocol and server SDK for agent environments."
 readme = "README.md"
 license = "Apache-2.0"
@@ -61,6 +61,10 @@ x11 = [
     "mss>=9,<11",
     "python-xlib>=0.33,<0.34",
 ]
+# A Windows driver on WindowsDriver: mss captures the desktop and SendInput, through ctypes, sends its input.
+windows = [
+    "mss>=9,<11",
+]
 dev = [
     "a2a-sdk[http-server]>=0.3.26,<0.4",
     "python-multipart>=0.0.13",

{agentenv_framework_protocol-0.1.506 → agentenv_framework_protocol-0.1.508}/src/agentenv_protocol/computer/__init__.py

@@ -100,11 +100,12 @@ if TYPE_CHECKING:
     )
     from .driver import DEFAULT_IDENTITY, ComputerDriverEnvironment, DriverDeclaration, computer_driver
     from ._macos import MacOSDriver
+    from ._windows import WindowsDriver
     from ._x11 import X11Driver
     from .recording import RecordedFrame, RecordedSegment, RecordHints, Recording, frames_in_thread
 
 # The names whose modules need the computer extra (mcp, av and Pillow), by module; MacOSDriver's, the macos extra too,
-# on first use of the Mac, and X11Driver's, the x11 extra.
+# on first use of the Mac, X11Driver's, the x11 extra, and WindowsDriver's, the windows extra.
 _ON_USE = {
     "driver": ("ComputerDriverEnvironment", "computer_driver", "DEFAULT_IDENTITY", "DriverDeclaration"),
     "card": ("COMPUTER", "COMPUTER_READY", "COMPUTER_DISPLAY", "COMPUTER_SPACES", "COMPUTER_POINTER",
@@ -112,6 +113,7 @@ _ON_USE = {
     "recording": ("Recording", "RecordHints", "RecordedFrame", "RecordedSegment", "frames_in_thread"),
     "_macos": ("MacOSDriver",),
     "_x11": ("X11Driver",),
+    "_windows": ("WindowsDriver",),
 }
 
 
@@ -124,7 +126,7 @@ def __getattr__(name: str) -> Any:
 
 __all__ = [
     "ComputerDriverEnvironment", "computer_driver", "DEFAULT_IDENTITY", "DriverDeclaration", "MacOSDriver",
-    "X11Driver",
+    "X11Driver", "WindowsDriver",
     "ComputerAppEnvironment", "computer_app", "install", "linux", "windows", "install_step", "ready_check", "Linux",
     "Windows", "Service",
     "Launch", "Ready", "Packages", "AptSource", "Download", "RunScript",

agentenv_framework_protocol-0.1.508/src/agentenv_protocol/computer/_desktop.py

@@ -0,0 +1,35 @@
+"""What the drivers on an OS's own input share: the pacing of clicks and drags, how many presses and points they take,
+a drag's path, and a capture of every monitor through mss, whose class each driver passes from its own import."""
+from __future__ import annotations
+
+import math
+from typing import Any
+
+from .errors import DeviceUnavailable
+from .operations import Image
+
+MAX_CLICK_COUNT = 3
+MAX_DRAG_POINTS = 50
+#: The pause between the presses of a double or triple click, in seconds.
+CLICK_GAP_S = 0.04
+#: How far apart a drag's pointer moves are, in pixels.
+DRAG_STEP_PX = 20
+
+
+def drag_steps(path: list) -> list[tuple[float, float]]:
+    """The points a drag moves through after its start, about every DRAG_STEP_PX pixels."""
+    steps = []
+    for (x0, y0), (x1, y1) in zip(path, path[1:]):
+        n = max(2, int(math.hypot(x1 - x0, y1 - y0) / DRAG_STEP_PX))
+        steps += [(x0 + (x1 - x0) * k / n, y0 + (y1 - y0) * k / n) for k in range(1, n + 1)]
+    return steps
+
+
+def mss_capture(mss_class: Any) -> Image:
+    """Every monitor, as mss's first monitor spans them, in BGRA."""
+    try:
+        with mss_class() as m:
+            shot = m.grab(m.monitors[0])
+            return Image(shot.width, shot.height, bytes(shot.bgra), "BGRA")
+    except Exception as e:
+        raise DeviceUnavailable(f"capture failed: {type(e).__name__}: {e}") from None

{agentenv_framework_protocol-0.1.506 → agentenv_framework_protocol-0.1.508}/src/agentenv_protocol/computer/_macos.py

@@ -14,6 +14,7 @@ import time
 from contextlib import contextmanager
 from typing import Any, Callable, Iterator
 
+from ._desktop import CLICK_GAP_S, DRAG_STEP_PX, MAX_CLICK_COUNT, MAX_DRAG_POINTS
 from .driver import ComputerDriverEnvironment
 from .errors import DeliveryFailed, DeviceUnavailable, Unsupported
 from .keys import PRINTABLE_KEYS
@@ -59,10 +60,6 @@ MOVED = 5
 SETTLE_S = 0.2
 #: How long a click holds its button down, as a hand does: a press released at once is lost now and then.
 CLICK_HOLD_S = 0.03
-#: The pause between the presses of a double or triple click.
-CLICK_GAP_S = 0.04
-#: How far apart a drag's pointer moves are, in pixels.
-DRAG_STEP_PX = 20
 #: The pause between a scroll's wheel notches, as a wheel turned by hand has: notches posted back to back reach an app
 #: as fewer.
 SCROLL_GAP_S = 0.05
@@ -81,8 +78,8 @@ class MacOSDriver(ComputerDriverEnvironment):
     released."""
 
     buttons = ("left", "right", "middle")
-    max_click_count = 3
-    max_drag_points = 50
+    max_click_count = MAX_CLICK_COUNT
+    max_drag_points = MAX_DRAG_POINTS
     keys = (*MODIFIERS, *NAMED, *PRINTABLE_KEYS)
 
     def __init__(self, settings: ComputerSettings | None = None, platform: Platform | None = None) -> None:

agentenv_framework_protocol-0.1.508/src/agentenv_protocol/computer/_windows.py

@@ -0,0 +1,316 @@
+"""computer/v1 on a Windows desktop, for a driver whose software serves tools of its own, as agent-desktop's does: mss
+captures the screen and SendInput sends the input, in the session the driver runs in. The process is aware of each
+monitor's DPI, so a point is a pixel of the capture, which spans every monitor. A recording polls those captures, as
+the base class does for a driver with no stream of its own. A driver subclasses ``WindowsDriver``, declares itself with
+``computer_driver``, and serves its own tools beside these. Needs the ``windows`` extra: mss."""
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import ctypes
+import functools
+import time
+
+import mss
+
+from ._desktop import CLICK_GAP_S, MAX_CLICK_COUNT, MAX_DRAG_POINTS, drag_steps, mss_capture
+from .driver import ComputerDriverEnvironment
+from .errors import DeliveryFailed, DeviceUnavailable, Unsupported
+from .operations import (Display, Image, ReadyCheck, capture, click, display, drag, key_down, key_up, pointer_down,
+                         pointer_move, pointer_position, pointer_up, press, scroll, type_text)
+from .platform import Platform
+from .settings import ComputerSettings
+
+INPUT_MOUSE, INPUT_KEYBOARD = 0, 1
+MOVE, ABSOLUTE, VIRTUAL_DESK = 0x0001, 0x8000, 0x4000
+WHEEL, HWHEEL, WHEEL_DELTA = 0x0800, 0x1000, 120
+#: Each button's press and release flags, and the X button it is, for the two that are one.
+BUTTONS = {"left": (0x0002, 0x0004, 0), "right": (0x0008, 0x0010, 0), "middle": (0x0020, 0x0040, 0),
+           "back": (0x0080, 0x0100, 1), "forward": (0x0080, 0x0100, 2)}
+EXTENDED, KEY_UP, UNICODE = 0x0001, 0x0002, 0x0004
+#: W3C key values and their virtual keys; any other single character is the key its keyboard layout gives it.
+VIRTUAL_KEYS = {
+    "Enter": 0x0D, "Tab": 0x09, "Escape": 0x1B, "Backspace": 0x08, "Delete": 0x2E, "ArrowUp": 0x26, "ArrowDown": 0x28,
+    "ArrowLeft": 0x25, "ArrowRight": 0x27, "Home": 0x24, "End": 0x23, "PageUp": 0x21, "PageDown": 0x22, "Insert": 0x2D,
+    "Control": 0xA2, "Shift": 0xA0, "Alt": 0xA4, "Meta": 0x5B, "Super": 0x5B, "CapsLock": 0x14, "ContextMenu": 0x5D,
+    "PrintScreen": 0x2C, "Pause": 0x13, " ": 0x20, **{f"F{n}": 0x6F + n for n in range(1, 25)},
+}
+#: The keys a keyboard sends with the extended flag, without which Windows reads the arrows as the keypad's.
+EXTENDED_KEYS = {0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x2C, 0x2D, 0x2E, 0x5B, 0x5D}
+#: The modifiers VkKeyScanW says a character needs, as the bits of its high byte.
+SHIFT_STATE = ((1, 0xA0), (2, 0xA2), (4, 0xA4))
+TYPED = {"\n": 0x0D, "\t": 0x09}
+#: SetProcessDpiAwarenessContext's per-monitor (v2) awareness.
+PER_MONITOR_AWARE_V2 = -4
+#: GetSystemMetrics' indexes of the virtual screen's left, top, width and height.
+VIRTUAL_SCREEN = (76, 77, 78, 79)
+DESKTOP_SWITCHDESKTOP = 0x0100
+
+
+class _MouseInput(ctypes.Structure):
+    _fields_ = [("dx", ctypes.c_long), ("dy", ctypes.c_long), ("mouseData", ctypes.c_ulong),
+                ("dwFlags", ctypes.c_ulong), ("time", ctypes.c_ulong), ("dwExtraInfo", ctypes.c_size_t)]
+
+
+class _KeybdInput(ctypes.Structure):
+    _fields_ = [("wVk", ctypes.c_ushort), ("wScan", ctypes.c_ushort), ("dwFlags", ctypes.c_ulong),
+                ("time", ctypes.c_ulong), ("dwExtraInfo", ctypes.c_size_t)]
+
+
+class _HardwareInput(ctypes.Structure):
+    _fields_ = [("uMsg", ctypes.c_ulong), ("wParamL", ctypes.c_ushort), ("wParamH", ctypes.c_ushort)]
+
+
+class _Event(ctypes.Union):
+    _fields_ = [("mi", _MouseInput), ("ki", _KeybdInput), ("hi", _HardwareInput)]
+
+
+class _Input(ctypes.Structure):
+    _fields_ = [("type", ctypes.c_ulong), ("u", _Event)]
+
+
+class _Point(ctypes.Structure):
+    _fields_ = [("x", ctypes.c_long), ("y", ctypes.c_long)]
+
+
+@functools.cache
+def _user32():
+    """Windows' user32, typed, with the per-monitor DPI awareness set once so that points and the capture's pixels
+    agree."""
+    user32 = ctypes.WinDLL("user32", use_last_error=True)
+    user32.SendInput.argtypes = (ctypes.c_uint, ctypes.POINTER(_Input), ctypes.c_int)
+    user32.SendInput.restype = ctypes.c_uint
+    user32.VkKeyScanW.argtypes = (ctypes.c_wchar,)
+    user32.VkKeyScanW.restype = ctypes.c_short
+    user32.MapVirtualKeyW.argtypes = (ctypes.c_uint, ctypes.c_uint)
+    user32.MapVirtualKeyW.restype = ctypes.c_uint
+    user32.OpenInputDesktop.argtypes = (ctypes.c_ulong, ctypes.c_bool, ctypes.c_ulong)
+    user32.OpenInputDesktop.restype = ctypes.c_void_p
+    user32.CloseDesktop.argtypes = (ctypes.c_void_p,)
+    user32.GetCursorPos.argtypes = (ctypes.POINTER(_Point),)
+    with contextlib.suppress(AttributeError):
+        user32.SetProcessDpiAwarenessContext(ctypes.c_void_p(PER_MONITOR_AWARE_V2))
+    return user32
+
+
+class WindowsDriver(ComputerDriverEnvironment):
+    """The computer/v1 operations, the pointer and keyboard groups among them, on a Windows desktop."""
+
+    buttons = ("left", "right", "middle", "back", "forward")
+    max_click_count = MAX_CLICK_COUNT
+    max_drag_points = MAX_DRAG_POINTS
+
+    def __init__(self, settings: ComputerSettings | None = None, platform: Platform | None = None) -> None:
+        super().__init__(settings, platform)
+        #: The virtual keys each key held by @key_down pressed, which @key_up releases.
+        self._held: dict[str, list[int]] = {}
+
+    @display
+    async def screen(self) -> Display:
+        _, _, width, height = _virtual_screen()
+        return Display(width, height, 1.0)
+
+    @capture
+    async def grab(self) -> Image:
+        return await asyncio.to_thread(mss_capture, mss.MSS)
+
+    async def ready_checks(self) -> list[ReadyCheck]:
+        user32 = _user32()
+        desktop = user32.OpenInputDesktop(0, False, DESKTOP_SWITCHDESKTOP)
+        if not desktop:
+            return [ReadyCheck("input_desktop", False, f"the input desktop isn't this session's: {_last_error()}")]
+        user32.CloseDesktop(desktop)
+        return [ReadyCheck("input_desktop", True, "the input desktop is this session's")]
+
+    @click
+    async def click_at(self, x, y, button, count, modifiers) -> None:
+        held = self._pressing(modifiers)
+
+        def act():
+            _send([_key(vk, True) for vk in held] + [_move(x, y)])
+            for i in range(count):
+                _send([_button(button, True), _button(button, False)])
+                if i + 1 < count:
+                    time.sleep(CLICK_GAP_S)
+            _send([_key(vk, False) for vk in reversed(held)])
+        await asyncio.to_thread(act)
+
+    @drag
+    async def drag_along(self, path, button, duration_ms, modifiers) -> None:
+        held = self._pressing(modifiers)
+        steps = drag_steps(path)
+        pause = duration_ms / 1000 / len(steps)
+        finish = [_button(button, False)] + [_key(vk, False) for vk in reversed(held)]
+        try:
+            await asyncio.to_thread(_send, [_key(vk, True) for vk in held] + [_move(*path[0]), _button(button, True)])
+            for px, py in steps:
+                await asyncio.to_thread(_send, [_move(px, py)])
+                await asyncio.sleep(pause)
+        except BaseException:
+            with contextlib.suppress(Exception):
+                await asyncio.to_thread(_send, finish)
+            raise
+        await asyncio.to_thread(_send, finish)
+
+    @scroll
+    async def scroll_at(self, x, y, direction, amount, modifiers) -> None:
+        held = self._pressing(modifiers)
+        await asyncio.to_thread(_send, [_key(vk, True) for vk in held] + [_move(x, y)] + [_wheel(direction)] * amount
+                                + [_key(vk, False) for vk in reversed(held)])
+
+    @pointer_move
+    async def move(self, x, y) -> None:
+        await asyncio.to_thread(_send, [_move(x, y)])
+
+    @pointer_down
+    async def down(self, button) -> None:
+        await asyncio.to_thread(_send, [_button(button, True)])
+
+    @pointer_up
+    async def up(self, button) -> None:
+        await asyncio.to_thread(_send, [_button(button, False)])
+
+    @pointer_position
+    async def where(self) -> tuple[int, int]:
+        point = _Point()
+        if not _user32().GetCursorPos(ctypes.byref(point)):
+            raise DeviceUnavailable(f"the pointer can't be read: {_last_error()}")
+        left, top, _, _ = _virtual_screen()
+        return point.x - left, point.y - top
+
+    @type_text
+    async def type_in(self, text) -> None:
+        """Each character as its key on the keyboard layout, with the modifiers it needs that no held key holds, so an
+        app sees the keys; a character off the layout as itself."""
+        hold = self.settings.tap_hold_ms / 1000
+        for ch in text:
+            vks = [TYPED[ch]] if ch in TYPED else _layout_keys(ch)
+            if vks is None:
+                downs, ups = _unicode(ch)
+            else:
+                pressed = [vk for vk in vks if vk not in self._down()]
+                downs, ups = [_key(vk, True) for vk in pressed], [_key(vk, False) for vk in reversed(pressed)]
+            await asyncio.to_thread(_tap, downs, ups, hold)
+            await asyncio.sleep(self.settings.type_delay_ms / 1000)
+
+    @press
+    async def press_keys(self, keys) -> None:
+        pressed = self._pressing(keys)
+        await asyncio.to_thread(_tap, [_key(vk, True) for vk in pressed], [_key(vk, False) for vk in reversed(pressed)],
+                                self.settings.tap_hold_ms / 1000)
+
+    @key_down
+    async def keys_down(self, keys) -> None:
+        """Press what the keys need that isn't down already; for each key, the virtual keys this pressed, so that
+        @key_up releases only those. Every key is looked up before the first press."""
+        down = self._down()
+        planned = {}
+        for key in dict.fromkeys(keys):
+            planned[key] = [vk for vk in _virtual_keys(key) if vk not in down]
+            down.update(planned[key])
+        await asyncio.to_thread(_send, [_key(vk, True) for vks in planned.values() for vk in vks])
+        for key, vks in planned.items():
+            self._held.setdefault(key, []).extend(vks)
+
+    @key_up
+    async def keys_up(self, keys) -> None:
+        """Release what each key's @key_down pressed, nothing for one whose keys were down already, and a key none
+        holds as itself; the keys stay held until Windows takes the releases, so a refused call can be made again."""
+        released = [vk for key in dict.fromkeys(keys)
+                    for vk in reversed(self._held[key] if key in self._held else _virtual_keys(key)[-1:])]
+        await asyncio.to_thread(_send, [_key(vk, False) for vk in released])
+        for key in keys:
+            self._held.pop(key, None)
+
+    def _down(self) -> set[int]:
+        return {vk for vks in self._held.values() for vk in vks}
+
+    def _pressing(self, keys) -> list[int]:
+        """The virtual keys that hold ``keys`` down together, in order, each once, but those held keys hold already."""
+        down = self._down()
+        return [vk for vk in dict.fromkeys(vk for key in keys for vk in _virtual_keys(key)) if vk not in down]
+
+
+def _send(events: list[_Input]) -> None:
+    """``events`` through SendInput, refused when Windows takes fewer than all, as it does when the input desktop isn't
+    this session's or a window of higher integrity is in front."""
+    if not events:
+        return
+    sent = _user32().SendInput(len(events), (_Input * len(events))(*events), ctypes.sizeof(_Input))
+    if sent != len(events):
+        raise DeliveryFailed(f"Windows took {sent} of {len(events)} input events: {_last_error()}")
+
+
+def _tap(downs: list[_Input], ups: list[_Input], hold_s: float) -> None:
+    _send(downs)
+    time.sleep(hold_s)
+    _send(ups)
+
+
+def _virtual_screen() -> tuple[int, int, int, int]:
+    user32 = _user32()
+    return tuple(user32.GetSystemMetrics(i) for i in VIRTUAL_SCREEN)
+
+
+def _move(x: float, y: float) -> _Input:
+    """A move to the pixel (x, y) of the capture, aimed at its centre in the virtual screen's 0 to 65535 coordinates,
+    so that Windows' rounding lands on it."""
+    _, _, width, height = _virtual_screen()
+    dx = min(65535, round((int(x) + 0.5) * 65536 / width))
+    dy = min(65535, round((int(y) + 0.5) * 65536 / height))
+    return _Input(INPUT_MOUSE, _Event(mi=_MouseInput(dx, dy, 0, MOVE | ABSOLUTE | VIRTUAL_DESK, 0, 0)))
+
+
+def _button(button: str, down: bool) -> _Input:
+    press_flag, release_flag, which = BUTTONS[button]
+    return _Input(INPUT_MOUSE, _Event(mi=_MouseInput(0, 0, which, press_flag if down else release_flag, 0, 0)))
+
+
+def _wheel(direction: str) -> _Input:
+    """One notch: up and right turn the wheel forward, down and left back."""
+    flag = HWHEEL if direction in ("left", "right") else WHEEL
+    notch = WHEEL_DELTA if direction in ("up", "right") else -WHEEL_DELTA
+    return _Input(INPUT_MOUSE, _Event(mi=_MouseInput(0, 0, ctypes.c_ulong(notch).value, flag, 0, 0)))
+
+
+def _key(vk: int, down: bool) -> _Input:
+    """A virtual key with its scan code, which apps that read the physical key, as Chromium's ``code`` does, need."""
+    flags = (EXTENDED if vk in EXTENDED_KEYS else 0) | (0 if down else KEY_UP)
+    return _Input(INPUT_KEYBOARD, _Event(ki=_KeybdInput(vk, _user32().MapVirtualKeyW(vk, 0), flags, 0, 0)))
+
+
+def _unicode(ch: str) -> tuple[list[_Input], list[_Input]]:
+    """A character typed as itself, whatever the keyboard layout: a key for each of its UTF-16 units, two for one
+    beyond the BMP."""
+    data = ch.encode("utf-16-le")
+    units = [int.from_bytes(data[i:i + 2], "little") for i in range(0, len(data), 2)]
+    return [_unit(u, 0) for u in units], [_unit(u, KEY_UP) for u in units]
+
+
+def _unit(unit: int, flags: int) -> _Input:
+    return _Input(INPUT_KEYBOARD, _Event(ki=_KeybdInput(0, unit, UNICODE | flags, 0, 0)))
+
+
+def _virtual_keys(key: str) -> list[int]:
+    """The virtual keys that make ``key``: a named key's, or a character's on the keyboard layout with the modifiers
+    it needs first."""
+    if key in VIRTUAL_KEYS:
+        return [VIRTUAL_KEYS[key]]
+    if len(key) == 1:
+        if (vks := _layout_keys(key)) is None:
+            raise Unsupported(f"{key!r} has no key on this keyboard layout; type it as text instead")
+        return vks
+    raise Unsupported(f"this driver has no Windows virtual key for {key!r}")
+
+
+def _layout_keys(ch: str) -> list[int] | None:
+    """The modifiers and key that type ``ch`` on the keyboard layout, None for a character it has no key for, as one
+    beyond the BMP, which VkKeyScanW can't take."""
+    scan = _user32().VkKeyScanW(ch) if ord(ch) <= 0xFFFF else -1
+    return None if scan == -1 else [vk for bit, vk in SHIFT_STATE if scan >> 8 & bit] + [scan & 0xFF]
+
+
+def _last_error() -> str:
+    return ctypes.WinError(ctypes.get_last_error()).strerror
+
+

{agentenv_framework_protocol-0.1.506 → agentenv_framework_protocol-0.1.508}/src/agentenv_protocol/computer/_x11.py

@@ -8,7 +8,6 @@ from __future__ import annotations
 
 import asyncio
 import contextlib
-import math
 import os
 import threading
 import time
@@ -23,6 +22,7 @@ from Xlib import display as xdisplay
 from Xlib.error import ConnectionClosedError, DisplayConnectionError
 from Xlib.ext import xtest
 
+from ._desktop import CLICK_GAP_S, MAX_CLICK_COUNT, MAX_DRAG_POINTS, drag_steps, mss_capture
 from .driver import ComputerDriverEnvironment
 from .errors import DeliveryFailed, DeviceUnavailable, Unsupported
 from .operations import (Display, Image, ReadyCheck, capture, click, display, drag, key_down, key_up, pointer_down,
@@ -44,13 +44,9 @@ KEYSYMS = {
     "PrintScreen": "Print", "Pause": "Pause", " ": "space", **{f"F{n}": f"F{n}" for n in range(1, 25)},
 }
 TYPED = {"\n": "Return", "\t": "Tab"}
-#: The pause between the presses of a double or triple click, in seconds.
-CLICK_GAP_S = 0.04
 #: How long a button's release may take to reach the app, and how often the driver looks.
 RELEASE_WAIT_S = 5.0
 RELEASE_POLL_S = 0.005
-#: How far apart a drag's pointer moves are, in pixels.
-DRAG_STEP_PX = 20
 DROPPED = (ConnectionClosedError, DisplayConnectionError, OSError)
 
 
@@ -58,8 +54,8 @@ class X11Driver(ComputerDriverEnvironment):
     """The computer/v1 operations, the pointer and keyboard groups among them, on an X11 desktop."""
 
     buttons = ("left", "right", "middle", "back", "forward")
-    max_click_count = 3
-    max_drag_points = 50
+    max_click_count = MAX_CLICK_COUNT
+    max_drag_points = MAX_DRAG_POINTS
     recording = Recording(max_fps=30)
 
     def __init__(self, settings: ComputerSettings | None = None, platform: Platform | None = None) -> None:
@@ -78,7 +74,7 @@ class X11Driver(ComputerDriverEnvironment):
 
     @capture
     async def grab(self) -> Image:
-        return await asyncio.to_thread(_grab)
+        return await asyncio.to_thread(mss_capture, mss.MSS)
 
     @record
     async def stream(self, hints: RecordHints) -> AsyncIterator[RecordedFrame]:
@@ -123,7 +119,7 @@ class X11Driver(ComputerDriverEnvironment):
     @drag
     async def drag_along(self, path, button, duration_ms, modifiers) -> None:
         syms = [_keysym(k) for k in modifiers]
-        steps = _steps(path)
+        steps = drag_steps(path)
         pause = duration_ms / 1000 / len(steps)
         held: list[int] = []
 
@@ -217,7 +213,8 @@ class X11Driver(ComputerDriverEnvironment):
     async def keys_up(self, keys) -> None:
         def act(c):
             for key in keys:
-                c.release(reversed(self._held.pop(key, None) or c.codes(_keysym(key))[-1:]))
+                held = self._held.pop(key, None)
+                c.release(reversed(c.codes(_keysym(key))[-1:] if held is None else held))
         await self._send(act)
 
     async def _read(self, fn: Callable[[_Connection], Any]) -> Any:
@@ -368,24 +365,6 @@ class _Connection:
         return {8 * i + bit for i, byte in enumerate(self.d.query_keymap()) for bit in range(8) if byte >> bit & 1}
 
 
-def _grab() -> Image:
-    try:
-        with mss.MSS() as m:
-            shot = m.grab(m.monitors[0])
-            return Image(shot.width, shot.height, bytes(shot.bgra), "BGRA")
-    except Exception as e:
-        raise DeviceUnavailable(f"capture failed: {type(e).__name__}: {e}") from None
-
-
-def _steps(path: list) -> list[tuple[float, float]]:
-    """The points a drag moves through after its start, about every DRAG_STEP_PX pixels."""
-    steps = []
-    for (x0, y0), (x1, y1) in zip(path, path[1:]):
-        n = max(2, int(math.hypot(x1 - x0, y1 - y0) / DRAG_STEP_PX))
-        steps += [(x0 + (x1 - x0) * k / n, y0 + (y1 - y0) * k / n) for k in range(1, n + 1)]
-    return steps
-
-
 def _keysym(key: str) -> int:
     if key in KEYSYMS:
         return XK.string_to_keysym(KEYSYMS[key])