agentenv-framework-protocol 0.1.443__tar.gz → 0.1.444__tar.gz

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/.gitignore

@@ -50,6 +50,7 @@ __pycache__/
 build/
 develop-eggs/
 dist/
+node_modules/
 downloads/
 eggs/
 .eggs/

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentenv-framework-protocol
-Version: 0.1.443
+Version: 0.1.444
 Summary: Open data-plane protocol and server SDK for agent environments.
 Project-URL: Homepage, https://www.agentenvframework.com
 Project-URL: Documentation, https://www.agentenvframework.com/docs
@@ -561,7 +561,8 @@ driver has them, through the `served_tools`, `served_extensions` and `served_ope
 - **The input queue.** Input calls run one at a time in arrival order; one that waits past
   `queue_wait_s` is refused `input_busy`. Screenshots and waits never queue.
 - **The computer skill.** `computer/v1` declares the seven tools, its params the driver's
-  features. The driver serves its skill through `skill/v1` (an `@agent_skill` handler on the base
+  features, among them `screen`: `desktop`, or `page` for a driver whose screen is a website
+  browser's active tab, which a computer env pairs with exactly one website browser. The driver serves its skill through `skill/v1` (an `@agent_skill` handler on the base
   class): a SKILL.md written from the computer as it booted (its platform, session user and home,
   the live screen size, what its input takes, its OS's shortcuts) and ending with the driver
   class's `notes`, markdown for what its card can't say. A driver whose software ships a skill of
@@ -569,7 +570,9 @@ driver has them, through the `served_tools`, `served_extensions` and `served_ope
   computer skill leads with it, the computer's part following.
 - **Framework extensions.** `computer-ready/v1` (`status`: real checks of the display and a
   capture, the driver's own `ready_checks()`, and one per app on the computer, asked of the app's
-  own server (see Computer apps); tools refuse `device_unavailable` until they pass once), `computer-display/v1`, `computer-spaces/v1` (the default space, written into the tool
+  own server (see Computer apps); tools refuse `device_unavailable` until they pass once; a check
+  that can't pass again is `final`, and while one fails the status carries `failed`, saying why,
+  which `boot.ready()` raises at once rather than waiting out its timeout), `computer-display/v1`, `computer-spaces/v1` (the default space, written into the tool
   descriptions), `computer-keyboard/v1`, `computer-exec/v1` (`run {argv, cwd?, env?, timeout_s?,
   request_id}`: output capped per stream, a repeated `request_id` answered with the first result) and,
   with `recording`, `computer-recording/v1` (below).
@@ -798,7 +801,7 @@ its card lists each one's card as a child environment, rooted at `/svc/<app>/`,
 requests to the app's server.
 
 `MailApp.declaration()` returns what the class declares, an `AppDeclaration`, which a `computer_app`
-env stores, and `python -m agentenv_protocol.computer._declare <folder> <file> <class>` prints it as
+env stores (`browser: true` for a website browser), and `python -m agentenv_protocol.computer._declare <folder> <file> <class>` prints it as
 JSON, or a driver's `DriverDeclaration`, without the class's own dependencies installed.
 
 ### Website browsers
@@ -825,8 +828,8 @@ are the profile's template. The base class adds:
   `AGENTENV_SITE_PROXY` set, it goes out through that proxy, with WebRTC's non-proxied UDP off.
   Alone it bypasses nothing, loopback included; in a computer it bypasses loopback (`127.0.0.1`,
   `[::1]`, `localhost`), so the apps beside it stay reachable while `*.localhost` names still go
-  through. Alone (`AGENTENV_BROWSER_ALONE=1`) it also takes `--window-size` and `--headless=new`,
-  except on the open web (`AGENTENV_BROWSER_OPEN_WEB=1`), and refuses to start with no proxy.
+  through. Alone, on a computer with no display (`XDG_SESSION_TYPE=none`, a browser computer's), it
+  also takes `--window-size` and `--headless=new`, and refuses to start with no proxy.
   `declaration()` refuses a subclass whose command carries any of these, or `--remote-allow-origins`, or
   any other way to open a debugging port.
 - **Before each launch** it empties its cache, so no launch inherits another's, and writes the deploy's
@@ -859,7 +862,10 @@ are the profile's template. The base class adds:
   launch (with its cause: `boot`, `server_restart`, `launch`, `data/add` or `data/reset`), each stop it made
   and each exit it didn't, with its exit `code` or the `signal` that ended it, since the computer booted.
   Nothing relaunches a browser that exits on its own: in a computer the agent can, and alone it's an infra
-  failure, its ready check false.
+  failure, its ready check false. `page/viewport`, `page/capture` and `page/input`, framework-only, are
+  its active tab's page as a page driver sees it: its size, which headless is its window's; a PNG of it,
+  or `blank` for a tab that paints nothing within 3 s (1 s with no URL yet); and CDP input calls
+  (`Input.dispatchMouseEvent`, `Input.dispatchKeyEvent`, `Input.insertText`), sent to it in order.
 - **Its skill**: sites by their real names, files to upload in `~/Uploads` and downloads in
   `~/Downloads`, folders it makes.
 

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/README.md

@@ -505,7 +505,8 @@ driver has them, through the `served_tools`, `served_extensions` and `served_ope
 - **The input queue.** Input calls run one at a time in arrival order; one that waits past
   `queue_wait_s` is refused `input_busy`. Screenshots and waits never queue.
 - **The computer skill.** `computer/v1` declares the seven tools, its params the driver's
-  features. The driver serves its skill through `skill/v1` (an `@agent_skill` handler on the base
+  features, among them `screen`: `desktop`, or `page` for a driver whose screen is a website
+  browser's active tab, which a computer env pairs with exactly one website browser. The driver serves its skill through `skill/v1` (an `@agent_skill` handler on the base
   class): a SKILL.md written from the computer as it booted (its platform, session user and home,
   the live screen size, what its input takes, its OS's shortcuts) and ending with the driver
   class's `notes`, markdown for what its card can't say. A driver whose software ships a skill of
@@ -513,7 +514,9 @@ driver has them, through the `served_tools`, `served_extensions` and `served_ope
   computer skill leads with it, the computer's part following.
 - **Framework extensions.** `computer-ready/v1` (`status`: real checks of the display and a
   capture, the driver's own `ready_checks()`, and one per app on the computer, asked of the app's
-  own server (see Computer apps); tools refuse `device_unavailable` until they pass once), `computer-display/v1`, `computer-spaces/v1` (the default space, written into the tool
+  own server (see Computer apps); tools refuse `device_unavailable` until they pass once; a check
+  that can't pass again is `final`, and while one fails the status carries `failed`, saying why,
+  which `boot.ready()` raises at once rather than waiting out its timeout), `computer-display/v1`, `computer-spaces/v1` (the default space, written into the tool
   descriptions), `computer-keyboard/v1`, `computer-exec/v1` (`run {argv, cwd?, env?, timeout_s?,
   request_id}`: output capped per stream, a repeated `request_id` answered with the first result) and,
   with `recording`, `computer-recording/v1` (below).
@@ -742,7 +745,7 @@ its card lists each one's card as a child environment, rooted at `/svc/<app>/`,
 requests to the app's server.
 
 `MailApp.declaration()` returns what the class declares, an `AppDeclaration`, which a `computer_app`
-env stores, and `python -m agentenv_protocol.computer._declare <folder> <file> <class>` prints it as
+env stores (`browser: true` for a website browser), and `python -m agentenv_protocol.computer._declare <folder> <file> <class>` prints it as
 JSON, or a driver's `DriverDeclaration`, without the class's own dependencies installed.
 
 ### Website browsers
@@ -769,8 +772,8 @@ are the profile's template. The base class adds:
   `AGENTENV_SITE_PROXY` set, it goes out through that proxy, with WebRTC's non-proxied UDP off.
   Alone it bypasses nothing, loopback included; in a computer it bypasses loopback (`127.0.0.1`,
   `[::1]`, `localhost`), so the apps beside it stay reachable while `*.localhost` names still go
-  through. Alone (`AGENTENV_BROWSER_ALONE=1`) it also takes `--window-size` and `--headless=new`,
-  except on the open web (`AGENTENV_BROWSER_OPEN_WEB=1`), and refuses to start with no proxy.
+  through. Alone, on a computer with no display (`XDG_SESSION_TYPE=none`, a browser computer's), it
+  also takes `--window-size` and `--headless=new`, and refuses to start with no proxy.
   `declaration()` refuses a subclass whose command carries any of these, or `--remote-allow-origins`, or
   any other way to open a debugging port.
 - **Before each launch** it empties its cache, so no launch inherits another's, and writes the deploy's
@@ -803,7 +806,10 @@ are the profile's template. The base class adds:
   launch (with its cause: `boot`, `server_restart`, `launch`, `data/add` or `data/reset`), each stop it made
   and each exit it didn't, with its exit `code` or the `signal` that ended it, since the computer booted.
   Nothing relaunches a browser that exits on its own: in a computer the agent can, and alone it's an infra
-  failure, its ready check false.
+  failure, its ready check false. `page/viewport`, `page/capture` and `page/input`, framework-only, are
+  its active tab's page as a page driver sees it: its size, which headless is its window's; a PNG of it,
+  or `blank` for a tab that paints nothing within 3 s (1 s with no URL yet); and CDP input calls
+  (`Input.dispatchMouseEvent`, `Input.dispatchKeyEvent`, `Input.insertText`), sent to it in order.
 - **Its skill**: sites by their real names, files to upload in `~/Uploads` and downloads in
   `~/Downloads`, folders it makes.
 

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "agentenv-framework-protocol"
-version = "0.1.443"
+version = "0.1.444"
 description = "Open data-plane protocol and server SDK for agent environments."
 readme = "README.md"
 license = "Apache-2.0"

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/src/agentenv_protocol/computer/app.py

@@ -236,7 +236,7 @@ class AppDeclaration(BaseModel):
     """What a ComputerAppEnvironment class declares, as a ``computer_app`` env stores it: the platforms it ``supports``,
     as a driver names them; ``os``, what it declares for each OS type; ``settings`` and ``build_settings``, the JSON
     schemas of its ``Settings`` and of its ``BuildSettings`` when it declares any; its ``skill``; and the ``data``
-    methods, ``install_steps`` and ``ready_checks`` it serves, by name."""
+    methods, ``install_steps`` and ``ready_checks`` it serves, by name; and ``browser``, true for a website browser."""
 
     model_config = ConfigDict(extra="forbid")
 
@@ -249,6 +249,7 @@ class AppDeclaration(BaseModel):
     data: list[str]
     install_steps: list[InstallStep]
     ready_checks: list[AppReadyCheck]
+    browser: Optional[bool] = None
 
     @model_validator(mode="before")
     @classmethod

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/src/agentenv_protocol/computer/apps.py

@@ -8,6 +8,7 @@ from __future__ import annotations
 
 import asyncio
 import copy
+import json
 from pathlib import Path
 from typing import Optional
 
@@ -16,7 +17,7 @@ import httpx
 from .. import client
 from ..types import RPC_PATH, WELL_KNOWN_PATH, EnvironmentCard
 from .app import COMPUTER_APP, FAILURE, SOCKET
-from .app_runtime import CHECK_TIMEOUT_S, AppReadiness
+from .app_runtime import CHECK_TIMEOUT_S, LAUNCHES, AppReadiness
 from .operations import ReadyCheck
 
 #: Where a composed computer installs its apps, a folder each.
@@ -54,7 +55,7 @@ async def check(folder: Path) -> ReadyCheck:
 async def children(folders: list[Path]) -> list[EnvironmentCard]:
     """Each app in ``folders`` as a child environment of its computer: its server's card, rooted at ``/svc/<name>/``,
     or, while its server isn't answering, its name and route alone."""
-    cards = await asyncio.gather(*(_card(folder) for folder in folders))
+    cards = await asyncio.gather(*(served_card(folder) for folder in folders))
     return [EnvironmentCard.model_validate(child_card(folder.name, card)) if card is not None
             else EnvironmentCard(name=folder.name, url=f"/svc/{folder.name}{RPC_PATH}")
             for folder, card in zip(folders, cards)]
@@ -85,7 +86,7 @@ def child_card(name: str, card: dict) -> dict:
 
 async def owned_paths(folders: list[Path]) -> list[str]:
     """The paths under the session user's home that the apps in ``folders`` keep their data in, as their cards say."""
-    cards = await asyncio.gather(*(_card(folder) for folder in folders))
+    cards = await asyncio.gather(*(served_card(folder) for folder in folders))
     return [path for card in cards if card is not None
             for path in client.extension_params(card, COMPUTER_APP).get("paths") or []]
 
@@ -104,7 +105,7 @@ def endpoint(card: dict, method: str) -> str:
     return client.find_extension_method(card, COMPUTER_APP, method)["endpoint"]
 
 
-async def _card(folder: Path) -> Optional[dict]:
+async def served_card(folder: Path) -> Optional[dict]:
     """The card the app in ``folder`` serves, or None while its server isn't answering."""
     try:
         async with _client(folder / SOCKET) as http:
@@ -113,6 +114,33 @@ async def _card(folder: Path) -> Optional[dict]:
         return None
 
 
+def ended_on_its_own(folder: Path) -> Optional[str]:
+    """Why the app in ``folder`` can't be ready again where nothing launches it but its server: its launch log says it
+    ended on its own, an infrastructure failure; None while it hasn't."""
+    for entry in launch_log(folder):
+        if entry.get("event") == "exit":
+            how = (f"by signal {entry['signal']}" if "signal" in entry else
+                   f"with code {entry['code']}" if "code" in entry else "in a way that wasn't recorded")
+            return (f"it ended on its own at {entry.get('at')}, {how}, and nothing launches it again: an "
+                    "infrastructure failure")
+    return None
+
+
+def launch_log(folder: Path) -> list[dict]:
+    """Each entry of the launch log of the app in ``folder`` that's whole, in order."""
+    try:
+        lines = (folder / LAUNCHES).read_text(encoding="utf-8").splitlines()
+    except FileNotFoundError:
+        return []
+    entries = []
+    for line in lines:
+        try:
+            entries.append(json.loads(line))
+        except ValueError:
+            continue
+    return entries
+
+
 def _unanswered(folder: Path) -> str:
     try:
         return f"its server failed to start: {(folder / FAILURE).read_text()}"

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/src/agentenv_protocol/computer/card.py

@@ -89,7 +89,7 @@ def computer_features(driver: ComputerDriverEnvironment) -> dict:
         "max_wait_ms": driver.max_wait_ms, "max_drag_points": driver.max_drag_points,
         "max_duration_ms": driver.max_duration_ms, "max_key_repeat": driver.max_key_repeat,
         "max_scroll_amount": driver.max_scroll_amount, "delivery": "foreground", "platform": driver.platform.as_dict(),
-        "supports": list(driver.supports), "default_data_paths": list(DEFAULT_DATA_PATHS),
+        "supports": list(driver.supports), "default_data_paths": list(DEFAULT_DATA_PATHS), "screen": driver.screen_kind,
     }
 
 

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/src/agentenv_protocol/computer/checks.py

@@ -37,6 +37,8 @@ def check_features(cls: type) -> None:
                 problems.append("keys are W3C key values in their W3C case")
         except AgentEnvError as e:
             problems.append(f"keys: {e.message}")
+    if cls.screen_kind not in ("desktop", "page"):
+        problems.append(f"screen_kind is desktop or page, got {cls.screen_kind!r}")
     if ("set_display" in declared(cls)) != bool(cls.display_modes):
         problems.append("display_modes and @set_display come together")
     problems += check_recording(cls.recording)

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/src/agentenv_protocol/computer/driver.py

@@ -4,7 +4,7 @@ computer's apps as child environments, each reached at ``/svc/<name>/`` through
 from __future__ import annotations
 
 import asyncio
-from typing import Any, Callable, ClassVar
+from typing import Any, Callable, ClassVar, Literal
 
 import httpx
 from pydantic import BaseModel, ConfigDict
@@ -79,6 +79,9 @@ class ComputerDriverEnvironment(Tools, Extensions, DataPlane, AgentEnvEnvironmen
     recording: ClassVar[Recording | None] = Recording()
     #: What an agent should know about this driver's computer beyond what its card says, as markdown: the end of its skill.
     notes: ClassVar[str] = ""
+    #: What the computer's screen shows, its card's ``screen``: a ``desktop``, or a website browser's ``page``, its
+    #: active tab.
+    screen_kind: ClassVar[Literal["desktop", "page"]] = "desktop"
     tool_access = True
     #: What the base class keeps for the driver, built by ``__init__``.
     _computer: ComputerRuntime | None = None

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/src/agentenv_protocol/computer/extensions.py

@@ -45,8 +45,12 @@ class Extensions:
                description="Whether the computer is usable now, by real checks; never sends input.")
     async def _computer_ready_status(self) -> dict:
         checks = await self._computer.ready_checks()
-        return {"ready": all(c.ok for c in checks),
-                "checks": [{"name": c.name, "ok": c.ok, "detail": c.detail} for c in checks]}
+        status = {"ready": all(c.ok for c in checks),
+                  "checks": [{"name": c.name, "ok": c.ok, "detail": c.detail, **({"final": True} if c.final else {})}
+                             for c in checks]}
+        if failed := [c for c in checks if c.final and not c.ok]:
+            status["failed"] = "; ".join(f"{c.name}: {c.detail}" for c in failed)
+        return status
 
     @extension(COMPUTER_DISPLAY, op="get", method="GET", description="Get or set the display's resolution and scale.")
     async def _computer_display_get(self) -> dict:

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/src/agentenv_protocol/computer/operations.py

@@ -74,6 +74,8 @@ class ReadyCheck:
     name: str
     ok: bool
     detail: str = ""
+    #: Failed for good: the check can't pass again, so the computer has failed.
+    final: bool = False
 
 
 @dataclass(frozen=True)

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/src/agentenv_protocol/computer/platform.py

@@ -28,14 +28,15 @@ class Platform:
 
 def detect(environ: dict | None = None, os_release: Path = Path("/etc/os-release")) -> Platform:
     """The running platform: the OS, ``/etc/os-release``'s ID and VERSION_ID on Linux, the machine, and the
-    display from ``XDG_SESSION_TYPE``, else ``DISPLAY`` (x11) or ``WAYLAND_DISPLAY`` (wayland)."""
+    display from ``XDG_SESSION_TYPE``, ``none`` for a computer with no display, else ``DISPLAY`` (x11) or
+    ``WAYLAND_DISPLAY`` (wayland)."""
     environ = os.environ if environ is None else environ
     name = _OS.get(sys.platform, sys.platform)
     release = _os_release(os_release) if name == "linux" else {}
     display = None
     if name == "linux":
         session = (environ.get("XDG_SESSION_TYPE") or "").strip().lower()
-        if session in ("x11", "wayland"):
+        if session in ("x11", "wayland", "none"):
             display = session
         elif environ.get("DISPLAY"):
             display = "x11"

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/src/agentenv_protocol/website_browser.py

@@ -2,8 +2,9 @@
 its build installs and starts; this class adds what agentenv owns, the same in both: its flags after the subclass's
 command, the unattended-use policies, a CDP connection over the browser's debugging pipe, the settings every browser
 takes, a fresh cache and the deploy's prompt answers at every launch, another window for a launch while it runs, its
-skill and ``browser/v1`` on its card. Whether it runs alone, on the open web and through which site proxy come from
-variables agentenv sets, never from its settings."""
+skill and ``browser/v1`` on its card. It runs alone, headless, on a computer with no display, a browser computer's,
+whose screen is its page; which site proxy it goes through comes from a variable agentenv sets, never from its
+settings."""
 from __future__ import annotations
 
 import asyncio
@@ -23,11 +24,11 @@ from pydantic import BaseModel, Field, field_validator
 
 from .agent_env_environment import agent_skill, extension
 from .computer import AppDeclaration, ComputerAppEnvironment, Launch, install_step, ready_check
+from .computer import app as computer_app
+from .computer.errors import DeviceUnavailable, InvalidArgument
 
 logger = logging.getLogger(__name__)
 
-ALONE_ENV = "AGENTENV_BROWSER_ALONE"
-OPEN_WEB_ENV = "AGENTENV_BROWSER_OPEN_WEB"
 SITE_PROXY_ENV = "AGENTENV_SITE_PROXY"
 BROWSER = "urn:agentenv:browser/v1"
 #: The flags agentenv sets, by prefix, which a subclass's command may not carry: among them every way to open a debugging
@@ -51,6 +52,9 @@ HASH_CHUNK_BYTES = 1 << 20
 AGENT = "agent"
 #: The targets agentenv's connection follows.
 PAGES = [{"type": "page"}]
+#: The CDP input a computer's page driver may send to the active tab, and how long its screenshot may take.
+PAGE_INPUT = frozenset({"Input.dispatchMouseEvent", "Input.dispatchKeyEvent", "Input.insertText"})
+CAPTURE_TIMEOUT_S, UNNAVIGATED_CAPTURE_TIMEOUT_S = 3.0, 1.0
 #: Alone, loopback goes through the proxy too. In a computer, the apps on loopback beside it stay reachable; ``<-loopback>``
 #: removes Chromium's own rule, so ``*.localhost`` names still don't.
 ALONE_BYPASS = "<-loopback>"
@@ -187,8 +191,8 @@ class BrowserControl:
     ``dialogs`` says to: always leaving a page that asks before it's left, so no answer of agentenv's blocks a
     navigation."""
 
-    def __init__(self, devtools: DevToolsPipe, dialogs: str) -> None:
-        self.devtools, self.dialogs = devtools, dialogs
+    def __init__(self, devtools: DevToolsPipe, dialogs: str, viewport: Optional[tuple[int, int]] = None) -> None:
+        self.devtools, self.dialogs, self.viewport = devtools, dialogs, viewport
         #: When this record starts: a server that restarts starts another.
         self.since = _now()
         self.tabs: dict[str, dict] = {}
@@ -209,7 +213,8 @@ class BrowserControl:
         downloads unattended: into ``~/Downloads``, a name downloaded again kept beside the first."""
         self.ended(STOP)
         await self.devtools.call("Target.setDiscoverTargets", {"discover": True, "filter": PAGES})
-        await self.devtools.call("Target.setAutoAttach", {"autoAttach": True, "waitForDebuggerOnStart": False,
+        await self.devtools.call("Target.setAutoAttach", {"autoAttach": True,
+                                                          "waitForDebuggerOnStart": self.viewport is not None,
                                                           "flatten": True, "filter": PAGES})
         await self.devtools.call("Browser.setDownloadBehavior", {"behavior": "default", "eventsEnabled": True})
 
@@ -268,6 +273,36 @@ class BrowserControl:
                 "closed_tabs": list(self.closed_tabs), "downloads": list(self.downloads.values()),
                 "dialogs": list(self.dialog_log)}
 
+    async def active_session(self) -> str:
+        """The session of the active tab, else of the first open one; DeviceUnavailable with no tab open."""
+        active = await self.active_tab() or next(iter(self.tabs), None)
+        session = next((session for session, target in self._sessions.items() if target == active), None)
+        if session is None:
+            raise DeviceUnavailable("the browser has no tab open")
+        return session
+
+    async def capture(self) -> Optional[str]:
+        """The active tab's page as it shows now, a PNG, base64; None for a tab that paints nothing within
+        ``CAPTURE_TIMEOUT_S``, which a desktop browser shows blank, or sooner for a tab with no URL yet, such as a new
+        tab whose navigation the browser blocked, which may never paint."""
+        session = await self.active_session()
+        navigated = self.tabs.get(self._sessions[session], {}).get("url")
+        try:
+            shot = await self.devtools.call("Page.captureScreenshot", {"format": "png", "optimizeForSpeed": True},
+                                            session_id=session,
+                                            timeout_s=CAPTURE_TIMEOUT_S if navigated else UNNAVIGATED_CAPTURE_TIMEOUT_S)
+        except asyncio.TimeoutError:
+            return None
+        return shot["data"]
+
+    async def dispatch(self, events: list[dict]) -> None:
+        """Each of ``events``, a CDP input call, sent to the active tab in order; InvalidArgument for any other call."""
+        if refused := [event.get("method") for event in events if event.get("method") not in PAGE_INPUT]:
+            raise InvalidArgument(f"page input takes {', '.join(sorted(PAGE_INPUT))}, not {refused}")
+        session = await self.active_session()
+        for event in events:
+            await self.devtools.call(event["method"], event.get("params") or {}, session_id=session)
+
     async def _refresh(self, target: Optional[str] = None) -> None:
         """The URL and title of the tab ``target``, or of every open tab, as the browser has them now: it sends no
         event when a page's title changes."""
@@ -307,9 +342,19 @@ class BrowserControl:
     def _on_Target_attachedToTarget(self, params: dict, session: Optional[str]) -> None:
         if params["targetInfo"].get("type") == "page":
             self._sessions[params["sessionId"]] = params["targetInfo"]["targetId"]
-            self._spawn(self.devtools.call("Page.enable", session_id=params["sessionId"]))
-            # A page attached as it opens, a popup say, waits for this before it navigates.
-            self._spawn(self.devtools.call("Runtime.runIfWaitingForDebugger", session_id=params["sessionId"]))
+            self._spawn(self._let_run(params["sessionId"]))
+
+    async def _let_run(self, session: str) -> None:
+        """A page it attached to, enabled and, headless, held to its window's size, then told to run: a page attached as
+        it opens, a popup say, or any new page headless, waits for that before it runs."""
+        try:
+            await self.devtools.call("Page.enable", session_id=session)
+            if self.viewport is not None:
+                width, height = self.viewport
+                await self.devtools.call("Emulation.setDeviceMetricsOverride", {
+                    "width": width, "height": height, "deviceScaleFactor": 1, "mobile": False}, session_id=session)
+        finally:
+            await self.devtools.call("Runtime.runIfWaitingForDebugger", session_id=session)
 
     def _on_Target_detachedFromTarget(self, params: dict, session: Optional[str]) -> None:
         self._sessions.pop(params["sessionId"], None)
@@ -433,16 +478,19 @@ class WebsiteBrowserEnvironment(ComputerAppEnvironment):
         for folder in (UPLOADS, DOWNLOADS):
             (Path.home() / folder).mkdir(exist_ok=True)
         #: agentenv's control of it over that connection, and the record it keeps.
-        self.control = BrowserControl(self.devtools, self.settings.dialogs)
+        self.control = BrowserControl(self.devtools, self.settings.dialogs,
+                                      (self.settings.window.width, self.settings.window.height) if _alone()
+                                      else None)
         self.devtools.on_close(self._pipe_closed)
 
     @classmethod
     def declaration(cls) -> AppDeclaration:
-        """What the class declares; ValueError for a launch whose command carries a flag agentenv owns."""
+        """What the class declares, a website browser; ValueError for a launch whose command carries a flag agentenv
+        owns."""
         for declared in cls._installs.values():
             if declared.launch is not None:
                 _refuse_owned(cls.__name__, declared.launch)
-        return super().declaration()
+        return super().declaration().model_copy(update={"browser": True})
 
     def profile_dir(self) -> Path:
         return Path.home() / self.profile
@@ -469,7 +517,7 @@ class WebsiteBrowserEnvironment(ComputerAppEnvironment):
                              "names none")
         if alone:
             flags.append(f"--window-size={settings.window.width},{settings.window.height}")
-        if _headless():
+        if alone:
             flags.append("--headless=new")
         return flags
 
@@ -491,7 +539,7 @@ class WebsiteBrowserEnvironment(ComputerAppEnvironment):
 
     def _command_line_tabs(self) -> list[str]:
         """The start tabs its command line opens: headless Chromium takes one, and the rest open over its pipe."""
-        return self.settings.start_urls[:1] if _headless() else self.settings.start_urls
+        return self.settings.start_urls[:1] if _alone() else self.settings.start_urls
 
     async def _launch_fds(self) -> dict[int, int]:
         """Before each launch, an empty cache and the deploy's prompt answers in its profile; and a new debugging pipe:
@@ -568,10 +616,27 @@ class WebsiteBrowserEnvironment(ComputerAppEnvironment):
         "A website browser agentenv runs: its build, the clock its pages see and, alone, its window. record returns "
         "what agentenv's own connection to it saw: its open tabs, each with whether it's active, the tabs closed and "
         "who closed them, its downloads with their size and sha256, its JavaScript dialogs and who answered each, and "
-        "each launch, stop and exit of it since the computer booted, with its cause."))
+        "each launch, stop and exit of it since the computer booted, with its cause. page/viewport, page/capture and "
+        "page/input are its active tab's page as a computer's page driver sees it: its size in pixels, which headless "
+        "is its window's, a PNG of it, and CDP input calls sent to it in order."))
     async def browser_record(self) -> dict:
         return {**await self.control.record(), "launches": self._computer_app.launches()}
 
+    @extension(BROWSER, op="page/viewport", method="GET")
+    async def browser_page_viewport(self) -> dict:
+        window = self.settings.window
+        return {"width": window.width, "height": window.height}
+
+    @extension(BROWSER, op="page/capture", method="GET")
+    async def browser_page_capture(self) -> dict:
+        png = await self.control.capture()
+        return {"png": png} if png is not None else {"blank": True}
+
+    @extension(BROWSER, op="page/input")
+    async def browser_page_input(self, events: list[dict]) -> dict:
+        await self.control.dispatch(events)
+        return {}
+
     def extension_params(self, uri: str) -> dict:
         if uri == BROWSER:
             params: dict[str, Any] = {"chromium": self.version, "clock": "real"}
@@ -586,11 +651,8 @@ class WebsiteBrowserEnvironment(ComputerAppEnvironment):
 
 
 def _alone() -> bool:
-    return os.environ.get(ALONE_ENV) == "1"
-
-
-def _open_web() -> bool:
-    return os.environ.get(OPEN_WEB_ENV) == "1"
+    """Whether it runs on a computer with no display, a browser computer's, alone and headless."""
+    return computer_app.detect().display == "none"
 
 
 def _shm_bytes(path: str = "/dev/shm") -> int:
@@ -601,11 +663,6 @@ def _shm_bytes(path: str = "/dev/shm") -> int:
     return stats.f_frsize * stats.f_blocks
 
 
-def _headless() -> bool:
-    """Alone it runs headless, but headful on Xvfb for the open web, whose sites block headless browsers."""
-    return _alone() and not _open_web()
-
-
 def _refuse_owned(name: str, launch: Launch) -> None:
     if owned := [arg for arg in launch.command if arg.startswith(OWNED_FLAGS)]:
         raise ValueError(f"{name}'s launch sets {', '.join(owned)}, which agentenv sets for every website browser")

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/tests/test_computer_driver.py

@@ -115,6 +115,7 @@ def test_unknown_predicates_are_refused():
     ({"keys": ("enter",)}, "keys are W3C key values in their W3C case"),
     ({"keys": ("Return",)}, "keys: 'Return' is not a W3C key value"),
     ({"display_modes": ((10, 10),)}, "display_modes and @set_display come together"),
+    ({"screen_kind": "tablet"}, "screen_kind is desktop or page, got 'tablet'"),
 ])
 def test_features_are_checked_when_the_class_is_defined(features, error):
     with pytest.raises(TypeError, match=error):
@@ -169,7 +170,7 @@ async def test_the_card_declares_the_contract_the_driver_implements():
     computer = extensions["urn:agentenv:computer/v1"]["params"]
     assert computer["buttons"] == ["left", "right", "middle"] and computer["max_click_count"] == 3
     assert computer["keys"] == "all" and computer["platform"] == LINUX.as_dict() and computer["delivery"] == "foreground"
-    assert computer["default_data_paths"] == ["Desktop", "Documents", "Downloads"]
+    assert computer["default_data_paths"] == ["Desktop", "Documents", "Downloads"] and computer["screen"] == "desktop"
     assert "methods" not in computer and extensions["urn:agentenv:skill/v1"]["params"]["methods"]["get"]["method"] == "GET"
     assert extensions["urn:agentenv:computer-pointer/v1"]["params"]["buttons"] == ["left", "right", "middle"]
     assert set(extensions["urn:agentenv:computer-keyboard/v1"]["params"]["methods"]) == {"get", "set"}
@@ -294,6 +295,21 @@ async def test_the_drivers_own_ready_checks_count():
     async with http_client(Raising()) as http:
         raised = (await http.get("/agentenv/ext/computer-ready/status")).json()
     assert raised["checks"][-1] == {"name": "driver", "ok": False, "detail": "RuntimeError: probe crashed"}
+    assert "failed" not in before and "failed" not in raised
+
+
+@pytest.mark.asyncio
+async def test_a_final_check_that_fails_has_the_status_say_the_computer_failed_and_why():
+    class Ended(FakeDriver):
+        async def ready_checks(self):
+            return [ReadyCheck("browser", False, "it ended on its own", final=True),
+                    ReadyCheck("session", True, "logged in", final=True)]
+
+    async with http_client(Ended()) as http:
+        status = (await http.get("/agentenv/ext/computer-ready/status")).json()
+    assert status["ready"] is False and status["failed"] == "browser: it ended on its own"
+    assert status["checks"][-2:] == [{"name": "browser", "ok": False, "detail": "it ended on its own", "final": True},
+                                     {"name": "session", "ok": True, "detail": "logged in", "final": True}]
 
 
 @pytest.mark.asyncio

{agentenv_framework_protocol-0.1.443 → agentenv_framework_protocol-0.1.444}/tests/test_website_browser.py

@@ -19,10 +19,10 @@ import pytest
 from agentenv_protocol.computer import APP_SETTINGS_ENV, ComputerAppEnvironment, Launch, Ready, app, linux, session
 from agentenv_protocol import website_browser
 from agentenv_protocol.computer import app_runtime
+from agentenv_protocol.computer.errors import DeviceUnavailable, InvalidArgument
 from agentenv_protocol.computer.platform import Platform
-from agentenv_protocol.website_browser import (ALONE_BYPASS, ALONE_ENV, BROWSER, COMPUTER_BYPASS, OPEN_WEB_ENV,
-                                               POLICIES, SITE_PROXY_ENV, SKILL, BrowserControl, DevToolsError,
-                                               WebsiteBrowserEnvironment)
+from agentenv_protocol.website_browser import (ALONE_BYPASS, BROWSER, COMPUTER_BYPASS, POLICIES, SITE_PROXY_ENV, SKILL,
+                                               BrowserControl, DevToolsError, WebsiteBrowserEnvironment)
 
 PROXY = "http://127.0.0.1:3128"
 OWNED = ["--user-data-dir", "--disk-cache-dir", "--remote-debugging-pipe", "--no-first-run",
@@ -80,7 +80,7 @@ def browser_home(monkeypatch, tmp_path):
     monkeypatch.setenv("HOME", str(tmp_path / "home"))
     (tmp_path / "home").mkdir()
     monkeypatch.setenv("ENVIRONMENT_NAME", "chromium")
-    for name in (ALONE_ENV, OPEN_WEB_ENV, SITE_PROXY_ENV, APP_SETTINGS_ENV):
+    for name in (SITE_PROXY_ENV, APP_SETTINGS_ENV):
         monkeypatch.delenv(name, raising=False)
     return tmp_path / "home"
 
@@ -89,6 +89,11 @@ def _settings(monkeypatch, **values) -> None:
     monkeypatch.setenv(APP_SETTINGS_ENV, json.dumps({"chromium": values}))
 
 
+def _alone(monkeypatch) -> None:
+    """On a computer with no display, a browser computer's."""
+    monkeypatch.setattr(app, "detect", lambda: Platform(os="linux", display="none"))
+
+
 def test_in_a_computer_its_flags_follow_its_command_then_its_start_tabs_and_its_window_is_waited_for(browser_home):
     launch = Browser().launch
 
@@ -110,7 +115,7 @@ def test_in_a_computer_with_a_site_proxy_only_loopback_bypasses_it(monkeypatch):
 
 
 def test_alone_it_goes_through_its_proxy_headless_at_its_window_and_its_launch_waits_on_no_window(monkeypatch):
-    monkeypatch.setenv(ALONE_ENV, "1")
+    _alone(monkeypatch)
     monkeypatch.setenv(SITE_PROXY_ENV, PROXY)
     _settings(monkeypatch, window={"width": 1024, "height": 768})
 
@@ -130,7 +135,7 @@ def test_alone_its_launch_keeps_the_port_and_command_it_waits_on_and_their_timeo
     class Waiting(WebsiteBrowserEnvironment):
         pass
 
-    monkeypatch.setenv(ALONE_ENV, "1")
+    _alone(monkeypatch)
     monkeypatch.setenv(SITE_PROXY_ENV, PROXY)
 
     waiting = Waiting()
@@ -140,18 +145,8 @@ def test_alone_its_launch_keeps_the_port_and_command_it_waits_on_and_their_timeo
     assert waiting._computer_app.timeout_s == 45
 
 
-def test_alone_on_the_open_web_it_runs_headful(monkeypatch):
-    monkeypatch.setenv(ALONE_ENV, "1")
-    monkeypatch.setenv(OPEN_WEB_ENV, "1")
-    monkeypatch.setenv(SITE_PROXY_ENV, PROXY)
-
-    command = Browser().launch.command
-
-    assert "--window-size=1280,800" in command and not any(flag.startswith("--headless") for flag in command)
-
-
 def test_alone_it_refuses_to_launch_without_a_site_proxy(monkeypatch):
-    monkeypatch.setenv(ALONE_ENV, "1")
+    _alone(monkeypatch)
 
     with pytest.raises(ValueError, match=SITE_PROXY_ENV):
         Browser()
@@ -215,7 +210,7 @@ def test_its_settings_give_its_start_tabs_language_and_time_zone_and_no_start_ur
 @pytest.mark.asyncio
 async def test_alone_its_server_launches_it_at_once_in_its_environment_with_no_display_wait_or_launcher(
         monkeypatch, tmp_path):
-    monkeypatch.setenv(ALONE_ENV, "1")
+    _alone(monkeypatch)
     monkeypatch.setenv(SITE_PROXY_ENV, PROXY)
     monkeypatch.delenv("DISPLAY", raising=False)
     monkeypatch.chdir(tmp_path)
@@ -314,7 +309,7 @@ async def test_a_launch_while_it_runs_opens_a_window_over_its_pipe_and_starts_no
 async def test_headless_it_opens_one_start_tab_from_its_command_line_and_the_rest_over_its_pipe(monkeypatch, tmp_path):
     monkeypatch.chdir(tmp_path)
     monkeypatch.setattr(app_runtime, "LAUNCHED", tmp_path / "launched")
-    monkeypatch.setenv(ALONE_ENV, "1")
+    _alone(monkeypatch)
     monkeypatch.setenv(SITE_PROXY_ENV, PROXY)
     _settings(monkeypatch, start_urls=["https://slack.com", "https://linear.app", "https://github.com"])
     browser = FakeBrowser()
@@ -400,7 +395,7 @@ def test_its_policies_are_agentenvs_with_its_own_over_them_where_its_build_reads
 
 def test_its_card_declares_browser_v1_with_its_build_and_clock_and_alone_its_window(monkeypatch):
     in_computer = Browser().extension_params(BROWSER)
-    monkeypatch.setenv(ALONE_ENV, "1")
+    _alone(monkeypatch)
     monkeypatch.setenv(SITE_PROXY_ENV, PROXY)
     alone = Browser().extension_params(BROWSER)
 
@@ -503,6 +498,111 @@ async def test_each_page_it_attaches_to_is_told_to_run_so_a_popup_it_attached_to
 
     assert ("Page.enable", {}, "s-popup") in devtools.calls
     assert ("Runtime.runIfWaitingForDebugger", {}, "s-popup") in devtools.calls
+    assert not any(method == "Emulation.setDeviceMetricsOverride" for method, _, _ in devtools.calls)
+    assert next(params for method, params, _ in devtools.calls if method == "Target.setAutoAttach")[
+        "waitForDebuggerOnStart"] is False
+
+
+@pytest.mark.asyncio
+async def test_alone_each_page_it_attaches_to_is_held_to_its_window_before_it_runs(tmp_path):
+    devtools = StubDevTools()
+    control = BrowserControl(devtools, "agent", (1024, 768))
+    await control.attach()
+    devtools.send("Target.attachedToTarget", {"sessionId": "s-popup", "waitingForDebugger": True, **_page("popup")})
+    await asyncio.sleep(0)
+
+    sent = [method for method, _, session in devtools.calls if session == "s-popup"]
+    assert next(params for method, params, _ in devtools.calls if method == "Target.setAutoAttach")[
+        "waitForDebuggerOnStart"] is True
+    assert sent == ["Page.enable", "Emulation.setDeviceMetricsOverride", "Runtime.runIfWaitingForDebugger"]
+    assert ("Emulation.setDeviceMetricsOverride", {"width": 1024, "height": 768, "deviceScaleFactor": 1,
+                                                   "mobile": False}, "s-popup") in devtools.calls
+
+
+@pytest.mark.asyncio
+async def test_a_page_it_couldnt_prepare_is_still_told_to_run_so_it_never_stays_held():
+    def refused(params, session):
+        raise DevToolsError("Target closed")
+
+    devtools = StubDevTools({"Page.enable": refused})
+    control = BrowserControl(devtools, "agent", (1024, 768))
+    await control.attach()
+    devtools.send("Target.attachedToTarget", {"sessionId": "s-popup", "waitingForDebugger": True, **_page("popup")})
+    await asyncio.sleep(0)
+
+    assert [method for method, _, session in devtools.calls if session == "s-popup"] == [
+        "Page.enable", "Runtime.runIfWaitingForDebugger"]
+
+
+class Painting(StubDevTools):
+    """A browser whose every tab has focus and paints, but those whose sessions are in ``blank``, which never answer a
+    capture; it records each capture's timeout."""
+
+    def __init__(self, blank=()):
+        super().__init__({"Runtime.evaluate": lambda params, session: {"result": {"value": ["visible", True]}}})
+        self.blank, self.timeouts = set(blank), []
+
+    async def call(self, method, params=None, *, session_id=None, timeout_s=None):
+        if method != "Page.captureScreenshot":
+            return await super().call(method, params, session_id=session_id, timeout_s=timeout_s)
+        self.timeouts.append(timeout_s)
+        if session_id in self.blank:
+            raise asyncio.TimeoutError
+        return {"data": f"png of {session_id}"}
+
+
+async def _one_tab(devtools, url) -> BrowserControl:
+    control = BrowserControl(devtools, "agent")
+    await control.attach()
+    devtools.send("Target.targetCreated", _page("a", url))
+    devtools.send("Target.attachedToTarget", {"sessionId": "s-a", **_page("a", url)})
+    await asyncio.sleep(0)
+    return control
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("url, blank, png, timeout", [
+    ("https://mail.test/", (), "png of s-a", 3.0),
+    ("https://mail.test/", ("s-a",), None, 3.0),
+    ("", ("s-a",), None, 1.0),
+])
+async def test_its_capture_is_the_active_tabs_page_or_none_for_one_that_paints_nothing_sooner_with_no_url(
+        url, blank, png, timeout):
+    devtools = Painting(blank)
+    control = await _one_tab(devtools, url)
+
+    assert await control.capture() == png and devtools.timeouts == [timeout]
+
+
+@pytest.mark.asyncio
+async def test_its_page_input_goes_to_the_active_tab_in_order_and_takes_only_input_calls():
+    devtools = Painting()
+    control = await _one_tab(devtools, "https://mail.test/")
+    events = [{"method": "Input.dispatchMouseEvent", "params": {"type": "mousePressed", "x": 1, "y": 2}},
+              {"method": "Input.insertText", "params": {"text": "hi"}}]
+
+    await control.dispatch(events)
+    with pytest.raises(InvalidArgument, match="Page.navigate"):
+        await control.dispatch([*events, {"method": "Page.navigate", "params": {"url": "https://elsewhere.test/"}}])
+
+    sent = [(method, params, session) for method, params, session in devtools.calls if method.startswith("Input.")]
+    assert sent == [("Input.dispatchMouseEvent", {"type": "mousePressed", "x": 1, "y": 2}, "s-a"),
+                    ("Input.insertText", {"text": "hi"}, "s-a")]
+
+
+@pytest.mark.asyncio
+async def test_with_no_tab_open_its_capture_and_input_are_unavailable():
+    control = BrowserControl(Painting(), "agent")
+    await control.attach()
+
+    with pytest.raises(DeviceUnavailable, match="no tab open"):
+        await control.capture()
+    with pytest.raises(DeviceUnavailable, match="no tab open"):
+        await control.dispatch([])
+
+
+def test_its_declaration_says_it_is_a_website_browser_and_another_apps_says_nothing():
+    assert Browser.declaration().browser is True and Mail.declaration().browser is None
 
 
 @pytest.mark.asyncio