agentenv-framework-protocol 0.1.451__tar.gz → 0.1.454__tar.gz

{agentenv_framework_protocol-0.1.451 → agentenv_framework_protocol-0.1.454}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentenv-framework-protocol
-Version: 0.1.451
+Version: 0.1.454
 Summary: Open data-plane protocol and server SDK for agent environments.
 Project-URL: Homepage, https://www.agentenvframework.com
 Project-URL: Documentation, https://www.agentenvframework.com/docs
@@ -37,6 +37,7 @@ Requires-Dist: uvicorn>=0.30; extra == 'agent'
 Provides-Extra: app
 Requires-Dist: mcp<2,>=1.25; extra == 'app'
 Requires-Dist: uvicorn>=0.30; extra == 'app'
+Requires-Dist: websockets>=13; extra == 'app'
 Provides-Extra: computer
 Requires-Dist: av>=17; extra == 'computer'
 Requires-Dist: mcp<2,>=1.25; extra == 'computer'
@@ -208,7 +209,8 @@ app without one, so a server that mounted is a server whose tool calls are bound
 To answer a call before the tool runs, override `on_tool_call(self, context)`: return `None` to let
 it proceed, or an `mcp.types.CallToolResult` (`isError=True` for a refusal) to send that as the
 tool's reply. An `@extension` method may declare the same parameter; it is filled from the request's
-`AgentEnv-Role` header. Code that acts as a caller outside any dispatch, such as a test calling a
+`AgentEnv-Role` header. It may also declare a parameter annotated `starlette.requests.Request`, which
+gets the HTTP request itself, its headers say, and stays out of the advertised schema. Code that acts as a caller outside any dispatch, such as a test calling a
 handler directly or a seeding script, binds a context it builds:
 `with bound(ToolContext.as_caller("alice")): ...`.
 
@@ -839,6 +841,11 @@ requests to the app's server, taking the control token:
   closes, the other then closed with its code; refused before they open without the token, or with no
   such app or one not answering. The `computer` extra brings `websockets` for this.
 
+An app's `agent_routes` (`/cdp/` for a website browser) are path prefixes of its own server that
+agents reach this way without the control token: the driver reads them from the app's card
+(`computer-app/v1`'s `agent_routes`) and lets a call under one through, while the app checks it
+itself. A path with a `.` or `..` segment, escaped or not, never counts as under one.
+
 `MailApp.declaration()` returns what the class declares, an `AppDeclaration`, which a `computer_app`
 env stores (`browser: true` for a website browser), and `python -m agentenv_protocol.computer._declare <folder> <file> <class>` prints it as
 JSON, or a driver's `DriverDeclaration`, without the class's own dependencies installed.
@@ -918,6 +925,38 @@ are the profile's template. The base class adds:
   - `files/choose {paths}`, files in `~/Uploads` for the file chooser open on it, which headless, and so
     alone, it intercepts;
   - `page/state`, the active tab's id, URL and title and the dialog or file chooser open on it.
+  - `cdp {user}`, `user`'s CDP address once the browser is ready, `http_url` to discover from and
+    `ws_url` to connect to, both at the address the call reached it at with a token agentenv mints in
+    the path: the same one each time it's asked for. Through the driver they're under
+    `/svc/<app>/cdp/<token>/`. `user` is a name the caller chooses, an agent's `deploy_agent` step id.
+    The server keeps each token in memory and only its SHA-256, with its user and base address, in
+    `.agentenv/cdp-tokens.json`, which code in the computer's session can read but not use; an address
+    outlives a relaunch of the browser and of the server, and the computer's reset ends it.
+- **Its CDP relay**, at those addresses: `json/version` and `json/list` (also with a trailing slash,
+  and `json`) answered from the browser, since a browser on a pipe has none, with its WebSocket at the
+  address the client reached it at; and `devtools/browser`, where each WebSocket connection gets a
+  browser-level session of its own over the pipe (`Target.attachToBrowserTarget`), with its answers
+  and the events of its sessions and those it attaches, and no one else's. Only flattened sessions are
+  served, as Playwright, Puppeteer and Browser Use use. A token agentenv didn't mint is refused, and
+  when the browser goes, each connection is closed with 1012 so its client can connect again.
+  A guard sits on every command, a guardrail, since the sandbox's network is the boundary:
+  - `Browser.close`, `Browser.crash` and `Browser.crashGpuProcess` are answered `{}` and end only that
+    connection, with 1000;
+  - `Target.exposeDevToolsProtocol`, `Target.sendMessageToTarget` (only flattened sessions are
+    served, whose commands the guard sees), a `Target.createBrowserContext` with its own proxy, and a
+    `Page.navigate` or `Target.createTarget` to the browser's own pages (`file:`, `chrome:`,
+    `chrome-extension:`, `chrome-untrusted:`, `view-source:`, `devtools:`, and `about:` but
+    `about:blank` and `about:srcdoc`) get a CDP error, `agentenv refuses …`;
+  - `Browser.setDownloadBehavior` and `Page.setDownloadBehavior` are sent on as agentenv's own, the
+    browser's default into `~/Downloads`, keeping only the context and whether to send events;
+  - when the deploy answers dialogs, an agent's `Page.handleJavaScriptDialog` is answered `{}` and not
+    sent on;
+  - a message that isn't a command, an object with a string `method`, and object `params` and a string
+    `sessionId` if any, gets a CDP error with code -32600.
+
+  `record` lists each connection as `cdp_connections`: its user, when it opened and when it closed,
+  and the first 100 commands the guard refused, changed, answered or closed it on.
+  Puppeteer connects by `ws_url`: its `browserURL` drops the address's path.
 - **Its skill**: sites by their real names, files to upload in `~/Uploads` and downloads in
   `~/Downloads`, folders it makes.
 

{agentenv_framework_protocol-0.1.451 → agentenv_framework_protocol-0.1.454}/README.md

@@ -150,7 +150,8 @@ app without one, so a server that mounted is a server whose tool calls are bound
 To answer a call before the tool runs, override `on_tool_call(self, context)`: return `None` to let
 it proceed, or an `mcp.types.CallToolResult` (`isError=True` for a refusal) to send that as the
 tool's reply. An `@extension` method may declare the same parameter; it is filled from the request's
-`AgentEnv-Role` header. Code that acts as a caller outside any dispatch, such as a test calling a
+`AgentEnv-Role` header. It may also declare a parameter annotated `starlette.requests.Request`, which
+gets the HTTP request itself, its headers say, and stays out of the advertised schema. Code that acts as a caller outside any dispatch, such as a test calling a
 handler directly or a seeding script, binds a context it builds:
 `with bound(ToolContext.as_caller("alice")): ...`.
 
@@ -781,6 +782,11 @@ requests to the app's server, taking the control token:
   closes, the other then closed with its code; refused before they open without the token, or with no
   such app or one not answering. The `computer` extra brings `websockets` for this.
 
+An app's `agent_routes` (`/cdp/` for a website browser) are path prefixes of its own server that
+agents reach this way without the control token: the driver reads them from the app's card
+(`computer-app/v1`'s `agent_routes`) and lets a call under one through, while the app checks it
+itself. A path with a `.` or `..` segment, escaped or not, never counts as under one.
+
 `MailApp.declaration()` returns what the class declares, an `AppDeclaration`, which a `computer_app`
 env stores (`browser: true` for a website browser), and `python -m agentenv_protocol.computer._declare <folder> <file> <class>` prints it as
 JSON, or a driver's `DriverDeclaration`, without the class's own dependencies installed.
@@ -860,6 +866,38 @@ are the profile's template. The base class adds:
   - `files/choose {paths}`, files in `~/Uploads` for the file chooser open on it, which headless, and so
     alone, it intercepts;
   - `page/state`, the active tab's id, URL and title and the dialog or file chooser open on it.
+  - `cdp {user}`, `user`'s CDP address once the browser is ready, `http_url` to discover from and
+    `ws_url` to connect to, both at the address the call reached it at with a token agentenv mints in
+    the path: the same one each time it's asked for. Through the driver they're under
+    `/svc/<app>/cdp/<token>/`. `user` is a name the caller chooses, an agent's `deploy_agent` step id.
+    The server keeps each token in memory and only its SHA-256, with its user and base address, in
+    `.agentenv/cdp-tokens.json`, which code in the computer's session can read but not use; an address
+    outlives a relaunch of the browser and of the server, and the computer's reset ends it.
+- **Its CDP relay**, at those addresses: `json/version` and `json/list` (also with a trailing slash,
+  and `json`) answered from the browser, since a browser on a pipe has none, with its WebSocket at the
+  address the client reached it at; and `devtools/browser`, where each WebSocket connection gets a
+  browser-level session of its own over the pipe (`Target.attachToBrowserTarget`), with its answers
+  and the events of its sessions and those it attaches, and no one else's. Only flattened sessions are
+  served, as Playwright, Puppeteer and Browser Use use. A token agentenv didn't mint is refused, and
+  when the browser goes, each connection is closed with 1012 so its client can connect again.
+  A guard sits on every command, a guardrail, since the sandbox's network is the boundary:
+  - `Browser.close`, `Browser.crash` and `Browser.crashGpuProcess` are answered `{}` and end only that
+    connection, with 1000;
+  - `Target.exposeDevToolsProtocol`, `Target.sendMessageToTarget` (only flattened sessions are
+    served, whose commands the guard sees), a `Target.createBrowserContext` with its own proxy, and a
+    `Page.navigate` or `Target.createTarget` to the browser's own pages (`file:`, `chrome:`,
+    `chrome-extension:`, `chrome-untrusted:`, `view-source:`, `devtools:`, and `about:` but
+    `about:blank` and `about:srcdoc`) get a CDP error, `agentenv refuses …`;
+  - `Browser.setDownloadBehavior` and `Page.setDownloadBehavior` are sent on as agentenv's own, the
+    browser's default into `~/Downloads`, keeping only the context and whether to send events;
+  - when the deploy answers dialogs, an agent's `Page.handleJavaScriptDialog` is answered `{}` and not
+    sent on;
+  - a message that isn't a command, an object with a string `method`, and object `params` and a string
+    `sessionId` if any, gets a CDP error with code -32600.
+
+  `record` lists each connection as `cdp_connections`: its user, when it opened and when it closed,
+  and the first 100 commands the guard refused, changed, answered or closed it on.
+  Puppeteer connects by `ws_url`: its `browserURL` drops the address's path.
 - **Its skill**: sites by their real names, files to upload in `~/Uploads` and downloads in
   `~/Downloads`, folders it makes.
 

{agentenv_framework_protocol-0.1.451 → agentenv_framework_protocol-0.1.454}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "agentenv-framework-protocol"
-version = "0.1.451"
+version = "0.1.454"
 description = "Open data-plane protocol and server SDK for agent environments."
 readme = "README.md"
 license = "Apache-2.0"
@@ -36,10 +36,11 @@ agent = [
     "uvicorn>=0.30",
 ]
 # A computer app's own server: ComputerAppEnvironment serves its card and computer-app/v1 over mcp's FastMCP, on a Unix
-# socket through uvicorn.
+# socket through uvicorn, and a website browser's CDP relay over WebSockets.
 app = [
     "mcp>=1.25,<2",
     "uvicorn>=0.30",
+    "websockets>=13",
 ]
 # A computer driver: ComputerDriverEnvironment serves its tools over mcp's FastMCP, crops and encodes frames, records
 # the display as H.264 with PyAV (computer-recording/v1), and relays WebSockets to its apps on /svc/<app>/.

{agentenv_framework_protocol-0.1.451 → agentenv_framework_protocol-0.1.454}/src/agentenv_protocol/agent_env_environment.py

@@ -22,7 +22,7 @@ from pydantic import TypeAdapter, ValidationError
 from pydantic.fields import FieldInfo
 from starlette.requests import Request
 from starlette.responses import Response
-from starlette.routing import Route
+from starlette.routing import Route, WebSocketRoute
 
 from .errors import AgentEnvError
 from .tool_access import DISABLE as TOOL_DISABLE, ENABLE as TOOL_ENABLE, PATHS as TOOL_ACCESS_PATHS, ToolAccess, caller_role
@@ -797,6 +797,19 @@ def _is_tool_context_annotation(annotation: Any) -> bool:
     return any(t is ToolContext for t in _unwrap_annotation(annotation))
 
 
+def _is_request_annotation(annotation: Any) -> bool:
+    if isinstance(annotation, str):
+        return Request.__name__ in _string_annotation_names(annotation)
+    return any(t is Request for t in _unwrap_annotation(annotation))
+
+
+def _request_slot(fn: Callable) -> str | None:
+    """Name of an extension handler's parameter annotated ``Request``, which receives the HTTP request, or None."""
+    hints = _resolved_hints(fn)
+    return next((name for name, param in inspect.signature(fn).parameters.items() if name not in ("self", "cls")
+                 and _is_request_annotation(hints.get(name, param.annotation))), None)
+
+
 def _fastmcp_context_type() -> type | None:
     """FastMCP's Context class when mcp is loaded, else None. mcp is an optional dependency, so it is
     looked up rather than imported; a Context-annotated handler has necessarily imported it already."""
@@ -1034,7 +1047,8 @@ def _schema_from_signature(fn: Callable) -> dict:
         if name in ("self", "cls") or param.kind in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD):
             continue
         annotation = hints.get(name, param.annotation)
-        if _is_tool_context_annotation(annotation) or _is_fastmcp_context_annotation(annotation):
+        if _is_tool_context_annotation(annotation) or _is_fastmcp_context_annotation(annotation) or \
+                _is_request_annotation(annotation):
             continue
         prop = _schema_for_annotation(annotation)
         if param.default is inspect.Parameter.empty:
@@ -1505,12 +1519,12 @@ def _coerce_params(fn: Callable, params: dict) -> dict:
 
 def _extension_handler(fn: Callable, http_method: str) -> Callable:
     """REST handler for an @extension route: bind request params to the handler and return JSON. A
-    ``ToolContext`` parameter is filled from the request headers and never bound
-    from the params."""
-    slot = _tool_context_slot(fn)
+    ``ToolContext`` parameter is filled from the request headers, and a ``Request`` one with the request itself;
+    neither is bound from the params."""
+    slot, request_slot = _tool_context_slot(fn), _request_slot(fn)
     signature = inspect.signature(fn)
-    if slot:
-        signature = signature.replace(parameters=[p for p in signature.parameters.values() if p.name != slot])
+    signature = signature.replace(parameters=[p for p in signature.parameters.values()
+                                              if p.name not in (slot, request_slot)])
     op = fn.__name__ or "invoke"
 
     async def handler(request: Request) -> Response:
@@ -1530,7 +1544,8 @@ def _extension_handler(fn: Callable, http_method: str) -> Callable:
         except TypeError as e:
             return _error_response(AgentEnvError("invalid_params", str(e)))
         with bound(context_from_http(request, op, params)) as tool_context:
-            call_kwargs = {**params, slot: tool_context} if slot else params
+            call_kwargs = {**params, **({slot: tool_context} if slot else {}),
+                           **({request_slot: request} if request_slot else {})}
             try:
                 result = await _maybe_await(fn(**call_kwargs))
             except AgentEnvError as e:
@@ -1574,6 +1589,16 @@ def control_guard() -> Callable[[Callable], Callable]:
     return guard
 
 
+def add_websocket_route(app: Any, path: str, endpoint: Callable) -> None:
+    """``endpoint`` served at ``path`` as a WebSocket route of ``app``, an mcp or a standalone fastmcp FastMCP, which
+    have no API for one."""
+    for routes in ("_custom_starlette_routes", "_additional_http_routes"):
+        if isinstance(getattr(app, routes, None), list):
+            getattr(app, routes).append(WebSocketRoute(path, endpoint))
+            return
+    raise TypeError(f"WebSocket routes go on an mcp or a fastmcp FastMCP, not a {type(app).__name__}")
+
+
 def _tool_access_handler(access: ToolAccess, action: str) -> Callable:
     async def handler(request: Request) -> Response:
         raw = await request.body()

agentenv_framework_protocol-0.1.454/src/agentenv_protocol/browser_cdp.py

@@ -0,0 +1,339 @@
+"""The CDP addresses a website browser hands agents: a token agentenv mints for each user, of which only a hash is kept
+where the computer's session can read it, discovery answered from the browser, and a relay that gives each WebSocket
+connection its own browser-level session over the browser's one debugging pipe, with a guard on every command."""
+from __future__ import annotations
+
+import asyncio
+import contextlib
+import hashlib
+import json
+import logging
+import os
+import secrets
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, Literal, Optional
+
+from starlette.requests import Request
+from starlette.websockets import WebSocket, WebSocketDisconnect
+
+if TYPE_CHECKING:
+    from .website_browser import DevToolsPipe
+
+logger = logging.getLogger(__name__)
+
+#: How many random bytes a token holds.
+TOKEN_BYTES = 32
+#: The route agents reach the relay under, through the driver's /svc/<app>/.
+ROUTE = "/cdp/"
+#: The close code and reason a connection gets when its browser goes away, which a client may connect again after.
+RELAUNCHED = (1012, "the browser relaunched: connect again")
+#: The close code and reason a connection gets when its client falls behind by more than ``OUTBOX_LIMIT_BYTES``.
+TOO_SLOW = (1013, "the client fell too far behind: connect again")
+#: How much a connection's client may have waiting to be sent to it before it is closed; a single answer bigger than
+#: this still goes when nothing else waits.
+OUTBOX_LIMIT_BYTES = 64 << 20
+#: The close code and reason a connection gets after it closes or crashes the browser, which ends only it.
+CLOSED = (1000, "the browser stays up for the env: only this connection ended")
+#: CDP's own answer to a command for a session the connection doesn't have.
+NO_SESSION = {"code": -32001, "message": "Session with given id not found."}
+#: The code of a CDP error for a command not carried out, as Chromium answers its own refusals.
+SERVER_ERROR = -32000
+#: The answer to a message that isn't a command the relay can read, with JSON-RPC's code for an invalid request.
+MALFORMED = {"code": -32600, "message": "a command is an object with a string method, and object params and a "
+                                        "string sessionId if any"}
+#: The commands that would end the browser every connection shares: each ends only the connection that sends it.
+ENDS_BROWSER = frozenset({"Browser.close", "Browser.crash", "Browser.crashGpuProcess"})
+#: The commands that set how and where downloads are saved, which agentenv keeps as its own.
+SETS_DOWNLOADS = frozenset({"Browser.setDownloadBehavior", "Page.setDownloadBehavior"})
+#: The schemes of the browser's own files, settings and tools, which an agent's navigation or new tab may not open.
+OWN_SCHEMES = frozenset({"file", "chrome", "chrome-extension", "chrome-untrusted", "view-source", "devtools"})
+#: The ``about:`` pages that aren't the browser's own under another name.
+ABOUT_PAGES = frozenset({"about:blank", "about:srcdoc"})
+#: How many of a connection's guarded commands its record keeps.
+GUARDED_KEPT = 100
+#: What the record says the guard did, by its action.
+GUARDED = {"refuse": "refused", "send": "changed", "answer": "answered", "close": "closed"}
+
+
+@dataclass
+class Address:
+    """A user's CDP address: ``http_url``, the base clients discover from, and ``ws_url``, the browser's WebSocket."""
+
+    user: str
+    http_url: str
+    ws_url: str
+
+
+def reached(request: Request) -> str:
+    """The address a request reached the browser at, scheme, host and path prefix: what the driver forwarded, else the
+    request's own."""
+    proto = request.headers.get("x-forwarded-proto") or request.url.scheme
+    host = request.headers.get("x-forwarded-host") or request.headers.get("host") or request.url.netloc
+    return f"{proto}://{host}{request.headers.get('x-forwarded-prefix', '')}".rstrip("/")
+
+
+def websocket_form(url: str) -> str:
+    return "wss://" + url.removeprefix("https://") if url.startswith("https://") else \
+        "ws://" + url.removeprefix("http://")
+
+
+def _hash(token: str) -> str:
+    return hashlib.sha256(token.encode()).hexdigest()
+
+
+def _address(user: str, base: str, token: str) -> Address:
+    root = f"{base}{ROUTE}{token}"
+    return Address(user, f"{root}/", f"{websocket_form(root)}/devtools/browser")
+
+
+class Tokens:
+    """The tokens minted, one per user: in memory, so asking twice gives the same address, and in ``path`` only by
+    hash, with the user and the base address, so an address still works after the server restarts while the
+    computer's session, which can read the file, can't use one."""
+
+    def __init__(self, path: Path) -> None:
+        self.path = path
+        self._minted: dict[str, Address] = {}
+        try:
+            stored = json.loads(path.read_text())
+            self._known = {digest: (entry["user"], entry["base"]) for digest, entry in stored.items()}
+        except (OSError, ValueError, TypeError, KeyError):
+            self._known = {}
+
+    def mint(self, user: str, base: str) -> Address:
+        """``user``'s address under ``base``: the one minted before, else a new one."""
+        if user not in self._minted:
+            token = secrets.token_urlsafe(TOKEN_BYTES)
+            self._minted[user] = _address(user, base, token)
+            self._known[_hash(token)] = (user, base)
+            self._save()
+        return self._minted[user]
+
+    def address(self, token: str) -> Optional[Address]:
+        """The address ``token`` was minted for, or None for one that wasn't."""
+        known = self._known.get(_hash(token))
+        return _address(*known, token) if known is not None else None
+
+    def _save(self) -> None:
+        self.path.parent.mkdir(parents=True, exist_ok=True)
+        part = self.path.with_name(f".{self.path.name}.part")
+        part.write_text(json.dumps({digest: {"user": user, "base": base}
+                                    for digest, (user, base) in self._known.items()}))
+        os.replace(part, self.path)
+
+
+@dataclass
+class Verdict:
+    """What the guard does with an agent's command: ``send`` it on with ``params``, ``changed`` or not; ``answer`` it
+    with ``{}`` itself; answer it and ``close`` the connection; or ``refuse`` it for ``reason``."""
+
+    action: Literal["send", "answer", "close", "refuse"]
+    params: dict
+    reason: str = ""
+    changed: bool = False
+
+
+def guard(method: str, params: dict, *, dialogs_answered: bool) -> Verdict:
+    """What the relay does with an agent's ``method``, so it can't end the browser others share, route around the site
+    proxy, save downloads elsewhere, open the browser's own pages or race the dialogs agentenv answers when
+    ``dialogs_answered``: a guardrail, since the sandbox's network is the boundary."""
+    if method in ENDS_BROWSER:
+        return Verdict("close", params)
+    if method == "Target.exposeDevToolsProtocol":
+        return Verdict("refuse", params, "it would give a page's script the browser's own CDP channel")
+    if method == "Target.sendMessageToTarget":
+        return Verdict("refuse", params, "only flattened sessions are served, whose commands the guard sees")
+    if method == "Target.createBrowserContext" and {"proxyServer", "proxyBypassList"} & params.keys():
+        return Verdict("refuse", params, "a context's traffic goes through the env's site proxy")
+    if method in ("Page.navigate", "Target.createTarget") and (scheme := _own_scheme(params.get("url"))):
+        return Verdict("refuse", params, f"{scheme}: URLs open the browser's own files, settings and tools")
+    if method in SETS_DOWNLOADS:
+        pinned = {"behavior": "default",
+                  **{key: params[key] for key in ("browserContextId", "eventsEnabled") if key in params}}
+        return Verdict("send", pinned, changed=pinned != params)
+    if method == "Page.handleJavaScriptDialog" and dialogs_answered:
+        return Verdict("answer", params)
+    return Verdict("send", params)
+
+
+def _own_scheme(url: Any) -> Optional[str]:
+    """``url``'s scheme when it's one of the browser's own, read as the browser reads it, without tabs and newlines and
+    with spaces and controls trimmed from its ends; an ``about:`` page other than a blank one counts."""
+    if not isinstance(url, str):
+        return None
+    url = url.translate({ord(c): None for c in "\t\n\r"}).strip("".join(map(chr, range(0x21))))
+    scheme, colon, _ = url.partition(":")
+    scheme = scheme.lower()
+    if not colon:
+        return None
+    if scheme == "about":
+        return None if url.lower().split("#")[0].split("?")[0] in ABOUT_PAGES else scheme
+    return scheme if scheme in OWN_SCHEMES else None
+
+
+class _Connection:
+    """One client's WebSocket, its sessions over the pipe (its browser-level one and those it attached), its unanswered
+    commands, what waits to be sent to it, and its entry in the record."""
+
+    def __init__(self, websocket: WebSocket, user: str, session: str, record: dict) -> None:
+        self.websocket, self.user, self.session, self.record = websocket, user, session, record
+        self.sessions = {session}
+        self.commands: set[asyncio.Future] = set()
+        self.outbox: asyncio.Queue[str | tuple[int, str]] = asyncio.Queue()
+        self.waiting_bytes = 0
+        self.closing = False
+
+
+class Relay:
+    """Each WebSocket connection given its own browser-level session over ``devtools``, its commands sent on it past the
+    guard and the answers and events of its sessions sent back, so several clients share one browser; and a record of
+    who connected when, and what the guard did to their commands. ``dialogs_answered`` says agentenv answers the
+    browser's dialogs itself."""
+
+    def __init__(self, devtools: DevToolsPipe, *, dialogs_answered: bool = False) -> None:
+        self.devtools, self.dialogs_answered = devtools, dialogs_answered
+        self.connections: list[dict] = []
+        self._by_session: dict[str, _Connection] = {}
+        self._open: set[_Connection] = set()
+        devtools.listen(self._event)
+
+    async def serve(self, websocket: WebSocket, user: str) -> None:
+        """``websocket``, a client of ``user``'s, relayed over a browser-level session of its own until either ends."""
+        try:
+            session = (await self.devtools.call("Target.attachToBrowserTarget"))["sessionId"]
+        except Exception as e:
+            await websocket.close(code=1013, reason=f"the browser isn't answering: {type(e).__name__}")
+            return
+        record = {"user": user, "opened_at": _now(), "guarded": []}
+        connection = _Connection(websocket, user, session, record)
+        self._by_session[session] = connection
+        self._open.add(connection)
+        self.connections.append(record)
+        await websocket.accept()
+        sending = asyncio.ensure_future(self._send(connection))
+        try:
+            while True:
+                message = await websocket.receive()
+                if message["type"] == "websocket.disconnect":
+                    break
+                text = message.get("text") if message.get("text") is not None else message["bytes"].decode()
+                command = asyncio.ensure_future(self._command(connection, json.loads(text)))
+                connection.commands.add(command)
+                command.add_done_callback(connection.commands.discard)
+        except (WebSocketDisconnect, ValueError):
+            pass
+        finally:
+            record["closed_at"] = _now()
+            connection.closing = True
+            self._end(connection)
+            for task in [*connection.commands, sending]:
+                task.cancel()
+            await asyncio.gather(*connection.commands, sending, return_exceptions=True)
+            with contextlib.suppress(Exception):
+                await self.devtools.call("Target.detachFromTarget", {"sessionId": session})
+
+    def ended(self) -> None:
+        """The browser has gone: every connection is closed with a reason a client may connect again after."""
+        for connection in list(self._open):
+            self._close(connection, RELAUNCHED)
+
+    async def _command(self, connection: _Connection, message: Any) -> None:
+        if connection.closing:
+            return
+        if not _well_formed(message):
+            self._put(connection, {"id": message.get("id") if isinstance(message, dict) else None, "error": MALFORMED})
+            return
+        method, params = message["method"], message.get("params") or {}
+        session = message.get("sessionId") or connection.session
+        verdict = guard(method, params, dialogs_answered=self.dialogs_answered)
+        if session not in connection.sessions:
+            verdict, answer = None, {"error": NO_SESSION}
+        elif verdict.action == "refuse":
+            answer = {"error": {"code": SERVER_ERROR, "message": f"agentenv refuses {method}: {verdict.reason}"}}
+        elif verdict.action != "send":
+            answer = {"result": {}}
+        else:
+            try:
+                answer = await self.devtools.request(method, verdict.params, session_id=session)
+            except ConnectionError as e:
+                answer = {"error": {"code": SERVER_ERROR, "message": str(e)}}
+        if verdict is not None and (verdict.action != "send" or verdict.changed):
+            self._guarded(connection, method, verdict)
+        reply = {"id": message.get("id"), **{key: answer[key] for key in ("result", "error") if key in answer}}
+        if message.get("sessionId"):
+            reply["sessionId"] = message["sessionId"]
+        self._put(connection, reply)
+        if verdict is not None and verdict.action == "close":
+            self._close(connection, CLOSED, dropping=False)
+
+    @staticmethod
+    def _guarded(connection: _Connection, method: str, verdict: Verdict) -> None:
+        """What the guard did to ``connection``'s ``method``, in its record, up to ``GUARDED_KEPT`` of them."""
+        if len(guarded := connection.record["guarded"]) < GUARDED_KEPT:
+            guarded.append({"method": method, "did": GUARDED[verdict.action], "at": _now()})
+
+    def _event(self, message: dict) -> None:
+        connection = self._by_session.get(message.get("sessionId"))
+        if connection is None:
+            return
+        params = message.get("params") or {}
+        if message.get("method") == "Target.attachedToTarget" and params.get("sessionId"):
+            connection.sessions.add(params["sessionId"])
+            self._by_session[params["sessionId"]] = connection
+        elif message.get("method") == "Target.detachedFromTarget" and params.get("sessionId"):
+            connection.sessions.discard(params["sessionId"])
+            self._by_session.pop(params["sessionId"], None)
+        if message["sessionId"] == connection.session:
+            message = {key: value for key, value in message.items() if key != "sessionId"}
+        self._put(connection, message)
+
+    def _put(self, connection: _Connection, message: dict) -> None:
+        """``message`` waiting to be sent to ``connection``'s client, which is closed instead, with what waited dropped,
+        once it has fallen too far behind."""
+        if connection.closing:
+            return
+        text = json.dumps(message)
+        if connection.waiting_bytes and connection.waiting_bytes + len(text) > OUTBOX_LIMIT_BYTES:
+            self._close(connection, TOO_SLOW)
+            return
+        connection.waiting_bytes += len(text)
+        connection.outbox.put_nowait(text)
+
+    def _close(self, connection: _Connection, close: tuple[int, str], *, dropping: bool = True) -> None:
+        """``connection`` closed with ``close`` after what waits to be sent to it, or with that dropped when
+        ``dropping``; nothing is sent after it."""
+        connection.closing = True
+        self._end(connection)
+        if dropping:
+            while not connection.outbox.empty():
+                connection.outbox.get_nowait()
+            connection.waiting_bytes = 0
+        connection.outbox.put_nowait(close)
+
+    def _end(self, connection: _Connection) -> None:
+        self._open.discard(connection)
+        for session in connection.sessions:
+            self._by_session.pop(session, None)
+
+    async def _send(self, connection: _Connection) -> None:
+        with contextlib.suppress(WebSocketDisconnect, RuntimeError):
+            while True:
+                message = await connection.outbox.get()
+                if isinstance(message, tuple):
+                    await connection.websocket.close(code=message[0], reason=message[1])
+                    return
+                connection.waiting_bytes -= len(message)
+                await connection.websocket.send_text(message)
+
+
+def _well_formed(message: Any) -> bool:
+    """Whether ``message`` is a command the relay can read: an object with a string ``method``, and object ``params``
+    and a string ``sessionId`` if any."""
+    return isinstance(message, dict) and isinstance(message.get("method"), str) and \
+        isinstance(message.get("params") or {}, dict) and isinstance(message.get("sessionId") or "", str)
+
+
+def _now() -> str:
+    return datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z")

{agentenv_framework_protocol-0.1.451 → agentenv_framework_protocol-0.1.454}/src/agentenv_protocol/computer/app.py

@@ -374,6 +374,9 @@ class ComputerAppEnvironment(AgentEnvEnvironment):
     skill: ClassVar[Optional[str]] = None
     #: How long one of its data calls may take, a hook or its paths' default, before it fails.
     data_timeout_s: ClassVar[float] = 300.0
+    #: Path prefixes of its own server's routes, each ending in ``/``, that agents reach through the driver without the
+    #: computer's control token; the app checks a call to one itself.
+    agent_routes: ClassVar[tuple[str, ...]] = ()
 
     def __init_subclass__(cls, **kwargs: Any) -> None:
         """Take the class's data hooks over, for the base class's handlers to serve around its contract; and serve its
@@ -473,7 +476,7 @@ class ComputerAppEnvironment(AgentEnvEnvironment):
     def extension_params(self, uri: str) -> dict:
         if uri == COMPUTER_APP:
             return {"launch": self.launch.model_dump(mode="json", exclude_none=True) if self.launch else None,
-                    "paths": self.paths}
+                    "paths": self.paths, **({"agent_routes": list(self.agent_routes)} if self.agent_routes else {})}
         return super().extension_params(uri)
 
     def served_operations(self, operations: list[str]) -> list[str]:

{agentenv_framework_protocol-0.1.451 → agentenv_framework_protocol-0.1.454}/src/agentenv_protocol/computer/apps.py

@@ -84,6 +84,12 @@ def child_card(name: str, card: dict) -> dict:
     return card
 
 
+def agent_routes(card: dict) -> list[str]:
+    """The path prefixes under its own route that an app's ``card`` opens to agents, which reach them through the
+    driver without the computer's control token and the app checks itself."""
+    return list(client.extension_params(card, COMPUTER_APP).get("agent_routes") or [])
+
+
 async def owned_paths(folders: list[Path]) -> list[str]:
     """The paths under the session user's home that the apps in ``folders`` keep their data in, as their cards say."""
     cards = await asyncio.gather(*(served_card(folder) for folder in folders))