agentenv-framework-protocol 0.1.444__tar.gz → 0.1.445__tar.gz

{agentenv_framework_protocol-0.1.444 → agentenv_framework_protocol-0.1.445}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentenv-framework-protocol
-Version: 0.1.444
+Version: 0.1.445
 Summary: Open data-plane protocol and server SDK for agent environments.
 Project-URL: Homepage, https://www.agentenvframework.com
 Project-URL: Documentation, https://www.agentenvframework.com/docs
@@ -529,6 +529,7 @@ pixels), at the display's size times its scale. Input operations may return an `
 | `@set_display` | no, with `display_modes` | `computer-display/v1` `set` (`get` is always served) |
 | `@pointer_move`, `@pointer_down`, `@pointer_up`, `@pointer_position` | all or none | `computer-pointer/v1` and its five tools; click and scroll at the pointer |
 | `@key_down`, `@key_up` | both or none | `computer-keyboard/v1` and its two tools |
+| `@navigate`, `@history`, `@tabs`, `@dialog`, `@upload`, `@page_state` | all or none, for a screen that is a website browser's page | `computer-page/v1` and its five tools; the `page` of every frame |
 | `@run_in_session` | no; a Linux default | `computer-exec/v1` |
 | `@write_files` | no; a Linux default | `data/add` |
 | `@record` | no | the stream `computer-recording/v1` records from; without it the base class polls `@capture` |
@@ -553,13 +554,21 @@ driver has them, through the `served_tools`, `served_extensions` and `served_ope
   group tools. Every result is structured, with a one-line summary as its text block; input results
   carry `applied`, `delivery` and `others_since_frame`, and `screenshot: true` returns a fresh frame
   after acting. Click and scroll leave out `x` and `y` to act at the pointer on a computer with
-  `computer-pointer/v1`; elsewhere a call without them is refused `invalid_argument`.
+  `computer-pointer/v1`; elsewhere a call without them is refused `invalid_argument`. With the page
+  group, `computer_navigate` opens a URL in the active tab, `computer_history` goes `back`, `forward`
+  or `reload`s, `computer_tabs` lists, opens, switches and closes tabs by the browser's own ids and
+  answers with the tabs after it, `computer_dialog` answers the page's open dialog and
+  `computer_upload` gives its open file chooser files in `~/Uploads`.
 - **Frames and spaces.** Points are pixels of a named frame (your latest by default), or a
   0..`norm_scale` grid over it with `space: "norm"`. A point outside the frame or the range is
   refused (`out_of_bounds`), never clamped. The last 64 frames are kept, and a frame goes stale
-  (`stale_frame`) when the display geometry it was captured at changes.
+  (`stale_frame`) when the display geometry it was captured at changes. With the page group, every
+  frame carries `page`: the active tab's `tab`, `url` and `title`, and the `dialog` (`type`, `message`,
+  `default_text`) or `file_chooser` (`multiple`) open on it, which its summary names too.
 - **The input queue.** Input calls run one at a time in arrival order; one that waits past
-  `queue_wait_s` is refused `input_busy`. Screenshots and waits never queue.
+  `queue_wait_s` is refused `input_busy`. Screenshots and waits never queue. While a dialog is open
+  on the page, input it blocks (all but `computer_tabs`, `computer_dialog` and `computer_upload`) is
+  refused `dialog_open`, with `applied: "no"`.
 - **The computer skill.** `computer/v1` declares the seven tools, its params the driver's
   features, among them `screen`: `desktop`, or `page` for a driver whose screen is a website
   browser's active tab, which a computer env pairs with exactly one website browser. The driver serves its skill through `skill/v1` (an `@agent_skill` handler on the base
@@ -855,7 +864,8 @@ are the profile's template. The base class adds:
   each as of its page's last load, with `closed_at` and
   `closed_by` (`agentenv`, over this connection; `unknown`, anyone else; `exit` or `stop`, when the browser
   ended on its own or its server stopped it); `downloads`, into `~/Downloads`, each with its state
-  and, completed, its size and sha256; and `dialogs`, each with its type, message, answer and who gave it.
+  and, completed, its size and sha256; and `dialogs`, each with its type, message, default text, answer
+  and who gave it.
   `hidden_page()` gives agentenv a page of its own for reading state, in no window and never in the record.
 - **`browser/v1`** on its card: `chromium` (its `version`), `clock: "real"` and, alone, `window`; and
   `record` (GET), framework-only, which returns that record with `launches`, the app server's log of each
@@ -864,8 +874,18 @@ are the profile's template. The base class adds:
   Nothing relaunches a browser that exits on its own: in a computer the agent can, and alone it's an infra
   failure, its ready check false. `page/viewport`, `page/capture` and `page/input`, framework-only, are
   its active tab's page as a page driver sees it: its size, which headless is its window's; a PNG of it,
-  or `blank` for a tab that paints nothing within 3 s (1 s with no URL yet); and CDP input calls
-  (`Input.dispatchMouseEvent`, `Input.dispatchKeyEvent`, `Input.insertText`), sent to it in order.
+  or `blank` for a tab that paints nothing within 3 s (1 s with no URL yet), its last capture while
+  a dialog keeps it from painting; and CDP input calls (`Input.dispatchMouseEvent`,
+  `Input.dispatchKeyEvent`, `Input.insertText`), sent to it in order. Also framework-only, for page
+  drivers and agentenv's own steps alike:
+  - `navigate {url}` and `history {direction}` (`back`, `forward` or `reload`), in the active tab;
+  - `tabs/list`, `tabs/open {url?}`, `tabs/activate {tab}` and `tabs/close {tab}`, by the browser's own tab
+    ids. The active tab is the one a dialog is open on, else the one whose page has focus, else the one
+    agentenv last brought to the front;
+  - `dialog/answer {accept, text?}`, the agent's answer to the dialog open on the active tab;
+  - `files/choose {paths}`, files in `~/Uploads` for the file chooser open on it, which headless, and so
+    alone, it intercepts;
+  - `page/state`, the active tab's id, URL and title and the dialog or file chooser open on it.
 - **Its skill**: sites by their real names, files to upload in `~/Uploads` and downloads in
   `~/Downloads`, folders it makes.
 

{agentenv_framework_protocol-0.1.444 → agentenv_framework_protocol-0.1.445}/README.md

@@ -473,6 +473,7 @@ pixels), at the display's size times its scale. Input operations may return an `
 | `@set_display` | no, with `display_modes` | `computer-display/v1` `set` (`get` is always served) |
 | `@pointer_move`, `@pointer_down`, `@pointer_up`, `@pointer_position` | all or none | `computer-pointer/v1` and its five tools; click and scroll at the pointer |
 | `@key_down`, `@key_up` | both or none | `computer-keyboard/v1` and its two tools |
+| `@navigate`, `@history`, `@tabs`, `@dialog`, `@upload`, `@page_state` | all or none, for a screen that is a website browser's page | `computer-page/v1` and its five tools; the `page` of every frame |
 | `@run_in_session` | no; a Linux default | `computer-exec/v1` |
 | `@write_files` | no; a Linux default | `data/add` |
 | `@record` | no | the stream `computer-recording/v1` records from; without it the base class polls `@capture` |
@@ -497,13 +498,21 @@ driver has them, through the `served_tools`, `served_extensions` and `served_ope
   group tools. Every result is structured, with a one-line summary as its text block; input results
   carry `applied`, `delivery` and `others_since_frame`, and `screenshot: true` returns a fresh frame
   after acting. Click and scroll leave out `x` and `y` to act at the pointer on a computer with
-  `computer-pointer/v1`; elsewhere a call without them is refused `invalid_argument`.
+  `computer-pointer/v1`; elsewhere a call without them is refused `invalid_argument`. With the page
+  group, `computer_navigate` opens a URL in the active tab, `computer_history` goes `back`, `forward`
+  or `reload`s, `computer_tabs` lists, opens, switches and closes tabs by the browser's own ids and
+  answers with the tabs after it, `computer_dialog` answers the page's open dialog and
+  `computer_upload` gives its open file chooser files in `~/Uploads`.
 - **Frames and spaces.** Points are pixels of a named frame (your latest by default), or a
   0..`norm_scale` grid over it with `space: "norm"`. A point outside the frame or the range is
   refused (`out_of_bounds`), never clamped. The last 64 frames are kept, and a frame goes stale
-  (`stale_frame`) when the display geometry it was captured at changes.
+  (`stale_frame`) when the display geometry it was captured at changes. With the page group, every
+  frame carries `page`: the active tab's `tab`, `url` and `title`, and the `dialog` (`type`, `message`,
+  `default_text`) or `file_chooser` (`multiple`) open on it, which its summary names too.
 - **The input queue.** Input calls run one at a time in arrival order; one that waits past
-  `queue_wait_s` is refused `input_busy`. Screenshots and waits never queue.
+  `queue_wait_s` is refused `input_busy`. Screenshots and waits never queue. While a dialog is open
+  on the page, input it blocks (all but `computer_tabs`, `computer_dialog` and `computer_upload`) is
+  refused `dialog_open`, with `applied: "no"`.
 - **The computer skill.** `computer/v1` declares the seven tools, its params the driver's
   features, among them `screen`: `desktop`, or `page` for a driver whose screen is a website
   browser's active tab, which a computer env pairs with exactly one website browser. The driver serves its skill through `skill/v1` (an `@agent_skill` handler on the base
@@ -799,7 +808,8 @@ are the profile's template. The base class adds:
   each as of its page's last load, with `closed_at` and
   `closed_by` (`agentenv`, over this connection; `unknown`, anyone else; `exit` or `stop`, when the browser
   ended on its own or its server stopped it); `downloads`, into `~/Downloads`, each with its state
-  and, completed, its size and sha256; and `dialogs`, each with its type, message, answer and who gave it.
+  and, completed, its size and sha256; and `dialogs`, each with its type, message, default text, answer
+  and who gave it.
   `hidden_page()` gives agentenv a page of its own for reading state, in no window and never in the record.
 - **`browser/v1`** on its card: `chromium` (its `version`), `clock: "real"` and, alone, `window`; and
   `record` (GET), framework-only, which returns that record with `launches`, the app server's log of each
@@ -808,8 +818,18 @@ are the profile's template. The base class adds:
   Nothing relaunches a browser that exits on its own: in a computer the agent can, and alone it's an infra
   failure, its ready check false. `page/viewport`, `page/capture` and `page/input`, framework-only, are
   its active tab's page as a page driver sees it: its size, which headless is its window's; a PNG of it,
-  or `blank` for a tab that paints nothing within 3 s (1 s with no URL yet); and CDP input calls
-  (`Input.dispatchMouseEvent`, `Input.dispatchKeyEvent`, `Input.insertText`), sent to it in order.
+  or `blank` for a tab that paints nothing within 3 s (1 s with no URL yet), its last capture while
+  a dialog keeps it from painting; and CDP input calls (`Input.dispatchMouseEvent`,
+  `Input.dispatchKeyEvent`, `Input.insertText`), sent to it in order. Also framework-only, for page
+  drivers and agentenv's own steps alike:
+  - `navigate {url}` and `history {direction}` (`back`, `forward` or `reload`), in the active tab;
+  - `tabs/list`, `tabs/open {url?}`, `tabs/activate {tab}` and `tabs/close {tab}`, by the browser's own tab
+    ids. The active tab is the one a dialog is open on, else the one whose page has focus, else the one
+    agentenv last brought to the front;
+  - `dialog/answer {accept, text?}`, the agent's answer to the dialog open on the active tab;
+  - `files/choose {paths}`, files in `~/Uploads` for the file chooser open on it, which headless, and so
+    alone, it intercepts;
+  - `page/state`, the active tab's id, URL and title and the dialog or file chooser open on it.
 - **Its skill**: sites by their real names, files to upload in `~/Uploads` and downloads in
   `~/Downloads`, folders it makes.
 

{agentenv_framework_protocol-0.1.444 → agentenv_framework_protocol-0.1.445}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "agentenv-framework-protocol"
-version = "0.1.444"
+version = "0.1.445"
 description = "Open data-plane protocol and server SDK for agent environments."
 readme = "README.md"
 license = "Apache-2.0"

{agentenv_framework_protocol-0.1.444 → agentenv_framework_protocol-0.1.445}/src/agentenv_protocol/computer/__init__.py

@@ -32,6 +32,7 @@ from .errors import (
     ComputerError,
     DeliveryFailed,
     DeviceUnavailable,
+    DialogOpen,
     InputBusy,
     Internal,
     InvalidArgument,
@@ -47,14 +48,22 @@ from .operations import (
     Display,
     Effect,
     ExecResult,
+    FileChooser,
     Image,
+    PageDialog,
+    PageState,
     ReadyCheck,
+    Tab,
     capture,
     click,
+    dialog,
     display,
     drag,
+    history,
     key_down,
     key_up,
+    navigate,
+    page_state,
     pointer_down,
     pointer_move,
     pointer_position,
@@ -64,7 +73,9 @@ from .operations import (
     run_in_session,
     scroll,
     set_display,
+    tabs,
     type_text,
+    upload,
     write_files,
 )
 from .dataplane import part_bytes
@@ -79,6 +90,7 @@ if TYPE_CHECKING:
         COMPUTER_DISPLAY,
         COMPUTER_EXEC,
         COMPUTER_KEYBOARD,
+        COMPUTER_PAGE,
         COMPUTER_POINTER,
         COMPUTER_READY,
         COMPUTER_RECORDING,
@@ -91,7 +103,7 @@ if TYPE_CHECKING:
 _ON_USE = {
     "driver": ("ComputerDriverEnvironment", "computer_driver", "DEFAULT_IDENTITY", "DriverDeclaration"),
     "card": ("COMPUTER", "COMPUTER_READY", "COMPUTER_DISPLAY", "COMPUTER_SPACES", "COMPUTER_POINTER",
-             "COMPUTER_KEYBOARD", "COMPUTER_EXEC", "COMPUTER_RECORDING"),
+             "COMPUTER_KEYBOARD", "COMPUTER_PAGE", "COMPUTER_EXEC", "COMPUTER_RECORDING"),
     "recording": ("Recording", "RecordHints", "RecordedFrame", "RecordedSegment", "frames_in_thread"),
 }
 
@@ -110,14 +122,17 @@ __all__ = [
     "AppDeclaration", "AppOs",
     "APP_SETTINGS_ENV", "part_bytes",
     "Display", "Image", "Effect", "ExecResult", "ReadyCheck", "Platform", "ComputerSettings", "SETTINGS_ENV",
+    "Tab", "PageState", "PageDialog", "FileChooser",
     "PRINTABLE_KEYS",
     "display", "set_display", "capture", "click", "drag", "scroll", "type_text", "press",
     "pointer_move", "pointer_down", "pointer_up", "pointer_position", "key_down", "key_up",
+    "navigate", "history", "tabs", "dialog", "upload", "page_state",
     "run_in_session", "write_files", "record", "Recording", "RecordHints", "RecordedFrame", "RecordedSegment",
     "frames_in_thread", "RecordingManifest", "ManifestSegment",
     "ComputerError", "InvalidArgument", "OutOfBounds", "StaleFrame", "InputBusy", "PermissionDenied", "Unsupported",
-    "BackgroundUnavailable", "TargetUnavailable", "DeviceUnavailable", "LimitExceeded", "Timeout", "DeliveryFailed",
+    "BackgroundUnavailable", "TargetUnavailable", "DeviceUnavailable", "DialogOpen", "LimitExceeded", "Timeout",
+    "DeliveryFailed",
     "Internal",
     "COMPUTER", "COMPUTER_READY", "COMPUTER_DISPLAY", "COMPUTER_SPACES", "COMPUTER_POINTER", "COMPUTER_KEYBOARD",
-    "COMPUTER_EXEC", "COMPUTER_RECORDING",
+    "COMPUTER_PAGE", "COMPUTER_EXEC", "COMPUTER_RECORDING",
 ]

{agentenv_framework_protocol-0.1.444 → agentenv_framework_protocol-0.1.445}/src/agentenv_protocol/computer/card.py

@@ -18,6 +18,7 @@ COMPUTER_DISPLAY = "urn:agentenv:computer-display/v1"
 COMPUTER_SPACES = "urn:agentenv:computer-spaces/v1"
 COMPUTER_POINTER = "urn:agentenv:computer-pointer/v1"
 COMPUTER_KEYBOARD = "urn:agentenv:computer-keyboard/v1"
+COMPUTER_PAGE = "urn:agentenv:computer-page/v1"
 COMPUTER_EXEC = "urn:agentenv:computer-exec/v1"
 COMPUTER_RECORDING = "urn:agentenv:computer-recording/v1"
 
@@ -29,6 +30,7 @@ GROUP_TOOLS = {
     "pointer": ("computer_pointer_move", "computer_pointer_down", "computer_pointer_up", "computer_pointer_position",
                 "computer_pointer_list"),
     "keyboard": ("computer_keyboard_down", "computer_keyboard_up"),
+    "page": ("computer_navigate", "computer_history", "computer_tabs", "computer_dialog", "computer_upload"),
 }
 #: The tools whose description names the default coordinate space.
 POINT_TOOLS = ("computer_screenshot", "computer_click", "computer_drag", "computer_scroll", "computer_pointer_move",
@@ -101,6 +103,12 @@ def declared_extensions(driver: ComputerDriverEnvironment) -> list[EnvironmentEx
         declared.append(EnvironmentExtension(
             uri=COMPUTER_POINTER, description="The computer_pointer_* tools, and click and scroll at the pointer.",
             params={"buttons": list(driver.buttons), "kinds": ["foreground"]}))
+    if driver.has("page"):
+        declared.append(EnvironmentExtension(
+            uri=COMPUTER_PAGE, description="The page tools, for a screen that is a website browser's active tab: "
+                                           "computer_navigate, computer_history, computer_tabs, computer_dialog and "
+                                           "computer_upload; each frame names the page it shows.",
+            params={"uploads": "~/Uploads"}))
     intake = IntakeDeclaration(get=[IntakeFormat(part="file", format="file-tree")])
     if driver.has("write_files"):
         intake.add = [IntakeFormat(part="file", format="file-tree", load="additive")]

{agentenv_framework_protocol-0.1.444 → agentenv_framework_protocol-0.1.445}/src/agentenv_protocol/computer/errors.py

@@ -55,6 +55,10 @@ class DeviceUnavailable(ComputerError):
     CODE, STATUS, APPLIED, RETRYABLE = "device_unavailable", 503, "no", True
 
 
+class DialogOpen(ComputerError):
+    CODE, STATUS, APPLIED = "dialog_open", 409, "no"
+
+
 class LimitExceeded(ComputerError):
     CODE, STATUS, APPLIED, RETRYABLE = "limit_exceeded", 429, "no", True
 

{agentenv_framework_protocol-0.1.444 → agentenv_framework_protocol-0.1.445}/src/agentenv_protocol/computer/frames.py

@@ -11,8 +11,8 @@ from datetime import datetime, timezone
 from PIL import Image as PILImage
 
 from .errors import Internal, StaleFrame
-from .operations import Display, Image
-from .results import DisplayData, FrameData
+from .operations import Display, Image, PageState
+from .results import DialogData, DisplayData, FileChooserData, FrameData, PageData
 
 KEEP = 64
 #: The JPEG quality a screenshot answers with when it names none.
@@ -34,6 +34,8 @@ class Frame:
     parent_id: str | None
     captured_at: datetime
     inputs: tuple = field(repr=False)
+    #: The page it shows, on a computer whose screen is a website browser's active tab.
+    page: PageState | None = None
 
     @property
     def width(self) -> int:
@@ -71,8 +73,21 @@ class Frame:
             display=DisplayData(width=self.display.width, height=self.display.height, scale=self.display.scale),
             parent_frame_id=self.parent_id,
             captured_at=self.captured_at.isoformat().replace("+00:00", "Z"),
+            page=_page_data(self.page),
         )
 
+    def note(self) -> str:
+        """What a summary adds for the page it shows: its title and URL, and an open dialog or file chooser."""
+        if self.page is None:
+            return ""
+        page = self.page
+        note = f' on "{page.title}" ({page.url})'
+        if page.dialog is not None:
+            note += f'; a {page.dialog.type} dialog is open: "{page.dialog.message}"'
+        if page.file_chooser is not None:
+            note += f"; a file chooser is open, for {'files' if page.file_chooser.multiple else 'one file'}"
+        return note
+
     def encode(self, fmt: str = "png", quality: int | None = None) -> tuple[str, str]:
         """The frame as base64 and its MIME type."""
         if fmt != "jpeg":
@@ -92,7 +107,7 @@ class FrameStore:
         self._latest: dict[str, str] = {}
 
     def add(self, image: PILImage.Image, display: Display, inputs: tuple, *, identity: str,
-            parent: Frame | None = None, origin: tuple[int, int] = (0, 0)) -> Frame:
+            parent: Frame | None = None, origin: tuple[int, int] = (0, 0), page: PageState | None = None) -> Frame:
         frame = Frame(
             id=f"f{self._next}",
             png=_png(image),
@@ -103,6 +118,7 @@ class FrameStore:
             parent_id=parent.id if parent is not None else None,
             captured_at=parent.captured_at if parent is not None else datetime.now(timezone.utc),
             inputs=parent.inputs if parent is not None else inputs,
+            page=parent.page if parent is not None else page,
         )
         self._next += 1
         self._frames[frame.id] = frame
@@ -153,6 +169,16 @@ def to_pil(capture: object, display: Display) -> PILImage.Image:
     return image
 
 
+def _page_data(page: PageState | None) -> PageData | None:
+    if page is None:
+        return None
+    dialog, chooser = page.dialog, page.file_chooser
+    return PageData(tab=page.tab, url=page.url, title=page.title,
+                    dialog=DialogData(type=dialog.type, message=dialog.message, default_text=dialog.default_text)
+                    if dialog is not None else None,
+                    file_chooser=FileChooserData(multiple=chooser.multiple) if chooser is not None else None)
+
+
 def _png(image: PILImage.Image) -> bytes:
     buffer = io.BytesIO()
     image.save(buffer, format="PNG")

{agentenv_framework_protocol-0.1.444 → agentenv_framework_protocol-0.1.445}/src/agentenv_protocol/computer/operations.py

@@ -12,6 +12,16 @@ Each operation is called with screen pixels, positionally, in the order below, a
     key_down(keys), key_up(keys)
     run_in_session(argv, cwd, env, timeout_s) -> ExecResult            write_files({path: bytes})
     record(hints) -> an async iterator of RecordedFrame or RecordedSegment, until it is closed
+
+The page group, for a driver whose screen is a website browser's active tab:
+
+    navigate(url)                                         history(direction), back, forward or reload
+    tabs(action, tab, url) -> [Tab, ...]                  list, open, switch or close; the tabs open after it
+    dialog(accept, text)                                  upload(paths), relative to the session user's ~/Uploads
+    page_state() -> PageState
+
+A page driver's ``capture`` may return its ``Image`` with the ``page`` it shows, read with it, which every frame then
+carries rather than one ``page_state`` reads after.
 """
 from __future__ import annotations
 
@@ -25,10 +35,13 @@ REQUIRED = ("display", "capture", "click", "drag", "scroll", "type_text", "press
 GROUPS = {
     "pointer": ("pointer_move", "pointer_down", "pointer_up", "pointer_position"),
     "keyboard": ("key_down", "key_up"),
+    "page": ("navigate", "history", "tabs", "dialog", "upload", "page_state"),
 }
 OPTIONAL = ("set_display", "run_in_session", "write_files", "record")
 INPUT = frozenset({"click", "drag", "scroll", "type_text", "press", "pointer_move", "pointer_down", "pointer_up",
-                   "key_down", "key_up"})
+                   "key_down", "key_up", "navigate", "history", "tabs", "dialog", "upload"})
+#: The input a page's open dialog blocks, which is refused while one is open there.
+DIALOG_BLOCKS = INPUT - {"tabs", "dialog", "upload"}
 
 _OP_ATTR = "_aec_operation"
 
@@ -44,11 +57,12 @@ class Display:
 @dataclass(frozen=True)
 class Image:
     """A capture: encoded ``png`` or ``jpeg`` bytes, or raw pixels in a Pillow raw mode (``RGB``, ``RGBA``, ``BGRA``,
-    ``BGRX``)."""
+    ``BGRX``); and, from a page driver, the page it shows."""
     width: int
     height: int
     data: bytes
     mode: str = "png"
+    page: PageState | None = None
 
 
 @dataclass(frozen=True)
@@ -60,6 +74,40 @@ class Effect:
     detail: str | None = None
 
 
+@dataclass(frozen=True)
+class Tab:
+    """A tab of a website browser: its id, the browser's own, and its page's URL and title."""
+    id: str
+    url: str
+    title: str
+    active: bool = False
+
+
+@dataclass(frozen=True)
+class PageDialog:
+    """A JavaScript dialog open on a page: ``alert``, ``confirm``, ``prompt`` or ``beforeunload``."""
+    type: str
+    message: str
+    default_text: str = ""
+
+
+@dataclass(frozen=True)
+class FileChooser:
+    """A file chooser a page opened, waiting for files: several, or one."""
+    multiple: bool = False
+
+
+@dataclass(frozen=True)
+class PageState:
+    """The page a page driver's screen shows: the active tab's id, URL and title, the dialog open on it, and the file
+    chooser it opened that waits."""
+    tab: str
+    url: str
+    title: str
+    dialog: PageDialog | None = None
+    file_chooser: FileChooser | None = None
+
+
 @dataclass(frozen=True)
 class ExecResult:
     exit_code: int | None
@@ -115,6 +163,12 @@ pointer_up = _marker("pointer_up")
 pointer_position = _marker("pointer_position")
 key_down = _marker("key_down")
 key_up = _marker("key_up")
+navigate = _marker("navigate")
+history = _marker("history")
+tabs = _marker("tabs")
+dialog = _marker("dialog")
+upload = _marker("upload")
+page_state = _marker("page_state")
 run_in_session = _marker("run_in_session")
 write_files = _marker("write_files")
 record = _marker("record")

{agentenv_framework_protocol-0.1.444 → agentenv_framework_protocol-0.1.445}/src/agentenv_protocol/computer/results.py

@@ -16,6 +16,25 @@ class DisplayData(BaseModel):
     scale: float
 
 
+class DialogData(BaseModel):
+    type: str
+    message: str
+    default_text: str = ""
+
+
+class FileChooserData(BaseModel):
+    multiple: bool
+
+
+class PageData(BaseModel):
+    """The page a frame shows, on a computer whose screen is a website browser's active tab."""
+    tab: str
+    url: str
+    title: str
+    dialog: Optional[DialogData] = None
+    file_chooser: Optional[FileChooserData] = None
+
+
 class FrameData(BaseModel):
     frame_id: str
     width: int
@@ -23,6 +42,7 @@ class FrameData(BaseModel):
     display: DisplayData
     parent_frame_id: Optional[str] = None
     captured_at: str
+    page: Optional[PageData] = None
 
 
 class ScreenshotResult(FrameData):
@@ -53,6 +73,17 @@ class InputResult(BaseModel):
     frame: Optional[FrameData] = None
 
 
+class TabData(BaseModel):
+    tab: str
+    url: str
+    title: str
+    active: bool
+
+
+class TabsResult(InputResult):
+    tabs: list[TabData]
+
+
 class WaitResult(BaseModel):
     summary: str
     waited_ms: int

{agentenv_framework_protocol-0.1.444 → agentenv_framework_protocol-0.1.445}/src/agentenv_protocol/computer/runtime.py

@@ -14,9 +14,9 @@ from mcp.types import CallToolResult
 from ..errors import AgentEnvError
 from ..tool_context import ToolContext
 from . import apps
-from .errors import DeliveryFailed, DeviceUnavailable, InputBusy, Internal
+from .errors import DeliveryFailed, DeviceUnavailable, DialogOpen, InputBusy, Internal
 from .frames import Frame, FrameStore, to_pil
-from .operations import GROUPS, INPUT, Display, ReadyCheck, resolve
+from .operations import DIALOG_BLOCKS, GROUPS, INPUT, Display, Image, PageState, ReadyCheck, resolve
 from .queue import InputQueue
 from .recording import RecordHints, Recordings
 from .results import EffectData, InputResult, send
@@ -80,8 +80,18 @@ class ComputerRuntime:
         counts as after the frame, which the frame may not show."""
         inputs = self.queue.snapshot()
         display = await self.display_now()
-        image = to_pil(await self.call("capture"), display)
-        return self.frames.add(image, display, inputs, identity=identity)
+        captured = await self.call("capture")
+        image = to_pil(captured, display)
+        page = captured.page if isinstance(captured, Image) and isinstance(captured.page, PageState) else None
+        if page is None and self.has("page"):
+            page = await self.page_now()
+        return self.frames.add(image, display, inputs, identity=identity, page=page)
+
+    async def page_now(self) -> PageState:
+        page = await self.call("page_state")
+        if not isinstance(page, PageState):
+            raise Internal(f"@page_state returned {page!r}, not a PageState")
+        return page
 
     async def pointer(self) -> tuple[int, int]:
         where = await self.call("pointer_position")
@@ -93,15 +103,20 @@ class ComputerRuntime:
 
     async def act(self, ctx: ToolContext, operation: str, args: tuple, *, frame: Frame | None, screenshot: bool,
                   summary: str, mapped: bool = False, at_pointer: bool = False, repeat: int = 1,
-                  hold: tuple | None = None, move_to: tuple | None = None, then: Callable | None = None) -> CallToolResult:
+                  hold: tuple | None = None, move_to: tuple | None = None, then: Callable | None = None,
+                  wrap: Callable[[InputResult, Any], InputResult] | None = None) -> CallToolResult:
         """One input call: its turn in the queue, the operation (after a move, repeated or held), then the result
         with others' input since ``frame`` and, when asked, a fresh frame. Whatever the call's turn can change is read
         once it has the turn: a point ``mapped`` from ``frame`` is checked again, since the display may have changed
-        while it waited, and ``at_pointer`` reads the pointer and puts it before ``args``. ``then`` runs as soon as
-        the input is delivered."""
+        while it waited, and ``at_pointer`` reads the pointer and puts it before ``args``; input a page's open dialog
+        blocks is refused. ``then`` runs as soon as the input is delivered, and ``wrap`` makes the result from the
+        operation's own."""
         identity = self.driver.caller_identity(ctx)
         sent = False
         async with self.queue.turn(self.driver.settings.queue_wait_s):
+            if operation in DIALOG_BLOCKS and self.has("page") and (dialog := (await self.page_now()).dialog):
+                raise DialogOpen(f'a {dialog.type} dialog is open on the page: "{dialog.message}"; answer it with '
+                                 "computer_dialog first")
             if mapped and frame is not None:
                 self.frames.check_current(frame, await self.display_now())
             if at_pointer:
@@ -129,14 +144,16 @@ class ComputerRuntime:
                 then()
         result = InputResult(summary=summary, applied="yes", delivery="foreground", others_since_frame=others,
                              effect=EffectData.of(effect))
+        if wrap is not None:
+            result = wrap(result, effect)
         if not screenshot:
             return send(result)
         try:
             new = await self.capture(identity)
         except AgentEnvError as e:
-            result.summary = f"{summary}; the screenshot after it failed ({e.code}: {e.message})"
+            result.summary = f"{result.summary}; the screenshot after it failed ({e.code}: {e.message})"
             return send(result)
-        result.summary, result.frame = f"{summary}, then captured {new.id}", new.result()
+        result.summary, result.frame = f"{result.summary}, then captured {new.id}{new.note()}", new.result()
         return send(result, new.encode())
 
     async def _hold_for(self, ms: int, keys: list[str]) -> None: