openadapt-types 0.1.0__tar.gz

openadapt_types-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,18 @@
+name: Publish to PyPI
+
+on:
+  push:
+    tags:
+      - "v*"
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: release
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v4
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@release/v1

openadapt_types-0.1.0/.github/workflows/test.yml

@@ -0,0 +1,16 @@
+name: test
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v4
+      - run: uv sync --extra dev
+      - run: uv run pytest

openadapt_types-0.1.0/.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.pyc
+*.pyo
+.venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.env

openadapt_types-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 OpenAdapt
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

openadapt_types-0.1.0/PKG-INFO

@@ -0,0 +1,145 @@
+Metadata-Version: 2.4
+Name: openadapt-types
+Version: 0.1.0
+Summary: Canonical Pydantic schemas for computer-use agents: ComputerState, Action, ActionResult, UINode
+Project-URL: Homepage, https://github.com/OpenAdaptAI/openadapt-types
+Project-URL: Repository, https://github.com/OpenAdaptAI/openadapt-types
+Project-URL: Issues, https://github.com/OpenAdaptAI/openadapt-types/issues
+Author-email: OpenAdapt <admin@openadapt.ai>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,computer-use,gui-automation,pydantic,schemas
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: pydantic>=2.0
+Provides-Extra: dev
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# openadapt-types
+
+Canonical Pydantic schemas for computer-use agents.
+
+```
+pip install openadapt-types
+```
+
+## What's in the box
+
+| Schema | Purpose |
+|--------|---------|
+| `ComputerState` | Screen state: screenshot + UI element graph + window context |
+| `UINode` | Single UI element with role, bbox, hierarchy, platform anchors |
+| `Action` | Agent action with typed action space + flexible targeting |
+| `ActionTarget` | Where to act: `node_id` > `description` > `(x, y)` coordinates |
+| `ActionResult` | Execution outcome with error taxonomy + state delta |
+| `Episode` / `Step` | Complete task trajectory (observation → action → result) |
+| `FailureRecord` | Classified failure for dataset pipelines |
+
+## Quick start
+
+```python
+from openadapt_types import (
+    Action, ActionTarget, ActionType,
+    ComputerState, UINode, BoundingBox,
+)
+
+# Describe what's on screen
+state = ComputerState(
+    viewport=(1920, 1080),
+    nodes=[
+        UINode(node_id="n0", role="window", name="My App", children_ids=["n1"]),
+        UINode(node_id="n1", role="button", name="Submit", parent_id="n0",
+               bbox=BoundingBox(x=500, y=400, width=100, height=40)),
+    ],
+)
+
+# Agent decides what to do
+action = Action(
+    type=ActionType.CLICK,
+    target=ActionTarget(node_id="n1"),
+    reasoning="Click Submit to proceed",
+)
+
+# Render element tree for LLM prompts
+print(state.to_text_tree())
+# [n0] window: My App
+#   [n1] button: Submit
+```
+
+## Action targeting
+
+`ActionTarget` supports three grounding strategies (in priority order):
+
+```python
+# 1. Element-based (preferred — most robust)
+ActionTarget(node_id="n1")
+
+# 2. Description-based (resolved by grounding module)
+ActionTarget(description="the blue submit button")
+
+# 3. Coordinate-based (fallback)
+ActionTarget(x=550, y=420)
+ActionTarget(x=0.29, y=0.39, is_normalized=True)
+```
+
+Agents SHOULD produce `node_id` or `description`. The runtime resolves to coordinates.
+
+## Compatibility with existing schemas
+
+Converters for three existing OpenAdapt schema formats:
+
+```python
+from openadapt_types._compat import (
+    from_benchmark_observation,   # openadapt-evals BenchmarkObservation
+    from_benchmark_action,        # openadapt-evals BenchmarkAction
+    from_ml_observation,          # openadapt-ml Observation
+    from_ml_action,               # openadapt-ml Action
+    from_omnimcp_screen_state,    # omnimcp ScreenState
+    from_omnimcp_action_decision, # omnimcp ActionDecision
+)
+
+# Convert existing data
+state = from_benchmark_observation(obs.__dict__)
+action = from_benchmark_action(act.__dict__)
+```
+
+## JSON Schema
+
+Export for language-agnostic tooling:
+
+```python
+import json
+from openadapt_types import ComputerState, Action, Episode
+
+# Get JSON Schema
+schema = ComputerState.model_json_schema()
+print(json.dumps(schema, indent=2))
+```
+
+## Design principles
+
+- **Pydantic v2** — runtime validation, JSON Schema export, fast serialization
+- **Pixels + structure** — always capture both visual and semantic UI state
+- **Node graph** — full element tree, not just focused element
+- **Platform-agnostic** — same schema for Windows, macOS, Linux, web
+- **Extension-friendly** — `raw`, `attributes`, `metadata` fields everywhere
+- **Backward compatible** — `_compat` converters for gradual migration
+
+## Dependencies
+
+Just `pydantic>=2.0`. No ML libraries, no heavy deps.
+
+## License
+
+MIT

openadapt_types-0.1.0/README.md

@@ -0,0 +1,117 @@
+# openadapt-types
+
+Canonical Pydantic schemas for computer-use agents.
+
+```
+pip install openadapt-types
+```
+
+## What's in the box
+
+| Schema | Purpose |
+|--------|---------|
+| `ComputerState` | Screen state: screenshot + UI element graph + window context |
+| `UINode` | Single UI element with role, bbox, hierarchy, platform anchors |
+| `Action` | Agent action with typed action space + flexible targeting |
+| `ActionTarget` | Where to act: `node_id` > `description` > `(x, y)` coordinates |
+| `ActionResult` | Execution outcome with error taxonomy + state delta |
+| `Episode` / `Step` | Complete task trajectory (observation → action → result) |
+| `FailureRecord` | Classified failure for dataset pipelines |
+
+## Quick start
+
+```python
+from openadapt_types import (
+    Action, ActionTarget, ActionType,
+    ComputerState, UINode, BoundingBox,
+)
+
+# Describe what's on screen
+state = ComputerState(
+    viewport=(1920, 1080),
+    nodes=[
+        UINode(node_id="n0", role="window", name="My App", children_ids=["n1"]),
+        UINode(node_id="n1", role="button", name="Submit", parent_id="n0",
+               bbox=BoundingBox(x=500, y=400, width=100, height=40)),
+    ],
+)
+
+# Agent decides what to do
+action = Action(
+    type=ActionType.CLICK,
+    target=ActionTarget(node_id="n1"),
+    reasoning="Click Submit to proceed",
+)
+
+# Render element tree for LLM prompts
+print(state.to_text_tree())
+# [n0] window: My App
+#   [n1] button: Submit
+```
+
+## Action targeting
+
+`ActionTarget` supports three grounding strategies (in priority order):
+
+```python
+# 1. Element-based (preferred — most robust)
+ActionTarget(node_id="n1")
+
+# 2. Description-based (resolved by grounding module)
+ActionTarget(description="the blue submit button")
+
+# 3. Coordinate-based (fallback)
+ActionTarget(x=550, y=420)
+ActionTarget(x=0.29, y=0.39, is_normalized=True)
+```
+
+Agents SHOULD produce `node_id` or `description`. The runtime resolves to coordinates.
+
+## Compatibility with existing schemas
+
+Converters for three existing OpenAdapt schema formats:
+
+```python
+from openadapt_types._compat import (
+    from_benchmark_observation,   # openadapt-evals BenchmarkObservation
+    from_benchmark_action,        # openadapt-evals BenchmarkAction
+    from_ml_observation,          # openadapt-ml Observation
+    from_ml_action,               # openadapt-ml Action
+    from_omnimcp_screen_state,    # omnimcp ScreenState
+    from_omnimcp_action_decision, # omnimcp ActionDecision
+)
+
+# Convert existing data
+state = from_benchmark_observation(obs.__dict__)
+action = from_benchmark_action(act.__dict__)
+```
+
+## JSON Schema
+
+Export for language-agnostic tooling:
+
+```python
+import json
+from openadapt_types import ComputerState, Action, Episode
+
+# Get JSON Schema
+schema = ComputerState.model_json_schema()
+print(json.dumps(schema, indent=2))
+```
+
+## Design principles
+
+- **Pydantic v2** — runtime validation, JSON Schema export, fast serialization
+- **Pixels + structure** — always capture both visual and semantic UI state
+- **Node graph** — full element tree, not just focused element
+- **Platform-agnostic** — same schema for Windows, macOS, Linux, web
+- **Extension-friendly** — `raw`, `attributes`, `metadata` fields everywhere
+- **Backward compatible** — `_compat` converters for gradual migration
+
+## Dependencies
+
+Just `pydantic>=2.0`. No ML libraries, no heavy deps.
+
+## License
+
+MIT

openadapt_types-0.1.0/openadapt_types/__init__.py

@@ -0,0 +1,59 @@
+"""openadapt-types: Canonical Pydantic schemas for computer-use agents.
+
+This package provides the shared type definitions used across the OpenAdapt
+ecosystem and designed for adoption by any computer-use agent project.
+
+Quick start::
+
+    from openadapt_types import ComputerState, Action, ActionType, UINode
+
+    state = ComputerState(
+        viewport=(1920, 1080),
+        nodes=[
+            UINode(node_id="n0", role="button", name="Submit"),
+        ],
+    )
+
+    action = Action(
+        type=ActionType.CLICK,
+        target=ActionTarget(node_id="n0"),
+    )
+"""
+
+from openadapt_types.action import (
+    Action,
+    ActionResult,
+    ActionTarget,
+    ActionType,
+)
+from openadapt_types.computer_state import (
+    BoundingBox,
+    ComputerState,
+    ElementRole,
+    ProcessInfo,
+    UINode,
+)
+from openadapt_types.episode import Episode, Step
+from openadapt_types.failure import FailureCategory, FailureRecord
+
+__version__ = "0.1.0"
+
+__all__ = [
+    # computer_state
+    "BoundingBox",
+    "ComputerState",
+    "ElementRole",
+    "ProcessInfo",
+    "UINode",
+    # action
+    "Action",
+    "ActionResult",
+    "ActionTarget",
+    "ActionType",
+    # episode
+    "Episode",
+    "Step",
+    # failure
+    "FailureCategory",
+    "FailureRecord",
+]

openadapt_types-0.1.0/openadapt_types/_compat.py

@@ -0,0 +1,262 @@
+"""Backward-compatibility converters for existing schemas.
+
+These functions convert from the three existing schema formats to
+the canonical ``openadapt-types`` models. They allow gradual migration:
+old code keeps working, new code uses ``openadapt-types`` directly.
+
+Usage::
+
+    from openadapt_types._compat import (
+        from_benchmark_observation,
+        from_benchmark_action,
+        from_ml_observation,
+        from_ml_action,
+        from_omnimcp_screen_state,
+        from_omnimcp_action_decision,
+    )
+
+These converters work with plain dicts (``.__dict__`` or ``asdict()``),
+not imported types, to avoid circular dependencies.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from .action import Action, ActionTarget, ActionType
+from .computer_state import BoundingBox, ComputerState, UINode
+
+
+# =====================================================================
+# From openadapt_evals.adapters.base (dataclass dicts)
+# =====================================================================
+
+
+def from_benchmark_observation(obs: dict[str, Any]) -> ComputerState:
+    """Convert a ``BenchmarkObservation.__dict__`` to :class:`ComputerState`.
+
+    Maps:
+    - ``screenshot`` → ``screenshot_png``
+    - ``screenshot_path`` → ``screenshot_path``
+    - ``viewport`` → ``viewport``
+    - ``accessibility_tree`` → ``accessibility_tree_raw``
+    - ``dom_html`` → ``dom_html``
+    - ``url``, ``window_title``, ``app_name`` → context fields
+    - ``focused_element`` → ``focused_node_id`` (if dict with node_id)
+    - ``raw_observation`` → ``raw``
+    """
+    focused = obs.get("focused_element")
+    focused_id = None
+    if isinstance(focused, dict):
+        focused_id = focused.get("node_id")
+
+    return ComputerState(
+        screenshot_png=obs.get("screenshot"),
+        screenshot_path=obs.get("screenshot_path"),
+        viewport=obs.get("viewport"),
+        accessibility_tree_raw=obs.get("accessibility_tree"),
+        dom_html=obs.get("dom_html"),
+        url=obs.get("url"),
+        focused_node_id=focused_id,
+        raw=obs.get("raw_observation"),
+    )
+
+
+def from_benchmark_action(act: dict[str, Any]) -> Action:
+    """Convert a ``BenchmarkAction.__dict__`` to :class:`Action`.
+
+    Maps:
+    - ``type`` → ``ActionType`` (string to enum)
+    - ``x``, ``y`` → ``ActionTarget.x``, ``ActionTarget.y``
+    - ``target_node_id`` → ``ActionTarget.node_id``
+    - ``text``, ``key``, ``modifiers`` → keyboard params
+    - ``scroll_direction``, ``scroll_amount`` → scroll params
+    - ``end_x``, ``end_y`` → ``drag_end``
+    - ``answer`` → ``answer``
+    - ``raw_action`` → ``raw``
+    """
+    action_type_str = act.get("type", "done")
+    try:
+        action_type = ActionType(action_type_str)
+    except ValueError:
+        action_type = ActionType.DONE
+
+    # Build target
+    target = None
+    x = act.get("x")
+    y = act.get("y")
+    node_id = act.get("target_node_id")
+    if x is not None or y is not None or node_id is not None:
+        target = ActionTarget(node_id=node_id, x=x, y=y)
+
+    # Build drag end
+    drag_end = None
+    end_x = act.get("end_x")
+    end_y = act.get("end_y")
+    if end_x is not None or end_y is not None:
+        drag_end = ActionTarget(x=end_x, y=end_y)
+
+    return Action(
+        type=action_type,
+        target=target,
+        text=act.get("text"),
+        key=act.get("key"),
+        modifiers=act.get("modifiers"),
+        scroll_direction=act.get("scroll_direction"),
+        scroll_amount=int(act["scroll_amount"]) if act.get("scroll_amount") is not None else None,
+        drag_end=drag_end,
+        answer=act.get("answer"),
+        raw=act.get("raw_action"),
+    )
+
+
+# =====================================================================
+# From openadapt_ml.schema.episode (Pydantic dicts)
+# =====================================================================
+
+
+def from_ml_observation(obs: dict[str, Any]) -> ComputerState:
+    """Convert an ``openadapt_ml.schema.episode.Observation.model_dump()``
+    to :class:`ComputerState`."""
+    return ComputerState(
+        screenshot_path=obs.get("screenshot_path"),
+        screenshot_base64=obs.get("screenshot_base64"),
+        viewport=obs.get("screen_size"),
+        accessibility_tree_raw=obs.get("a11y_tree"),
+        dom_html=obs.get("dom"),
+        url=obs.get("url"),
+        timestamp=obs.get("timestamp"),
+        raw=obs.get("raw"),
+    )
+
+
+def from_ml_action(act: dict[str, Any]) -> Action:
+    """Convert an ``openadapt_ml.schema.episode.Action.model_dump()``
+    to :class:`Action`."""
+    action_type_str = act.get("type", "done")
+    try:
+        action_type = ActionType(action_type_str)
+    except ValueError:
+        action_type = ActionType.DONE
+
+    # Coordinates
+    target = None
+    coords = act.get("coordinates")
+    norm = act.get("normalized_coordinates")
+    element = act.get("element")
+
+    if coords:
+        target = ActionTarget(x=float(coords["x"]), y=float(coords["y"]))
+    elif norm:
+        target = ActionTarget(x=norm[0], y=norm[1], is_normalized=True)
+    elif element and element.get("element_id"):
+        target = ActionTarget(node_id=element["element_id"])
+
+    # Drag
+    drag_end = None
+    end_coords = act.get("end_coordinates")
+    norm_end = act.get("normalized_end")
+    if end_coords:
+        drag_end = ActionTarget(x=float(end_coords["x"]), y=float(end_coords["y"]))
+    elif norm_end:
+        drag_end = ActionTarget(x=norm_end[0], y=norm_end[1], is_normalized=True)
+
+    return Action(
+        type=action_type,
+        target=target,
+        text=act.get("text"),
+        key=act.get("key"),
+        modifiers=act.get("modifiers"),
+        scroll_direction=act.get("scroll_direction"),
+        scroll_amount=act.get("scroll_amount"),
+        drag_end=drag_end,
+        url=act.get("url"),
+        app_name=act.get("app_name"),
+        duration_seconds=act.get("duration"),
+        raw=act.get("raw"),
+    )
+
+
+# =====================================================================
+# From omnimcp.types (dataclass dicts)
+# =====================================================================
+
+
+def from_omnimcp_screen_state(state: dict[str, Any]) -> ComputerState:
+    """Convert an ``omnimcp.types.ScreenState`` (as dict) to :class:`ComputerState`.
+
+    Maps:
+    - ``elements`` → ``nodes`` (UIElement → UINode)
+    - ``dimensions`` → ``viewport``
+    - ``timestamp`` → ``timestamp``
+    """
+    nodes: list[UINode] = []
+    for elem in state.get("elements", []):
+        bbox = None
+        bounds = elem.get("bounds")
+        dims = state.get("dimensions", (1, 1))
+        if bounds and len(bounds) == 4:
+            # omnimcp bounds are normalized (x, y, w, h)
+            vw, vh = dims
+            bbox = BoundingBox(
+                x=int(bounds[0] * vw),
+                y=int(bounds[1] * vh),
+                width=int(bounds[2] * vw),
+                height=int(bounds[3] * vh),
+            )
+        nodes.append(UINode(
+            node_id=str(elem.get("id", "")),
+            role=elem.get("type", "unknown"),
+            text=elem.get("content"),
+            bbox=bbox,
+            confidence=elem.get("confidence", 1.0),
+            attributes=elem.get("attributes", {}),
+        ))
+
+    return ComputerState(
+        nodes=nodes,
+        viewport=state.get("dimensions"),
+        timestamp=state.get("timestamp"),
+        source="omnimcp",
+    )
+
+
+def from_omnimcp_action_decision(decision: dict[str, Any]) -> Action:
+    """Convert an ``omnimcp.types.ActionDecision`` (as dict) to :class:`Action`.
+
+    Maps:
+    - ``action_type`` → ``ActionType``
+    - ``target_element_id`` → ``ActionTarget.node_id``
+    - ``parameters`` → keyboard/scroll/wait params
+    - ``analysis_reasoning`` → ``reasoning``
+    """
+    action_type_str = decision.get("action_type", "done")
+    type_map = {
+        "click": ActionType.CLICK,
+        "type": ActionType.TYPE,
+        "scroll": ActionType.SCROLL,
+        "press_key": ActionType.KEY,
+        "wait": ActionType.WAIT,
+        "finish": ActionType.DONE,
+        "launch_app": ActionType.OPEN_APP,
+    }
+    action_type = type_map.get(action_type_str, ActionType.DONE)
+
+    target = None
+    elem_id = decision.get("target_element_id")
+    if elem_id is not None:
+        target = ActionTarget(node_id=str(elem_id))
+
+    params = decision.get("parameters", {})
+
+    return Action(
+        type=action_type,
+        target=target,
+        text=params.get("text_to_type"),
+        key=params.get("key_info"),
+        scroll_direction=params.get("scroll_direction"),
+        scroll_amount=params.get("scroll_steps"),
+        app_name=params.get("app_name"),
+        duration_seconds=params.get("wait_duration_s"),
+        reasoning=decision.get("analysis_reasoning"),
+    )