hud-python 0.2.1__tar.gz → 0.2.2__tar.gz

{hud_python-0.2.1 → hud_python-0.2.2}/PKG-INFO

@@ -1,11 +1,11 @@
 Metadata-Version: 2.4
 Name: hud-python
-Version: 0.2.1
+Version: 0.2.2
 Summary: SDK for the HUD evaluation platform.
 Project-URL: Homepage, https://github.com/hud-evals/hud-sdk
 Project-URL: Bug Tracker, https://github.com/hud-evals/hud-sdk/issues
 Project-URL: Documentation, https://hud.so
-Author-email: Human Union Data SDK <founders@hud.so>
+Author-email: HUD SDK <founders@hud.so>
 License: MIT License
         
         Copyright (c) 2025 Human Union Data, Inc
@@ -37,8 +37,14 @@ Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Requires-Python: <3.14,>=3.10
 Requires-Dist: aiodocker>=0.24.0
+Requires-Dist: anthropic
 Requires-Dist: httpx<1,>=0.23.0
 Requires-Dist: inspect-ai>=0.3.80
+Requires-Dist: ipykernel
+Requires-Dist: langchain
+Requires-Dist: langchain-openai
+Requires-Dist: numpy
+Requires-Dist: openai
 Requires-Dist: pillow>=11.1.0
 Requires-Dist: pydantic-settings<3,>=2
 Requires-Dist: pydantic<3,>=2
@@ -117,10 +123,9 @@ async def main():
     obs, _ = await env.reset() # Gets first observation
     for i in range(5):
         actions, done = await agent.predict(obs)
-        if done:
-            break
-        
+
         obs, reward, terminated, info = await env.step(actions)
+        if done or terminated: break
     
     # Evaluate and close
     result = await env.evaluate()
@@ -132,22 +137,37 @@ if __name__ == "__main__":
 
 ```
 
+Alternatively, run a full evaluation set via the ```run_job``` command:
+
+```python
+from hud import load_taskset, run_job, ClaudeAgent
+
+# load
+taskset = load_taskset("GAIA")
+
+# evaluate
+job = await run_job(ClaudeAgent, taskset, "test-gaia-job")
+
+# get results OR view them in app.hud.so
+print(await job.get_analytics())
+```
+
 ## Documentation Sections
 
 Explore the core concepts and features of the SDK:
 
-*   **[Tasks and TaskSets](/concepts/task)**: Define goals, context, setup, and evaluation criteria for agent scenarios. This includes both interactive and **question-answering (QA)** style tasks.
-*   **[Environments](/concepts/environment)**: Understand the browser and OS runtimes where agents interact.
-*   **[Agents](/concepts/agent)**: Learn about the agent architecture (Claude, Operator) and how they process observations and predict actions.
-*   **[Adapters](/concepts/adapter)**: See how actions and observations are translated between agents and environments.
-*   **[Jobs](/concepts/job)**: Group related runs for analysis and viewing on the HUD platform.
-*   **[Trajectories](/concepts/trajectory)**: Understand the recorded data from each agent run.
+*   **[Tasks and TaskSets](https://documentation.hud.so/concepts/task)**: Define goals, context, setup, and evaluation criteria for agent scenarios. This includes both interactive and **question-answering (QA)** style tasks.
+*   **[Environments](https://documentation.hud.so/concepts/environment)**: Understand the browser and OS runtimes where agents interact.
+*   **[Agents](https://documentation.hud.so/concepts/agent)**: Learn about the agent architecture (Claude, Operator) and how they process observations and predict actions.
+*   **[Adapters](https://documentation.hud.so/concepts/adapter)**: See how actions and observations are translated between agents and environments.
+*   **[Jobs](https://documentation.hud.so/concepts/job)**: Group related runs for analysis and viewing on the HUD platform.
+*   **[Trajectories](https://documentation.hud.so/concepts/trajectory)**: Understand the recorded data from each agent run.
 *   **Advanced Topics**:
-    *   **[CLA Action Details](/advanced/cla-details)**: Explore the standardized action format.
-    *   **[Custom Environments](/advanced/custom-environments)**: Build your own Docker-based local or remote environments.
-    *   **[Advanced Environment Control](/advanced/environment-control)**: Use `invoke`, `execute`, and `_setup` for finer control.
+    *   **[CLA Action Details](https://documentation.hud.so/advanced/cla-details)**: Explore the standardized action format.
+    *   **[Custom Environments](https://documentation.hud.so/advanced/custom-environments)**: Build your own Docker-based local or remote environments.
+    *   **[Advanced Environment Control](https://documentation.hud.so/advanced/environment-control)**: Use `invoke`, `execute`, and `_setup` for finer control.
 
-*   **[Full API Reference](/api-reference/gym)**: Detailed specifications for all modules and classes.
+*   **[Full API Reference](https://documentation.hud.so/api-reference/gym)**: Detailed specifications for all modules and classes.
 
 ## [Examples](examples/)
 
@@ -160,7 +180,7 @@ We recommend you first take a look at the example notebooks showing how to use t
 
 ## Documentation
 
-For comprehensive guides, examples, and API reference, visit [our docs](https://docs.hud.so/introduction)
+For comprehensive guides, examples, and API reference, visit [our docs](https://documentation.hud.so/introduction)
 
 ## License
 

{hud_python-0.2.1 → hud_python-0.2.2}/README.md

@@ -58,10 +58,9 @@ async def main():
     obs, _ = await env.reset() # Gets first observation
     for i in range(5):
         actions, done = await agent.predict(obs)
-        if done:
-            break
-        
+
         obs, reward, terminated, info = await env.step(actions)
+        if done or terminated: break
     
     # Evaluate and close
     result = await env.evaluate()
@@ -73,22 +72,37 @@ if __name__ == "__main__":
 
 ```
 
+Alternatively, run a full evaluation set via the ```run_job``` command:
+
+```python
+from hud import load_taskset, run_job, ClaudeAgent
+
+# load
+taskset = load_taskset("GAIA")
+
+# evaluate
+job = await run_job(ClaudeAgent, taskset, "test-gaia-job")
+
+# get results OR view them in app.hud.so
+print(await job.get_analytics())
+```
+
 ## Documentation Sections
 
 Explore the core concepts and features of the SDK:
 
-*   **[Tasks and TaskSets](/concepts/task)**: Define goals, context, setup, and evaluation criteria for agent scenarios. This includes both interactive and **question-answering (QA)** style tasks.
-*   **[Environments](/concepts/environment)**: Understand the browser and OS runtimes where agents interact.
-*   **[Agents](/concepts/agent)**: Learn about the agent architecture (Claude, Operator) and how they process observations and predict actions.
-*   **[Adapters](/concepts/adapter)**: See how actions and observations are translated between agents and environments.
-*   **[Jobs](/concepts/job)**: Group related runs for analysis and viewing on the HUD platform.
-*   **[Trajectories](/concepts/trajectory)**: Understand the recorded data from each agent run.
+*   **[Tasks and TaskSets](https://documentation.hud.so/concepts/task)**: Define goals, context, setup, and evaluation criteria for agent scenarios. This includes both interactive and **question-answering (QA)** style tasks.
+*   **[Environments](https://documentation.hud.so/concepts/environment)**: Understand the browser and OS runtimes where agents interact.
+*   **[Agents](https://documentation.hud.so/concepts/agent)**: Learn about the agent architecture (Claude, Operator) and how they process observations and predict actions.
+*   **[Adapters](https://documentation.hud.so/concepts/adapter)**: See how actions and observations are translated between agents and environments.
+*   **[Jobs](https://documentation.hud.so/concepts/job)**: Group related runs for analysis and viewing on the HUD platform.
+*   **[Trajectories](https://documentation.hud.so/concepts/trajectory)**: Understand the recorded data from each agent run.
 *   **Advanced Topics**:
-    *   **[CLA Action Details](/advanced/cla-details)**: Explore the standardized action format.
-    *   **[Custom Environments](/advanced/custom-environments)**: Build your own Docker-based local or remote environments.
-    *   **[Advanced Environment Control](/advanced/environment-control)**: Use `invoke`, `execute`, and `_setup` for finer control.
+    *   **[CLA Action Details](https://documentation.hud.so/advanced/cla-details)**: Explore the standardized action format.
+    *   **[Custom Environments](https://documentation.hud.so/advanced/custom-environments)**: Build your own Docker-based local or remote environments.
+    *   **[Advanced Environment Control](https://documentation.hud.so/advanced/environment-control)**: Use `invoke`, `execute`, and `_setup` for finer control.
 
-*   **[Full API Reference](/api-reference/gym)**: Detailed specifications for all modules and classes.
+*   **[Full API Reference](https://documentation.hud.so/api-reference/gym)**: Detailed specifications for all modules and classes.
 
 ## [Examples](examples/)
 
@@ -101,7 +115,7 @@ We recommend you first take a look at the example notebooks showing how to use t
 
 ## Documentation
 
-For comprehensive guides, examples, and API reference, visit [our docs](https://docs.hud.so/introduction)
+For comprehensive guides, examples, and API reference, visit [our docs](https://documentation.hud.so/introduction)
 
 ## License
 

{hud_python-0.2.1 → hud_python-0.2.2}/docs/advanced/cla-details.mdx

@@ -69,6 +69,11 @@ Here are some key CLA types grouped by category:
 *   **`ScreenshotFetch`**: Requests a screenshot (used internally, typically not sent by agents directly).
 *   **`PositionFetch`**: Requests the current cursor position (used internally).
 
+### Response Actions
+
+*   **`ResponseAction`**: Used to submit a final text answer.
+    *   `text: str`: The final textual response from the agent.
+
 ### Custom Actions
 
 *   **`CustomAction`**: Allows defining arbitrary actions specific to a custom environment controller.

{hud_python-0.2.1 → hud_python-0.2.2}/docs/concepts/environment.mdx

@@ -48,6 +48,20 @@ env_os = await gym.make("OSWorld-Ubuntu")
 
 Environments created this way won't have a default `Task` associated unless you explicitly reset them with one later using `env.reset()`. The `gym.make()` function also automatically links the environment to an active [Job](/concepts/job) if one was defined using the `@job` decorator.
 
+## Available Environment Types
+
+The HUD SDK provides several standard environment types, specified via the `gym` attribute in a [Task](/concepts/task) or directly in `hud.gym.make()`:
+
+*   **`"hud-browser"`**: Provides a remote Chromium browser instance managed via Playwright. Ideal for web navigation, form interaction, and testing web applications.
+    *   [See `hud-browser` Details](../environments/hud-browser.mdx)
+*   **`"hud-ubuntu"`**: Provides a remote Ubuntu desktop environment accessed via VNC. Suitable for tasks involving GUI applications, file system interaction, or running Linux software.
+    *   [See `hud-ubuntu` Details](../environments/hud-ubuntu.mdx)
+*   **`"qa"`**: A non-interactive environment for question-answering tasks where the agent provides a direct textual response.
+    *   [See `qa` Environment Details](../environments/qa.mdx)
+*   **`CustomGym`**: Allows defining and running your own [Custom Environments](../advanced/custom-environments.mdx) using Docker, either locally or remotely. This provides maximum flexibility for specific testing needs.
+
+The `gym` attribute in a Task tells `hud.gym.make()` which environment to instantiate.
+
 ## Interaction Loop
 
 The standard interaction flow involves the [Agent](/concepts/agent) and the Environment:

{hud_python-0.2.1 → hud_python-0.2.2}/docs/concepts/task.mdx

@@ -30,11 +30,10 @@ task = Task(
     prompt="Log in to example.com with username 'test'",
     gym="hud-browser", # Request a browser environment
     setup=[ # Actions run by gym.make(task)
-        ("goto", "https://example.com/login"),
-        {"function": "wait_for_element", "args": ["#username"]}
+        ("goto", "https://example.com/login")
     ],
     evaluate={ # Logic run by env.evaluate()
-        "function": "check_login_status", 
+        "function": "page_contains", 
         "args": ["test"]
     }
 )
@@ -82,11 +81,13 @@ Load predefined sets from the HUD platform:
 ```python
 from hud import load_taskset
 
-taskset = await load_taskset("OSWorld-Ubuntu-Links")
+taskset = await load_taskset("OSWorld-Ubuntu")
 print(f"Number of tasks: {len(taskset)}") # TaskSet acts like a list
 first_task = taskset[0]
 ```
 
+Currently supported TaskSets available via `load_taskset` include OSWorld, GAIA, and WebVoyager subsets.
+
 ### Creating a TaskSet Manually
 
 ```python

{hud_python-0.2.1 → hud_python-0.2.2}/docs/docs.json

@@ -29,6 +29,14 @@
           "concepts/trajectory"
         ]
       },
+      {
+        "group": "Environments",
+        "pages": [
+          "environments/hud-browser",
+          "environments/hud-ubuntu",
+          "environments/qa"
+        ]
+      },
       {
         "group": "Advanced Topics",
         "pages": [

hud_python-0.2.2/docs/environments/hud-browser.mdx

@@ -0,0 +1,67 @@
+# HUD Browser Environment
+
+## Introduction
+
+The `hud-browser` environment provides a remote Chromium browser instance, managed by Playwright, for agents to interact with websites. It's ideal for tasks involving web navigation, form filling, information retrieval, and testing web applications.
+
+## Setup
+
+Setup actions for the `hud-browser` are defined in the `setup` attribute of a [Task](../concepts/task.mdx) and executed by `hud.gym.make()`. They typically involve browser controller functions.
+
+*   **`goto(url: str)`**: Navigates the browser to the specified `url`. Automatically prepends `http://` if no scheme is provided. Waits for `domcontentloaded` (up to 10s timeout) and adds a 1s wait for rendering.
+    ```python
+    # Example Task Setup:
+    setup=[("goto", "https://google.com")]
+    ```
+*   **Other common setup functions coming soon:** `wait_for_element`, `click`, `type`, `set_cookies` etc. 
+
+Refer to [Task Setup Configuration](../concepts/task.mdx#setup-configuration) for how to define these.
+
+## Step Interaction
+
+Agents interact with the browser environment by sending a list of [CLA Actions](../advanced/cla-details.mdx) to `env.step()`. An [Adapter](../concepts/adapter.mdx) typically handles the conversion from the agent model's output to the CLA format.
+
+Common CLAs used with `hud-browser`:
+*   [`ClickAction`](../advanced/cla-details.mdx#mouse-actions)
+*   [`MoveAction`](../advanced/cla-details.mdx#mouse-actions)
+*   [`TypeAction`](../advanced/cla-details.mdx#keyboard-actions)
+*   [`PressAction`](../advanced/cla-details.mdx#keyboard-actions)
+*   [`ScrollAction`](../advanced/cla-details.mdx#mouse-actions)
+*   [`DragAction`](../advanced/cla-details.mdx#mouse-actions)
+*   [`ResponseAction`](../advanced/cla-details.mdx#response-actions) (to submit a final text answer)
+
+*See [CLA Action Details](../advanced/cla-details.mdx) for the full specification.*
+
+## Evaluate
+
+The `evaluate` attribute of a [Task](../concepts/task.mdx) defines how success is measured using `env.evaluate()`. This calls functions within the browser controller.
+
+Built-in evaluation functions for `hud-browser`:
+
+*   **`url_match(expected_url: str)`**: Checks if the current browser URL exactly matches `expected_url`. Returns `1.0` for a match, `0.0` otherwise.
+    ```python
+    # Example Task Evaluation:
+    evaluate=("url_match", "https://google.com/search?q=expected")
+    ```
+*   **`page_contains(texts: list[str])`** (alias `contains_text`): Checks if *all* strings in `texts` are present in `page.content()`. Returns `1.0` if all texts are found, `0.0` otherwise.
+    ```python
+    # Example Task Evaluation:
+    evaluate=("page_contains", ["Search Results", "About 1,000,000 results"])
+    ```
+*   **`sheet_contains(texts: list[str])`**: Custom function for Google Sheets. Returns `1.0` if any text is found, `0.0` otherwise.
+    ```python
+    # Example Task Evaluation:
+    evaluate=("sheet_contains", ["Expected value in cell A1"])
+    ```
+*   **`cookie_exists(cookie_names: list[str])`**: Checks if all cookies in `cookie_names` exist in `context.cookies()`. Returns `1.0` if all exist, `0.0` otherwise.
+    ```python
+    # Example Task Evaluation:
+    evaluate=("cookie_exists", ["session_id", "user_pref"])
+    ```
+*   **`cookie_match(name_value_pairs: list[str])`**: Checks if cookies exist *and* match expected values. `name_value_pairs` format: `[name1, value1, name2, value2, ...]`. Returns `1.0` if all match, `0.0` otherwise.
+    ```python
+    # Example Task Evaluation:
+    evaluate=("cookie_match", ["user_id", "12345", "theme", "dark"])
+    ```
+
+Refer to [Task Evaluation Configuration](../concepts/task.mdx#evaluation-configuration) for more details. 

hud_python-0.2.2/docs/environments/hud-ubuntu.mdx

@@ -0,0 +1,55 @@
+# HUD Ubuntu Environment
+
+## Introduction
+
+The `hud-ubuntu` environment provides a remote Ubuntu OS instance with a graphical desktop, accessed via a VNC connection displayed in the browser. It's suitable for tasks requiring interaction with GUI applications, the file system, or running specific software within a Linux desktop environment.
+
+## Setup
+
+The environment setup simply launches the Ubuntu desktop session within the VNC viewer. 
+
+*Specific pre-launch setup functions (e.g., pre-installing packages, setting environment variables) are planned for future releases.*
+
+Currently, any necessary setup (like installing software or creating files) must be performed by the agent *after* the environment starts, using standard interaction actions.
+
+Refer to [Task Setup Configuration](../concepts/task.mdx#setup-configuration) for the general concept of how setup steps *could* be defined in a Task, although they are not currently implemented for this specific environment.
+
+## Step Interaction
+
+Agents interact with the Ubuntu desktop environment by sending a list of [CLA Actions](../advanced/cla-details.mdx) to `env.step()`. An [Adapter](../concepts/adapter.mdx) typically handles the conversion from the agent model's output to the CLA format.
+
+Available CLA actions for interacting with the graphical desktop:
+
+**Keyboard Actions:**
+*   [`TypeAction`](../advanced/cla-details.mdx#keyboard-actions): Simulates typing text into the focused application or window element.
+    ```python
+    # Example: Typing into a text editor
+    TypeAction(text="Hello, Ubuntu!")
+    ```
+*   [`PressAction`](../advanced/cla-details.mdx#keyboard-actions): For sending hotkeys (e.g., `Ctrl+C`, `Alt+F4`).
+*   [`KeyDownAction` / `KeyUpAction`](../advanced/cla-details.mdx#keyboard-actions): For holding/releasing modifier keys (e.g., holding Shift while clicking).
+
+**Mouse Actions:**
+*   [`ClickAction`](../advanced/cla-details.mdx#mouse-actions): To click on GUI elements (buttons, icons, menus, etc.).
+*   [`MoveAction`](../advanced/cla-details.mdx#mouse-actions): To move the mouse cursor to specific coordinates or elements.
+*   [`ScrollAction`](../advanced/cla-details.mdx#mouse-actions): To scroll within windows or applications.
+*   [`DragAction`](../advanced/cla-details.mdx#mouse-actions): To perform drag-and-drop operations.
+
+**Control & Response Actions:**
+*   [`WaitAction`](../advanced/cla-details.mdx#control-actions): To introduce pauses if needed.
+*   [`ResponseAction`](../advanced/cla-details.mdx#response-actions): Used by the agent to submit its final answer or result text.
+
+**Other Actions (less common for direct agent use):**
+*   [`ScreenshotFetch`](../advanced/cla-details.mdx#fetch-actions-get-information): Usually handled internally by the environment/agent loop.
+*   [`PositionFetch`](../advanced/cla-details.mdx#fetch-actions-get-information): Usually handled internally.
+*   [`CustomAction`](../advanced/cla-details.mdx#custom-actions): For potential future custom environment extensions.
+
+*See [CLA Action Details](../advanced/cla-details.mdx) for the full specification of each action and its parameters.*
+
+## Evaluate
+
+*Specific evaluation functions for `hud-ubuntu` (e.g., checking file content, application state, process status) are planned for future releases.*
+
+Currently, evaluation often relies on the agent submitting a final answer via `ResponseAction`, which can then be checked using generic QA evaluators defined in the [Task](../concepts/task.mdx) (like `response_includes`, `response_matches`). Alternatively, evaluation might involve visually inspecting the final state via the VNC connection or checking logs if the agent was tasked with producing specific output.
+
+Refer to [Task Evaluation Configuration](../concepts/task.mdx#evaluation-configuration) for the general concept of defining evaluation steps. 

hud_python-0.2.2/docs/environments/qa.mdx

@@ -0,0 +1,68 @@
+# QA Environment
+
+## Introduction
+
+The `qa` environment is a specialized, non-interactive environment designed for question-answering tasks. The agent receives context or a question via the `Task.prompt` and is expected to provide a final text response.
+
+## Setup
+
+No environment-specific setup actions are typically required for `qa` tasks. The question or context is provided directly in the `Task.prompt`.
+
+```python
+from hud.task import Task
+
+qa_task = Task(
+    prompt="What is the powerhouse of the cell?",
+    gym="qa" # Specify the QA environment
+    # evaluate=... (see below)
+)
+```
+
+Refer to [Task Setup Configuration](../concepts/task.mdx#setup-configuration) for the general concept of how setup steps *could* be defined in a Task, although they are generally not needed for `qa`.
+
+## Step Interaction
+
+Agents interact with the `qa` environment primarily by submitting their final answer.
+
+*   The agent receives the `Task.prompt` in the initial `Observation`.
+*   The agent processes the prompt and determines its answer.
+*   The agent sends a single [`ResponseAction`](../advanced/cla-details.mdx#response-actions) containing the answer text to `env.step()`.
+
+```python
+# Agent predicts and sends this action:
+from hud.adapters.common.types import ResponseAction
+
+action = ResponseAction(text="Mitochondria")
+# await env.step([action])
+```
+
+The environment stores the text from the first `ResponseAction` it receives in an internal `env.final_response` attribute for evaluation.
+
+**Other CLAs:** While technically part of the [CLA standard](../advanced/cla-details.mdx), other actions (like `ClickAction`, `TypeAction`, `ScrollAction`, etc.) are not processed or relevant in the standard `qa` environment.
+
+## Evaluate
+
+Evaluation logic is defined in the `evaluate` attribute of the [Task](../concepts/task.mdx) and triggered by `env.evaluate()`. This logic compares the `env.final_response` (the text submitted by the agent via `ResponseAction`) against expected criteria.
+
+Common evaluation methods for `qa` tasks:
+*   **`response_includes(substring: str | list[str])`**: Checks if the response text contains the specified `substring` or *all* of the substrings in the provided list.
+    ```python
+    # Example Task Evaluation (single string):
+    evaluate=("response_includes", "mitochondria") 
+    # Example Task Evaluation (list of strings):
+    # evaluate=("response_includes", ["powerhouse", "cell"]) 
+    ```
+*   **`response_is(expected_text: str)`**: Checks for an exact, case-sensitive match with the `expected_text`.
+    ```python
+    # Example Task Evaluation:
+    evaluate=("response_is", "Mitochondria")
+    ```
+*   **`response_match(pattern: str)`**: Checks if the response text matches the provided regular expression `pattern`.
+    ```python
+    # Example Task Evaluation:
+    evaluate=("response_match", r"^[Mm]itochondria.?$") # Case-insensitive, optional period
+    ```
+
+*Note: The exact names and availability of evaluation functions might evolve. Refer to specific evaluator documentation or examples for the most current details.*
+
+Refer to [Task Evaluation Configuration](../concepts/task.mdx#evaluation-configuration) for more details on defining evaluation logic. 

{hud_python-0.2.1 → hud_python-0.2.2}/docs/quickstart.mdx

@@ -71,7 +71,7 @@ async def main():
     # 4. Interaction Loop: Agent observes and acts
     print("Starting interaction loop...")
     # Get initial observation (screenshot, text, etc.) by stepping without actions
-    obs, _ = env.reset()
+    obs, _ = await env.reset()
 
     for i in range(5): # Limit to 5 steps for this example
         print(f"--- Step {i+1} ---")
@@ -89,10 +89,6 @@ async def main():
     result = await env.evaluate()       # Run the evaluation defined in the Task
     print(f"Evaluation result: {result}")
 
-    # Trajectory is automatically saved if a @job decorator is used
-    # trajectory = await env.get_trajectory() # You can optionally get trajectory data
-    # print(f"Trajectory ID: {trajectory.id}")
-
     print("Closing environment...")
     await env.close()                   # Clean up environment resources