hud-python 0.2.4__tar.gz → 0.2.5__tar.gz

{hud_python-0.2.4 → hud_python-0.2.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hud-python
-Version: 0.2.4
+Version: 0.2.5
 Summary: SDK for the HUD evaluation platform.
 Project-URL: Homepage, https://github.com/hud-evals/hud-sdk
 Project-URL: Bug Tracker, https://github.com/hud-evals/hud-sdk/issues
@@ -38,11 +38,13 @@ Classifier: Programming Language :: Python :: 3.13
 Requires-Python: <3.14,>=3.10
 Requires-Dist: aiodocker>=0.24.0
 Requires-Dist: anthropic
+Requires-Dist: dotenv>=0.9.9
 Requires-Dist: httpx<1,>=0.23.0
 Requires-Dist: inspect-ai>=0.3.80
 Requires-Dist: ipykernel
 Requires-Dist: langchain
 Requires-Dist: langchain-openai
+Requires-Dist: mcp
 Requires-Dist: numpy
 Requires-Dist: openai
 Requires-Dist: pillow>=11.1.0
@@ -50,6 +52,7 @@ Requires-Dist: pydantic-settings<3,>=2
 Requires-Dist: pydantic<3,>=2
 Requires-Dist: textdistance<5,>=4.5.0
 Requires-Dist: toml>=0.10.2
+Requires-Dist: wrapt>=1.14.0
 Provides-Extra: dev
 Requires-Dist: anthropic; extra == 'dev'
 Requires-Dist: dotenv; extra == 'dev'
@@ -66,15 +69,68 @@ Requires-Dist: pytest<9,>=8.1.1; extra == 'dev'
 Requires-Dist: ruff==0.11.8; extra == 'dev'
 Description-Content-Type: text/markdown
 
-# HUD
+<div align="left">
+  <img src="https://raw.githubusercontent.com/hud-evals/hud-sdk/main/docs/logo/hud_logo.svg" alt="HUD" width="150" style="margin-bottom: 20px;"/>
+</div>
+
+<h3>
+Create, evaluate, and improve AI agents across web browsers, desktop environments, and custom scenarios.
+</h3>
+
+> ### 🚀 Are you a startup building agents?
+> 
+> [📅 Hop on a call ](https://cal.com/jay-ram-z6st6w/demo) or [📧 founders@hud.so](mailto:founders@hud.so)
+> 
+> We're here to help with eval strategies, custom environments, or improving your agent architecture!
 
-A Python SDK for creating, evaluating, and benchmarking agent interactions with web browsers and OS environments.
 
 > **Early Release Notice**: This SDK is currently in early release status. The API is evolving and may change in future releases as we gather feedback and improve functionality.
 
 [![PyPI version](https://img.shields.io/pypi/v/hud-python)](https://pypi.org/project/hud-python/)
 
-[📚 Documentation](https://documentation.hud.so) | [🏠 Homepage](https://hud.so)
+## ✨ What You Can Do
+
+**Evaluate Existing Benchmarks**
+```python
+from hud import load_taskset, run_job, ClaudeAgent
+
+taskset = await load_taskset("WebVoyager")  # or GAIA, OSWorld-Ubuntu, Mind2Web
+job = await run_job(ClaudeAgent, taskset, "my-evaluation")
+```
+
+**Create Custom Tasks**
+```python
+from hud.task import Task
+
+task = Task(
+    prompt="Find and book the cheapest flight from NYC to Paris",
+    gym="hud-browser",
+    setup=("goto", "https://kayak.com"),
+    evaluate=("page_contains", "confirmation")
+)
+```
+
+**Build Custom Environments**
+```python
+from hud.types import CustomGym
+
+# Launch any website as an environment
+custom_gym = CustomGym(
+    image_or_build_context="nginx:alpine",
+    location="local"
+)
+
+# Or create complex Docker environments - see environments/ folder for examples
+```
+
+**Trace Tool Calls Alongside HUD Environments (or Independently)**
+```python
+import hud
+
+with hud.trace("my-agent-run"):
+    # Your agent code here - MCP calls automatically captured
+    result = await agent.run(task)
+```
 
 ## API Key Setup
 
@@ -119,7 +175,7 @@ async def main():
     # Create environment using the gym module
     env = await gym.make(task)
     
-    # Initialize Operator agent (API key is loaded automatically)
+    # Initialize Claude agent (API key is loaded automatically)
     agent = ClaudeAgent()
     
     # Agent loop with predict and step functions
@@ -137,7 +193,6 @@ async def main():
 
 if __name__ == "__main__":
     asyncio.run(main())
-
 ```
 
 Alternatively, run a full evaluation set via the ```run_job``` command:
@@ -145,32 +200,45 @@ Alternatively, run a full evaluation set via the ```run_job``` command:
 ```python
 from hud import load_taskset, run_job, ClaudeAgent
 
-# load
+# Load a benchmark
 taskset = load_taskset("GAIA")
 
-# evaluate
+# Evaluate
 job = await run_job(ClaudeAgent, taskset, "test-gaia-job")
 
-# get results OR view them in app.hud.so
+# Get results OR view them in app.hud.so
 print(await job.get_analytics())
 ```
 
+## Ready-to-Use TaskSets
+
+- **WebVoyager** - Web navigation and interaction
+- **Mind2Web** - Complex web application tasks  
+- **GAIA** - Question answering and reasoning
+- **OSWorld-Ubuntu** - Desktop interaction
+- **hud-samples** - Getting started examples
+
+## Community
+
+**Contributing Custom Environments**
+
+Add your environment to the `environments/` folder and submit a PR! Examples:
+- `environments/novnc_ubuntu/` - Ubuntu with VNC access 
+- `environments/pokemon_controller/` - Pokemon emulator environment (In Development)
+- `environments/qa_controller/` - Lightweight app sample
+
+See [Custom Environments Guide](https://docs.hud.so/environment-creation) for details.
+
 ## Documentation Sections
 
 Explore the core concepts and features of the SDK:
 
-*   **[Tasks and TaskSets](https://documentation.hud.so/concepts/task)**: Define goals, context, setup, and evaluation criteria for agent scenarios. This includes both interactive and **question-answering (QA)** style tasks.
-*   **[Environments](https://documentation.hud.so/concepts/environment)**: Understand the browser and OS runtimes where agents interact.
-*   **[Agents](https://documentation.hud.so/concepts/agent)**: Learn about the agent architecture (Claude, Operator) and how they process observations and predict actions.
-*   **[Adapters](https://documentation.hud.so/concepts/adapter)**: See how actions and observations are translated between agents and environments.
-*   **[Jobs](https://documentation.hud.so/concepts/job)**: Group related runs for analysis and viewing on the HUD platform.
-*   **[Trajectories](https://documentation.hud.so/concepts/trajectory)**: Understand the recorded data from each agent run.
-*   **Advanced Topics**:
-    *   **[CLA Action Details](https://documentation.hud.so/advanced/cla-details)**: Explore the standardized action format.
-    *   **[Custom Environments](https://documentation.hud.so/advanced/custom-environments)**: Build your own Docker-based local or remote environments.
-    *   **[Advanced Environment Control](https://documentation.hud.so/advanced/environment-control)**: Use `invoke`, `execute`, and `_setup` for finer control.
-
-*   **[Full API Reference](https://documentation.hud.so/api-reference/gym)**: Detailed specifications for all modules and classes.
+*   **[Task Creation](https://docs.hud.so/task-creation)**: Build custom evaluation scenarios with setup and evaluation criteria.
+*   **[Environments](https://docs.hud.so/environments/browser)**: Understand browser environments and create custom Docker-based environments.
+*   **[Agents](https://docs.hud.so/concepts/agent)**: Learn about the agent architecture (Claude, Operator) and how they process observations and predict actions.
+*   **[Jobs](https://docs.hud.so/concepts/job)**: Group related runs for analysis and viewing on the HUD platform.
+*   **[MCP Telemetry](https://docs.hud.so/telemetry/mcp)**: Automatic tracing of Model Context Protocol interactions.
+*   **[Full API Reference](https://docs.hud.so/api-reference/gym)**: Detailed specifications for all modules and classes.
 
 ## [Examples](examples/)
 
@@ -183,7 +251,7 @@ We recommend you first take a look at the example notebooks showing how to use t
 
 ## Documentation
 
-For comprehensive guides, examples, and API reference, visit [our docs](https://documentation.hud.so/introduction)
+For comprehensive guides, examples, and API reference, visit [our docs](https://docs.hud.so/introduction)
 
 ## License
 

hud_python-0.2.5/README.md

@@ -0,0 +1,201 @@
+<div align="left">
+  <img src="https://raw.githubusercontent.com/hud-evals/hud-sdk/main/docs/logo/hud_logo.svg" alt="HUD" width="150" style="margin-bottom: 20px;"/>
+</div>
+
+<h3>
+Create, evaluate, and improve AI agents across web browsers, desktop environments, and custom scenarios.
+</h3>
+
+> ### 🚀 Are you a startup building agents?
+> 
+> [📅 Hop on a call ](https://cal.com/jay-ram-z6st6w/demo) or [📧 founders@hud.so](mailto:founders@hud.so)
+> 
+> We're here to help with eval strategies, custom environments, or improving your agent architecture!
+
+
+> **Early Release Notice**: This SDK is currently in early release status. The API is evolving and may change in future releases as we gather feedback and improve functionality.
+
+[![PyPI version](https://img.shields.io/pypi/v/hud-python)](https://pypi.org/project/hud-python/)
+
+## ✨ What You Can Do
+
+**Evaluate Existing Benchmarks**
+```python
+from hud import load_taskset, run_job, ClaudeAgent
+
+taskset = await load_taskset("WebVoyager")  # or GAIA, OSWorld-Ubuntu, Mind2Web
+job = await run_job(ClaudeAgent, taskset, "my-evaluation")
+```
+
+**Create Custom Tasks**
+```python
+from hud.task import Task
+
+task = Task(
+    prompt="Find and book the cheapest flight from NYC to Paris",
+    gym="hud-browser",
+    setup=("goto", "https://kayak.com"),
+    evaluate=("page_contains", "confirmation")
+)
+```
+
+**Build Custom Environments**
+```python
+from hud.types import CustomGym
+
+# Launch any website as an environment
+custom_gym = CustomGym(
+    image_or_build_context="nginx:alpine",
+    location="local"
+)
+
+# Or create complex Docker environments - see environments/ folder for examples
+```
+
+**Trace Tool Calls Alongside HUD Environments (or Independently)**
+```python
+import hud
+
+with hud.trace("my-agent-run"):
+    # Your agent code here - MCP calls automatically captured
+    result = await agent.run(task)
+```
+
+## API Key Setup
+
+Before getting started, you'll need to obtain an API key:
+
+1. Visit [app.hud.so](https://app.hud.so) to create a free account and generate your API key
+2. Set it in your environment or .env file:
+
+```bash
+export HUD_API_KEY=your_api_key_here
+```
+
+## Quick Start
+
+### Installation
+
+```bash
+pip install hud-python
+```
+
+### Simple Browser Example with Claude Computer Use
+
+> This example uses the `@register_job("test-run")` decorator, so the results of this run will appear under the job named "test-run" on the your [HUD Jobs page](https://app.hud.so/jobs).
+
+Make sure your have defined your `ANTRHOPIC_API_KEY` in environment variables to run Claude.
+
+```python
+import asyncio
+from hud import gym, register_job
+from hud.task import Task
+from hud.agent import ClaudeAgent
+
+@register_job("test-run")
+async def main():
+    task = Task(
+        prompt="Insert the text 'capybara' into the search bar",
+        gym="hud-browser",
+        setup=("goto", "google.com"),
+        evaluate=("contains_text", "capybara")
+    )
+    
+    # Create environment using the gym module
+    env = await gym.make(task)
+    
+    # Initialize Claude agent (API key is loaded automatically)
+    agent = ClaudeAgent()
+    
+    # Agent loop with predict and step functions
+    obs, _ = await env.reset() # Gets first observation
+    for i in range(5):
+        actions, done = await agent.predict(obs)
+
+        obs, reward, terminated, info = await env.step(actions)
+        if done or terminated: break
+    
+    # Evaluate and close
+    result = await env.evaluate()
+    print(f"Evaluation result: {result}")
+    await env.close()
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+Alternatively, run a full evaluation set via the ```run_job``` command:
+
+```python
+from hud import load_taskset, run_job, ClaudeAgent
+
+# Load a benchmark
+taskset = load_taskset("GAIA")
+
+# Evaluate
+job = await run_job(ClaudeAgent, taskset, "test-gaia-job")
+
+# Get results OR view them in app.hud.so
+print(await job.get_analytics())
+```
+
+## Ready-to-Use TaskSets
+
+- **WebVoyager** - Web navigation and interaction
+- **Mind2Web** - Complex web application tasks  
+- **GAIA** - Question answering and reasoning
+- **OSWorld-Ubuntu** - Desktop interaction
+- **hud-samples** - Getting started examples
+
+## Community
+
+**Contributing Custom Environments**
+
+Add your environment to the `environments/` folder and submit a PR! Examples:
+- `environments/novnc_ubuntu/` - Ubuntu with VNC access 
+- `environments/pokemon_controller/` - Pokemon emulator environment (In Development)
+- `environments/qa_controller/` - Lightweight app sample
+
+See [Custom Environments Guide](https://docs.hud.so/environment-creation) for details.
+
+## Documentation Sections
+
+Explore the core concepts and features of the SDK:
+
+*   **[Task Creation](https://docs.hud.so/task-creation)**: Build custom evaluation scenarios with setup and evaluation criteria.
+*   **[Environments](https://docs.hud.so/environments/browser)**: Understand browser environments and create custom Docker-based environments.
+*   **[Agents](https://docs.hud.so/concepts/agent)**: Learn about the agent architecture (Claude, Operator) and how they process observations and predict actions.
+*   **[Jobs](https://docs.hud.so/concepts/job)**: Group related runs for analysis and viewing on the HUD platform.
+*   **[MCP Telemetry](https://docs.hud.so/telemetry/mcp)**: Automatic tracing of Model Context Protocol interactions.
+*   **[Full API Reference](https://docs.hud.so/api-reference/gym)**: Detailed specifications for all modules and classes.
+
+## [Examples](examples/)
+
+We recommend you first take a look at the example notebooks showing how to use the HUD SDK:
+
+1. [Browser Basics](examples/browser_use.ipynb) - Simple browser interaction with live view
+2. [Task Design](examples/tasks.ipynb) - Creating and customizing tasks
+3. [OSWorld](examples/osworld.ipynb) - Running the OSWorld benchmark
+4. [Local Development](examples/local.ipynb) - Setting up local custom environments
+
+## Documentation
+
+For comprehensive guides, examples, and API reference, visit [our docs](https://docs.hud.so/introduction)
+
+## License
+
+[MIT License](LICENSE)
+
+## Citation
+
+If you use this SDK in your research, please cite it as follows:
+
+```bibtex
+@software{hud2025agentevalplatform,
+  author = {HUD and Jay Ram and Lorenss Martinsons and Parth Patel and Oskars Putans and Govind Pimpale and Mayank Singamreddy and Nguyen Nhat Minh},
+  title = {{HUD: An Evaluation Platform for Agents}},
+  date = {2025-04},
+  url = {https://github.com/hud-evals/hud-sdk},
+  langid = {en}
+}
+```

hud_python-0.2.5/docs/advanced/tracing.mdx

@@ -0,0 +1,129 @@
+---
+title: 'Tracing'
+description: 'Capture and analyze MCP calls with tracing functionality'
+---
+
+# Tracing
+
+The HUD SDK provides tracing functionality to capture and analyze MCP (Model-Client-Provider) calls during agent execution. This is particularly useful for debugging, performance analysis, and understanding the interaction between your agent and external services.
+
+## Overview
+
+Tracing in HUD allows you to:
+
+-   Capture all MCP calls made by an agent within a specific code block or decorated function.
+-   Automatically upload these traces to the HUD platform ([app.hud.so](https://app.hud.so)) for detailed analysis.
+-   View comprehensive information about each call, including request/response payloads, timing, status, and any errors.
+-   Associate traces with specific task runs, jobs, and custom attributes for better organization and filtering.
+
+This capability is essential whether your agent is interacting with HUD environments (like `hud-browser`) or using its own set of MCP-based tools independently.
+
+## Using Tracing
+
+There are two main ways to use tracing in your code:
+
+### 1. Context Manager: `hud.trace()`
+
+Use the `trace()` context manager to wrap a block of code where you want to capture MCP calls:
+
+```python
+import hud
+from mcp_use import MCPAgent, MCPClient
+from langchain_openai import ChatOpenAI
+
+# Create MCP components
+client = MCPClient.from_dict({
+    "mcpServers": {
+        "example_server": {
+            "command": "npx",
+            "args": ["-y", "@example/mcp-server", "--option"],
+            "env": {
+                "OPTION_FLAG": "true"
+            }
+        }
+    }
+})
+llm = ChatOpenAI(model="gpt-4o")
+agent = MCPAgent(llm=llm, client=client, max_steps=5)
+
+# Wrap the agent execution with tracing
+with hud.trace("my_mcp_trace", attributes={"query": "Find information about X"}):
+    result = await agent.run(
+        "Find information about X",
+        max_steps=5,
+    )
+
+# Trace is automatically uploaded and available at https://app.hud.so/jobs/traces/[id]
+```
+
+**Parameters:**
+
+- `name` (str, optional): A name for this trace, useful for identification
+- `attributes` (dict, optional): Additional metadata to associate with the trace
+
+### 2. Decorator: `@hud.register_trace`
+
+Use the `@register_trace` decorator to automatically trace an entire function:
+
+```python
+import hud
+from mcp_use import MCPAgent, MCPClient
+from langchain_openai import ChatOpenAI
+
+@hud.register_trace(name="mcp_search_function", attributes={"type": "search"})
+async def perform_search(query: str):
+    client = MCPClient.from_dict({
+        "mcpServers": {
+            "search_server": {
+                "command": "npx",
+                "args": ["-y", "@search/mcp-server"],
+            }
+        }
+    })
+    llm = ChatOpenAI(model="gpt-4o")
+    agent = MCPAgent(llm=llm, client=client, max_steps=5)
+    
+    return await agent.run(query, max_steps=5)
+
+# Call the function - tracing happens automatically
+result = await perform_search("What is the capital of France?")
+```
+
+**Parameters:**
+
+- `name` (str, optional): A name for this trace, defaults to the function name
+- `attributes` (dict, optional): Additional metadata to associate with the trace
+
+## Viewing Traces
+
+After a trace is captured, it's automatically uploaded to the HUD platform. You'll see a log message with a URL where you can view the trace:
+
+```
+[hud] View trace at https://app.hud.so/jobs/traces/[trace_id]
+```
+
+The trace view shows:
+
+- Timeline of all MCP calls
+- Request and response payloads
+- Timing information
+- Error details (if any)
+
+## Best Practices
+
+- **Use descriptive names**: Choose meaningful names for your traces to make them easier to identify
+- **Add relevant attributes**: Include metadata that will help you filter and analyze traces later
+- **Limit trace scope**: Trace specific sections of code rather than entire applications to keep traces focused
+- **Clean up resources**: Traces are automatically uploaded when the context manager exits or the decorated function completes
+
+## Limitations
+
+- Tracing only captures MCP calls, not other types of API calls or internal function calls
+- Large traces with many calls may take longer to upload and display
+- Trace data is temporarily stored in memory before being uploaded
+
+## Related Concepts
+
+- [Job](/concepts/job): Jobs can contain multiple traces
+- [Environment](/concepts/environment): Environments can be associated with traces
+- [Task](/concepts/task): Tasks can be traced to analyze performance

hud_python-0.2.5/docs/advanced/uploading.mdx

@@ -0,0 +1,160 @@
+---
+title: 'Uploading TaskSets'
+description: 'Learn how to upload TaskSets and view them on the HUD platform'
+---
+
+# Uploading TaskSets
+
+TaskSets are collections of tasks that can be uploaded to the HUD platform for evaluation and sharing. This guide explains how to upload TaskSets and access them through the platform.
+
+## Creating and Uploading a TaskSet
+
+You can create a TaskSet from a list of tasks and upload it to the platform:
+
+```python
+from hud.task import Task
+from hud.taskset import TaskSet
+
+# Create tasks
+tasks = [
+    Task(
+        prompt="Navigate to example.com and verify the login page is displayed",
+        gym="hud-browser",
+        setup=[
+            ("goto", "https://example.com/login")
+        ],
+        evaluate={
+            "function": "page_contains",
+            "args": "Login"
+        }
+    ),
+    Task(
+        prompt="What is the capital of France?",
+        gym="hud-browser",
+        evaluate={
+            "function": "response_includes",
+            "args": "Paris"
+        }
+    )
+]
+
+# Create a TaskSet
+taskset = TaskSet(tasks=tasks)
+
+# Upload the TaskSet
+taskset_id = await taskset.upload("my-taskset")
+print(f"TaskSet uploaded with ID: {taskset_id}")
+```
+
+## TaskSet Parameters
+
+When creating a TaskSet, you can specify:
+
+- `name`: A descriptive name for your TaskSet
+- `description`: Detailed description of what the TaskSet evaluates
+- `tasks`: List of Task objects
+- `metadata`: Optional dictionary of metadata about the TaskSet
+
+## Task Configuration
+
+Each Task in a TaskSet can include:
+
+- `prompt`: The instruction or question for the agent
+- `gym`: The environment type (e.g., "hud-browser", "hud-ubuntu")
+- `setup`: Optional list of setup actions to run before the agent starts
+- `evaluate`: Function configuration to determine task success
+- `id`: Optional unique identifier for the task
+
+Common evaluation functions include:
+- `page_contains`: Checks if specific text exists on the page
+- `response_includes`: Verifies if the agent's final response contains expected text
+- `cookies_exist`: Checks if a set of cookies are present
+
+## Viewing TaskSets on the Platform
+
+After uploading, you can view and manage your TaskSets at [app.hud.so/evalsets](https://app.hud.so/evalsets). The platform provides:
+
+- List of all your uploaded TaskSets
+- Detailed view of individual tasks within each TaskSet
+- Task prompts and evaluation criteria
+- Evaluation results when agents are run against the TaskSet
+
+## Loading an Existing TaskSet
+
+You can load a previously uploaded TaskSet using its name:
+
+```python
+from hud.taskset import TaskSet
+
+# Load an existing TaskSet
+taskset = await TaskSet.load("taskset-name")
+
+# Access tasks
+for task in taskset.tasks:
+    print(f"Task ID: {task.id}")
+    print(f"Prompt: {task.prompt}")
+    print(f"Evaluation: {task.evaluate}")
+```
+
+## Best Practices
+
+1. **Task Organization**
+   - Give tasks clear, unique IDs
+   - Use descriptive prompts
+   - Group related tasks into themed TaskSets
+
+2. **Evaluation Design**
+   - Choose appropriate evaluation functions
+   - Provide clear success criteria
+   - Test evaluation logic before uploading
+
+3. **Documentation**
+   - Write clear task prompts
+   - Document expected agent behavior
+   - Include example solutions where appropriate
+
+## Running Evaluations
+
+Once uploaded, you can run agents against your TaskSet:
+
+```python
+from hud.job import run_job
+from my_agent import MyAgent
+
+# Run evaluation using the TaskSet
+job = await run_job(
+    agent_cls=MyAgent,
+    task_or_taskset=taskset,
+    job_name="Evaluation Run"
+)
+
+# View results on app.hud.so/jobs/{job.id}
+print(f"View results at: https://app.hud.so/jobs/{job.id}")
+```
+
+## Platform Features
+
+The HUD platform ([app.hud.so](https://app.hud.so)) provides several features for working with TaskSets:
+
+1. **TaskSet Management**
+   - Browse all uploaded TaskSets
+   - View individual task details
+   - Filter and search tasks
+   - Track evaluation history
+
+2. **Analysis Tools**
+   - Compare agent performance
+   - View detailed task results
+   - Export evaluation data
+   - Share results with team members
+
+3. **Collaboration**
+   - Share TaskSets with team members
+   - Collaborate on task creation
+   - Track changes and versions
+
+## Related Topics
+
+- [Task Creation](../concepts/task.mdx)
+- [Running Evaluations](../running-your-agent.mdx)
+- [Job Management](../concepts/job.mdx)