openadapt-ml 0.2.2__py3-none-any.whl → 0.3.1__py3-none-any.whl

openadapt_ml/benchmarks/vm_monitor.py

@@ -154,11 +154,14 @@ class VMMonitor:
     def check_waa_probe(self) -> tuple[bool, str | None]:
         """Check if WAA /probe endpoint responds.
 
+        The probe must run INSIDE the container to reach 172.30.0.2 (Docker internal network).
+
         Returns:
             Tuple of (ready, response_text).
         """
         try:
-            cmd = f"curl -s --connect-timeout {self.timeout} http://{self.config.internal_ip}:{self.config.waa_port}/probe"
+            # Run curl inside container to access Docker internal network
+            cmd = f"docker exec {self.config.docker_container} curl -s --max-time {self.timeout} http://{self.config.internal_ip}:{self.config.waa_port}/probe 2>/dev/null || echo FAIL"
             result = subprocess.run(
                 [
                     "ssh",
@@ -173,10 +176,10 @@ class VMMonitor:
                 ],
                 capture_output=True,
                 text=True,
-                timeout=self.timeout + 10,
+                timeout=self.timeout + 15,
             )
             response = result.stdout.strip()
-            if response and "error" not in response.lower():
+            if response and "FAIL" not in response and "error" not in response.lower():
                 return True, response
             return False, response or None
         except (subprocess.TimeoutExpired, Exception) as e:

{openadapt_ml-0.2.2.dist-info → openadapt_ml-0.3.1.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: openadapt-ml
-Version: 0.2.2
+Version: 0.3.1
 Summary: Model-agnostic, domain-agnostic ML engine for GUI automation agents
 Project-URL: Homepage, https://github.com/OpenAdaptAI/openadapt-ml
 Project-URL: Repository, https://github.com/OpenAdaptAI/openadapt-ml
@@ -58,7 +58,7 @@ Description-Content-Type: text/markdown
 
 # OpenAdapt-ML
 
-[![Build Status](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/publish.yml/badge.svg)](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/publish.yml)
+[![Build Status](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/release.yml/badge.svg)](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/release.yml)
 [![PyPI version](https://img.shields.io/pypi/v/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/)
 [![Downloads](https://img.shields.io/pypi/dm/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
@@ -88,6 +88,38 @@ The design is described in detail in [`docs/design.md`](docs/design.md).
 
 ---
 
+## Parallel WAA Benchmark Evaluation (New in v0.3.0)
+
+Run Windows Agent Arena benchmarks across multiple Azure VMs in parallel for faster evaluation:
+
+```bash
+# Create a pool of 5 workers
+uv run python -m openadapt_ml.benchmarks.cli pool-create --workers 5
+
+# Wait for all workers to be ready
+uv run python -m openadapt_ml.benchmarks.cli pool-wait
+
+# Run 154 tasks distributed across workers (~5x faster)
+uv run python -m openadapt_ml.benchmarks.cli pool-run --tasks 154
+```
+
+**Key features:**
+- **Parallel execution**: Distribute 154 WAA tasks across N workers
+- **Automatic task distribution**: Uses WAA's native `--worker_id`/`--num_workers` for round-robin assignment
+- **VNC access**: View each Windows VM via SSH tunnels (`localhost:8006`, `localhost:8007`, etc.)
+- **Cost tracking**: Monitor Azure VM costs in real-time
+
+**Performance:**
+| Workers | Estimated Time (154 tasks) |
+|---------|---------------------------|
+| 1       | ~50-80 hours              |
+| 5       | ~10-16 hours              |
+| 10      | ~5-8 hours                |
+
+See [WAA Benchmark Workflow](#waa-benchmark-workflow) for complete setup instructions.
+
+---
+
 ## 1. Installation
 
 ### 1.1 From PyPI (recommended)
@@ -1029,7 +1061,112 @@ uv run python -m openadapt_ml.benchmarks.cli screenshot --target terminal --no-t
 
 ---
 
-## 14. Limitations & Notes
+<a id="waa-benchmark-workflow"></a>
+
+## 14. WAA Benchmark Workflow
+
+Windows Agent Arena (WAA) is a benchmark of 154 tasks across 11 Windows domains. OpenAdapt-ML provides infrastructure to run WAA evaluations on Azure VMs with parallel execution.
+
+### 14.1 Prerequisites
+
+1. **Azure CLI**: `brew install azure-cli && az login`
+2. **OpenAI API Key**: Set in `.env` file (`OPENAI_API_KEY=sk-...`)
+3. **Azure quota**: Ddsv5 family VMs (8+ vCPUs per worker)
+
+### 14.2 Single VM Workflow
+
+For quick testing or small runs:
+
+```bash
+# Setup VM with WAA
+uv run python -m openadapt_ml.benchmarks.cli vm setup-waa
+
+# Start monitoring dashboard (auto-opens VNC, manages SSH tunnels)
+uv run python -m openadapt_ml.benchmarks.cli vm monitor
+
+# Run benchmark
+uv run python -m openadapt_ml.benchmarks.cli waa --num-tasks 10
+
+# Deallocate when done (stops billing)
+uv run python -m openadapt_ml.benchmarks.cli vm deallocate -y
+```
+
+### 14.3 Parallel Pool Workflow (Recommended)
+
+For full 154-task evaluations, use multiple VMs:
+
+```bash
+# 1. Create pool (provisions N Azure VMs with Docker + WAA)
+uv run python -m openadapt_ml.benchmarks.cli pool-create --workers 5
+
+# 2. Wait for all workers to be ready (Windows boot + WAA server startup)
+uv run python -m openadapt_ml.benchmarks.cli pool-wait
+
+# 3. Run benchmark across all workers
+#    Tasks are distributed using WAA's native --worker_id/--num_workers
+uv run python -m openadapt_ml.benchmarks.cli pool-run --tasks 154
+
+# 4. Monitor progress
+uv run python -m openadapt_ml.benchmarks.cli pool-status
+uv run python -m openadapt_ml.benchmarks.cli pool-logs
+
+# 5. Cleanup (delete all VMs - IMPORTANT to stop billing!)
+uv run python -m openadapt_ml.benchmarks.cli pool-cleanup
+```
+
+### 14.4 VNC Access to Workers
+
+View what each Windows VM is doing:
+
+```bash
+# Get worker IPs
+uv run python -m openadapt_ml.benchmarks.cli pool-status
+
+# Set up SSH tunnels (tunnels are created automatically, but you can also do this manually)
+ssh -f -N -L 8006:localhost:8006 azureuser@<worker-0-ip>  # localhost:8006
+ssh -f -N -L 8007:localhost:8006 azureuser@<worker-1-ip>  # localhost:8007
+# etc.
+
+# Open in browser
+open http://localhost:8006  # Worker 0
+open http://localhost:8007  # Worker 1
+```
+
+### 14.5 Architecture
+
+```
+Local Machine
+├── openadapt-ml CLI (pool-create, pool-wait, pool-run)
+│   └── SSH tunnels to each worker
+│
+Azure (N VMs, Standard_D8ds_v5)
+├── waa-pool-00
+│   └── Docker
+│       └── windowsarena/winarena:latest
+│           └── QEMU (Windows 11)
+│               ├── WAA Flask server (port 5000)
+│               └── Navi agent (GPT-4o-mini)
+├── waa-pool-01
+│   └── ...
+└── waa-pool-N
+    └── ...
+```
+
+### 14.6 Cost Estimates
+
+| VM Size | vCPUs | RAM | Cost/hr | 5 VMs for 10hrs |
+|---------|-------|-----|---------|-----------------|
+| Standard_D8ds_v5 | 8 | 32GB | ~$0.38 | ~$19 |
+
+**Tips:**
+- Always run `pool-cleanup` when done to delete VMs and stop billing
+- Use `vm deallocate` (not delete) to pause billing but keep disk
+- Set `--auto-shutdown-hours 2` on `vm monitor` for safety
+- Prices vary by Azure region
+
+---
+
+## 15. Limitations & Notes
 
 - **Apple Silicon / bitsandbytes**:
   - Example configs are sized for CPU / Apple Silicon development runs; see
@@ -1053,7 +1190,7 @@ For deeper architectural details, see [`docs/design.md`](docs/design.md).
 
 ---
 
-## 15. Roadmap
+## 16. Roadmap
 
 For the up-to-date, prioritized roadmap (including concrete implementation
 targets and agent-executable acceptance criteria), see

{openadapt_ml-0.2.2.dist-info → openadapt_ml-0.3.1.dist-info}/RECORD

@@ -10,10 +10,10 @@ openadapt_ml/benchmarks/__init__.py,sha256=FaEGc7pRM-eLUXEEpJXcIckwkIWKhfaDkaxGM
 openadapt_ml/benchmarks/agent.py,sha256=8UcS9skCy6l18fGYaYt0JzJmYSGNB_WxDWhApbM7QH0,26940
 openadapt_ml/benchmarks/azure.py,sha256=dCrxi90X5NmFNMTT-2WG4AF3-IOO4zQs7yPpnqR-jLc,28238
 openadapt_ml/benchmarks/azure_ops_tracker.py,sha256=NOW21LPagOWIThSCIotI5cBvve92dtIktRIDLuyJ2CI,19309
-openadapt_ml/benchmarks/cli.py,sha256=t4cIGN68GdphCX0AGkWJa_M6D4oUO_M0rfJDzD_POGA,62730
+openadapt_ml/benchmarks/cli.py,sha256=DwBZJEZF2XwajAazWWXxePbuH7J_W8G9N0y7iv3l7FI,288566
 openadapt_ml/benchmarks/trace_export.py,sha256=Zx-pryEuLe734YHY8MgJsNdj3I3TcTY61OQ9iurgGB0,21746
 openadapt_ml/benchmarks/viewer.py,sha256=Jztt_IoDW1u0WjPqlikfR8dunYzj66xCx0bMDDzJHQ8,41586
-openadapt_ml/benchmarks/vm_monitor.py,sha256=FzmRrzqm0sZTcydfqMtRefBLfTr4fjoyWCxdHLovUj0,35733
+openadapt_ml/benchmarks/vm_monitor.py,sha256=EYgPRok2MPqs8Yajg7EJaqyb4EtRpqt8URQMLhE9Ego,35991
 openadapt_ml/benchmarks/waa_deploy/Dockerfile,sha256=F4GzVUoAUHvGlTFj-gGIPSlncG-JIz1_JyeaHvTnLpA,10853
 openadapt_ml/benchmarks/waa_deploy/__init__.py,sha256=KV71HrrgETytfY0i4vFSi-yM0KjoQP2hd9Bl03cZ9yc,320
 openadapt_ml/benchmarks/waa_deploy/api_agent.py,sha256=A5ZFhtBTKz0Q1GarNV51JhkEJwAgJfm9tK4CTJ1UEnE,20040
@@ -110,7 +110,7 @@ openadapt_ml/training/trainer.py,sha256=yGK79alY9Z0xGRQ2r9EaiWbzGlmE5WZJQL_2TWgc
 openadapt_ml/training/trl_trainer.py,sha256=AL1KFWXMub4vWE2w8eoAoQbSgm2fXO82CIqXULLYwVo,13223
 openadapt_ml/training/viewer.py,sha256=rXpREFbDK_tsu719VUej6iXrgnB8eNP0SEuvB9NUUhA,128104
 openadapt_ml/training/viewer_components.py,sha256=XilaX7r4YXFMT1QkooNnPWqR14SpsiTf7YbrN_g-Lq0,5478
-openadapt_ml-0.2.2.dist-info/METADATA,sha256=XNDolFy-sWkfPPCZ36qFRUwKzMRLk9WhzLMVufPp8i8,36696
-openadapt_ml-0.2.2.dist-info/WHEEL,sha256=WLgqFyCfm_KASv4WHyYy0P3pM_m7J5L9k2skdKLirC8,87
-openadapt_ml-0.2.2.dist-info/licenses/LICENSE,sha256=2E5UY67RVLedJuNnwGudkAMtfM3LZNUcHgmaL89TAfw,1068
-openadapt_ml-0.2.2.dist-info/RECORD,,
+openadapt_ml-0.3.1.dist-info/METADATA,sha256=h5Xf2LEjMlBOsuwDCRiF5_cGlEwlgbRp8Vkqw1HOo4Q,40990
+openadapt_ml-0.3.1.dist-info/WHEEL,sha256=WLgqFyCfm_KASv4WHyYy0P3pM_m7J5L9k2skdKLirC8,87
+openadapt_ml-0.3.1.dist-info/licenses/LICENSE,sha256=2E5UY67RVLedJuNnwGudkAMtfM3LZNUcHgmaL89TAfw,1068
+openadapt_ml-0.3.1.dist-info/RECORD,,