openadapt-ml 0.2.2__tar.gz → 0.3.1__tar.gz

{openadapt_ml-0.2.2 → openadapt_ml-0.3.1}/CHANGELOG.md

@@ -1,6 +1,158 @@
 # CHANGELOG
 
 
+## v0.3.1 (2026-02-05)
+
+### Bug Fixes
+
+- **cli**: Resolve ruff linter errors
+  ([`210a31f`](https://github.com/OpenAdaptAI/openadapt-ml/commit/210a31fcc054238a08e609520bdf57c312600d72))
+
+- Replace bare `except:` with `except Exception:` - Remove unused f-string prefixes - Remove unused
+  variable assignments - Remove unused imports
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+### Documentation
+
+- **readme**: Add parallel WAA evaluation, fix build badge
+  ([#19](https://github.com/OpenAdaptAI/openadapt-ml/pull/19),
+  [`fea0a10`](https://github.com/OpenAdaptAI/openadapt-ml/commit/fea0a10c514b87a8a73310a142acb73a6b31146e))
+
+* docs(readme): add parallel WAA evaluation section, fix build badge
+
+- Fix broken build badge (publish.yml → release.yml) - Add prominent "Parallel WAA Benchmark
+  Evaluation" section near top - Add detailed "WAA Benchmark Workflow" section (#14) with: - Single
+  VM and parallel pool workflows - VNC access instructions - Architecture diagram - Cost estimates -
+  Update section numbering (Limitations → 15, Roadmap → 16)
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+* fix(readme): address self-review feedback
+
+- Fix anchor placement (move before heading for proper navigation) - Correct pool-delete →
+  pool-cleanup (actual command name) - Add pool-status example for getting worker IPs - Add "prices
+  vary by region" caveat
+
+---------
+
+Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
+
+
+## v0.3.0 (2026-02-05)
+
+### Bug Fixes
+
+- **cli**: Improve pool-create reliability and error handling
+  ([`f23bd57`](https://github.com/OpenAdaptAI/openadapt-ml/commit/f23bd571a76c361d9e46d99820728ffdedb5cef5))
+
+- Properly clean up test VM and associated resources during quota check - Use sudo for docker pull
+  (usermod not effective in same session) - Add pool-cleanup command for orphaned resources - Show
+  full error messages in pool creation failures
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+- **pool**: Use WAA native task distribution with --worker_id/--num_workers
+  ([`ef0d8c7`](https://github.com/OpenAdaptAI/openadapt-ml/commit/ef0d8c7ecf60b1644dbc5a40ed0a05b1b4c2f597))
+
+- Fixed task distribution: WAA ignores --start_idx/--num_tasks, use native --worker_id and
+  --num_workers parameters instead - Worker 0 gets tasks 0, N, 2N... Worker 1 gets tasks 1, N+1,
+  2N+1... - Use vanilla windowsarena/winarena image with correct IP (20.20.20.21) - Add container
+  reuse check (skip restart if already running) - Pass API key via env var instead of config file -
+  Fix QMP port exposure (7200) for QEMU control - Store Windows disk on /mnt for 300GB temp storage
+  (D8ds_v5)
+
+Tested: 2-worker pool running 4 tasks in parallel successfully
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+- **waa**: Use D4ds_v4 VM size for quota compatibility
+  ([`2a51a97`](https://github.com/OpenAdaptAI/openadapt-ml/commit/2a51a976f10db135ed79971162f5605de944dd6e))
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+- **waa**: Use D8ds_v5 VM size for Azure ML workers
+  ([`71a0fdd`](https://github.com/OpenAdaptAI/openadapt-ml/commit/71a0fddfa4216a49b3a37c1ff1cc2d98d1f605a1))
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+### Documentation
+
+- Add Azure ML log streaming and cost tracking guides
+  ([`59c3a3e`](https://github.com/OpenAdaptAI/openadapt-ml/commit/59c3a3ef852d747599de53fe74f660aea6d5b033))
+
+Document the new CLI commands for: - Live log streaming from Azure ML jobs - Cost tracking for
+  compute instances - Teardown procedures
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+### Features
+
+- **cli**: Add Azure ML log streaming, cost tracking, and teardown
+  ([`59e3cf7`](https://github.com/OpenAdaptAI/openadapt-ml/commit/59e3cf7fbdb53f07866709efd60b05a1aa511ed5))
+
+Add comprehensive Azure ML management commands: - azure-ml-stream: Stream logs from running jobs
+  using Python SDK with account key auth (works around DefaultAzureCredential permission issues) -
+  azure-ml-cost: Track compute instance uptime and estimated costs - azure-ml-teardown: Cancel jobs
+  and delete compute instances
+
+Also improves: - azure-ml-quota: Shows both ML Dedicated quota (what Azure ML actually uses) and
+  regular VM quota - Better error handling and logging throughout
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+- **cli**: Add Azure ML status, VNC, and monitor commands
+  ([`7985cff`](https://github.com/OpenAdaptAI/openadapt-ml/commit/7985cff95cc50d0313b3f3cb8ff5a1a1de039a71))
+
+New commands for end-to-end Azure ML automation: - azure-ml-status: Show jobs and compute instances
+  - azure-ml-vnc: Set up VNC tunnel to compute instance - azure-ml-monitor: Monitor jobs with auto
+  VNC setup
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+- **cli**: Add azure-ml-quota command for quota management
+  ([`eecb3a4`](https://github.com/OpenAdaptAI/openadapt-ml/commit/eecb3a461c2b5f6882755dab52f2cfb0c9a9616a))
+
+Semi-automated quota increase workflow: - Checks current quota for WAA-compatible VM families -
+  Shows which families have sufficient quota - Opens Azure Portal quota page with instructions -
+  Guides user through the request process
+
+Usage: uv run python -m openadapt_ml.benchmarks.cli azure-ml-quota
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+- **cli**: Add multi-VM pool commands for parallel WAA evaluation
+  ([`005664a`](https://github.com/OpenAdaptAI/openadapt-ml/commit/005664ab0ec6c425ac9ebe1af19b23e552b9bf90))
+
+Add pool-create, pool-wait, and pool-run commands for running WAA benchmarks across multiple VMs in
+  parallel:
+
+- pool-create --workers N: Create N VMs with Docker and WAA image - Parallel VM creation using
+  ThreadPoolExecutor - Auto-selects available region and VM size - Configures Docker with /mnt
+  storage - Registers pool for tracking
+
+- pool-wait: Wait for WAA to be ready on all workers - Starts WAA containers on each worker - Polls
+  /probe endpoint until ready - Configurable timeout
+
+- pool-run --tasks N: Distribute tasks across pool - Round-robin task distribution - Parallel
+  execution on all workers - Progress tracking in registry
+
+This enables ~5x faster benchmark completion with 5 workers, or full 154-task evaluation in ~10min
+  with 10+ workers.
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+### Refactoring
+
+- **waa**: Update submodule with SDK v2 migration
+  ([`5080ad6`](https://github.com/OpenAdaptAI/openadapt-ml/commit/5080ad697e88ff297dfbb14f2c0756f53ebfd496))
+
+Updates WindowsAgentArena submodule to include Azure ML SDK v2 migration that enables job submission
+  from macOS ARM64.
+
+Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
+
+
 ## v0.2.2 (2026-01-29)
 
 ### Bug Fixes

{openadapt_ml-0.2.2 → openadapt_ml-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: openadapt-ml
-Version: 0.2.2
+Version: 0.3.1
 Summary: Model-agnostic, domain-agnostic ML engine for GUI automation agents
 Project-URL: Homepage, https://github.com/OpenAdaptAI/openadapt-ml
 Project-URL: Repository, https://github.com/OpenAdaptAI/openadapt-ml
@@ -58,7 +58,7 @@ Description-Content-Type: text/markdown
 
 # OpenAdapt-ML
 
-[![Build Status](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/publish.yml/badge.svg)](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/publish.yml)
+[![Build Status](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/release.yml/badge.svg)](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/release.yml)
 [![PyPI version](https://img.shields.io/pypi/v/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/)
 [![Downloads](https://img.shields.io/pypi/dm/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
@@ -88,6 +88,38 @@ The design is described in detail in [`docs/design.md`](docs/design.md).
 
 ---
 
+## Parallel WAA Benchmark Evaluation (New in v0.3.0)
+
+Run Windows Agent Arena benchmarks across multiple Azure VMs in parallel for faster evaluation:
+
+```bash
+# Create a pool of 5 workers
+uv run python -m openadapt_ml.benchmarks.cli pool-create --workers 5
+
+# Wait for all workers to be ready
+uv run python -m openadapt_ml.benchmarks.cli pool-wait
+
+# Run 154 tasks distributed across workers (~5x faster)
+uv run python -m openadapt_ml.benchmarks.cli pool-run --tasks 154
+```
+
+**Key features:**
+- **Parallel execution**: Distribute 154 WAA tasks across N workers
+- **Automatic task distribution**: Uses WAA's native `--worker_id`/`--num_workers` for round-robin assignment
+- **VNC access**: View each Windows VM via SSH tunnels (`localhost:8006`, `localhost:8007`, etc.)
+- **Cost tracking**: Monitor Azure VM costs in real-time
+
+**Performance:**
+| Workers | Estimated Time (154 tasks) |
+|---------|---------------------------|
+| 1       | ~50-80 hours              |
+| 5       | ~10-16 hours              |
+| 10      | ~5-8 hours                |
+
+See [WAA Benchmark Workflow](#waa-benchmark-workflow) for complete setup instructions.
+
+---
+
 ## 1. Installation
 
 ### 1.1 From PyPI (recommended)
@@ -1029,7 +1061,112 @@ uv run python -m openadapt_ml.benchmarks.cli screenshot --target terminal --no-t
 
 ---
 
-## 14. Limitations & Notes
+<a id="waa-benchmark-workflow"></a>
+
+## 14. WAA Benchmark Workflow
+
+Windows Agent Arena (WAA) is a benchmark of 154 tasks across 11 Windows domains. OpenAdapt-ML provides infrastructure to run WAA evaluations on Azure VMs with parallel execution.
+
+### 14.1 Prerequisites
+
+1. **Azure CLI**: `brew install azure-cli && az login`
+2. **OpenAI API Key**: Set in `.env` file (`OPENAI_API_KEY=sk-...`)
+3. **Azure quota**: Ddsv5 family VMs (8+ vCPUs per worker)
+
+### 14.2 Single VM Workflow
+
+For quick testing or small runs:
+
+```bash
+# Setup VM with WAA
+uv run python -m openadapt_ml.benchmarks.cli vm setup-waa
+
+# Start monitoring dashboard (auto-opens VNC, manages SSH tunnels)
+uv run python -m openadapt_ml.benchmarks.cli vm monitor
+
+# Run benchmark
+uv run python -m openadapt_ml.benchmarks.cli waa --num-tasks 10
+
+# Deallocate when done (stops billing)
+uv run python -m openadapt_ml.benchmarks.cli vm deallocate -y
+```
+
+### 14.3 Parallel Pool Workflow (Recommended)
+
+For full 154-task evaluations, use multiple VMs:
+
+```bash
+# 1. Create pool (provisions N Azure VMs with Docker + WAA)
+uv run python -m openadapt_ml.benchmarks.cli pool-create --workers 5
+
+# 2. Wait for all workers to be ready (Windows boot + WAA server startup)
+uv run python -m openadapt_ml.benchmarks.cli pool-wait
+
+# 3. Run benchmark across all workers
+#    Tasks are distributed using WAA's native --worker_id/--num_workers
+uv run python -m openadapt_ml.benchmarks.cli pool-run --tasks 154
+
+# 4. Monitor progress
+uv run python -m openadapt_ml.benchmarks.cli pool-status
+uv run python -m openadapt_ml.benchmarks.cli pool-logs
+
+# 5. Cleanup (delete all VMs - IMPORTANT to stop billing!)
+uv run python -m openadapt_ml.benchmarks.cli pool-cleanup
+```
+
+### 14.4 VNC Access to Workers
+
+View what each Windows VM is doing:
+
+```bash
+# Get worker IPs
+uv run python -m openadapt_ml.benchmarks.cli pool-status
+
+# Set up SSH tunnels (tunnels are created automatically, but you can also do this manually)
+ssh -f -N -L 8006:localhost:8006 azureuser@<worker-0-ip>  # localhost:8006
+ssh -f -N -L 8007:localhost:8006 azureuser@<worker-1-ip>  # localhost:8007
+# etc.
+
+# Open in browser
+open http://localhost:8006  # Worker 0
+open http://localhost:8007  # Worker 1
+```
+
+### 14.5 Architecture
+
+```
+Local Machine
+├── openadapt-ml CLI (pool-create, pool-wait, pool-run)
+│   └── SSH tunnels to each worker
+│
+Azure (N VMs, Standard_D8ds_v5)
+├── waa-pool-00
+│   └── Docker
+│       └── windowsarena/winarena:latest
+│           └── QEMU (Windows 11)
+│               ├── WAA Flask server (port 5000)
+│               └── Navi agent (GPT-4o-mini)
+├── waa-pool-01
+│   └── ...
+└── waa-pool-N
+    └── ...
+```
+
+### 14.6 Cost Estimates
+
+| VM Size | vCPUs | RAM | Cost/hr | 5 VMs for 10hrs |
+|---------|-------|-----|---------|-----------------|
+| Standard_D8ds_v5 | 8 | 32GB | ~$0.38 | ~$19 |
+
+**Tips:**
+- Always run `pool-cleanup` when done to delete VMs and stop billing
+- Use `vm deallocate` (not delete) to pause billing but keep disk
+- Set `--auto-shutdown-hours 2` on `vm monitor` for safety
+- Prices vary by Azure region
+
+---
+
+## 15. Limitations & Notes
 
 - **Apple Silicon / bitsandbytes**:
   - Example configs are sized for CPU / Apple Silicon development runs; see
@@ -1053,7 +1190,7 @@ For deeper architectural details, see [`docs/design.md`](docs/design.md).
 
 ---
 
-## 15. Roadmap
+## 16. Roadmap
 
 For the up-to-date, prioritized roadmap (including concrete implementation
 targets and agent-executable acceptance criteria), see

{openadapt_ml-0.2.2 → openadapt_ml-0.3.1}/README.md

@@ -1,6 +1,6 @@
 # OpenAdapt-ML
 
-[![Build Status](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/publish.yml/badge.svg)](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/publish.yml)
+[![Build Status](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/release.yml/badge.svg)](https://github.com/OpenAdaptAI/openadapt-ml/actions/workflows/release.yml)
 [![PyPI version](https://img.shields.io/pypi/v/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/)
 [![Downloads](https://img.shields.io/pypi/dm/openadapt-ml.svg)](https://pypi.org/project/openadapt-ml/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
@@ -30,6 +30,38 @@ The design is described in detail in [`docs/design.md`](docs/design.md).
 
 ---
 
+## Parallel WAA Benchmark Evaluation (New in v0.3.0)
+
+Run Windows Agent Arena benchmarks across multiple Azure VMs in parallel for faster evaluation:
+
+```bash
+# Create a pool of 5 workers
+uv run python -m openadapt_ml.benchmarks.cli pool-create --workers 5
+
+# Wait for all workers to be ready
+uv run python -m openadapt_ml.benchmarks.cli pool-wait
+
+# Run 154 tasks distributed across workers (~5x faster)
+uv run python -m openadapt_ml.benchmarks.cli pool-run --tasks 154
+```
+
+**Key features:**
+- **Parallel execution**: Distribute 154 WAA tasks across N workers
+- **Automatic task distribution**: Uses WAA's native `--worker_id`/`--num_workers` for round-robin assignment
+- **VNC access**: View each Windows VM via SSH tunnels (`localhost:8006`, `localhost:8007`, etc.)
+- **Cost tracking**: Monitor Azure VM costs in real-time
+
+**Performance:**
+| Workers | Estimated Time (154 tasks) |
+|---------|---------------------------|
+| 1       | ~50-80 hours              |
+| 5       | ~10-16 hours              |
+| 10      | ~5-8 hours                |
+
+See [WAA Benchmark Workflow](#waa-benchmark-workflow) for complete setup instructions.
+
+---
+
 ## 1. Installation
 
 ### 1.1 From PyPI (recommended)
@@ -971,7 +1003,112 @@ uv run python -m openadapt_ml.benchmarks.cli screenshot --target terminal --no-t
 
 ---
 
-## 14. Limitations & Notes
+<a id="waa-benchmark-workflow"></a>
+
+## 14. WAA Benchmark Workflow
+
+Windows Agent Arena (WAA) is a benchmark of 154 tasks across 11 Windows domains. OpenAdapt-ML provides infrastructure to run WAA evaluations on Azure VMs with parallel execution.
+
+### 14.1 Prerequisites
+
+1. **Azure CLI**: `brew install azure-cli && az login`
+2. **OpenAI API Key**: Set in `.env` file (`OPENAI_API_KEY=sk-...`)
+3. **Azure quota**: Ddsv5 family VMs (8+ vCPUs per worker)
+
+### 14.2 Single VM Workflow
+
+For quick testing or small runs:
+
+```bash
+# Setup VM with WAA
+uv run python -m openadapt_ml.benchmarks.cli vm setup-waa
+
+# Start monitoring dashboard (auto-opens VNC, manages SSH tunnels)
+uv run python -m openadapt_ml.benchmarks.cli vm monitor
+
+# Run benchmark
+uv run python -m openadapt_ml.benchmarks.cli waa --num-tasks 10
+
+# Deallocate when done (stops billing)
+uv run python -m openadapt_ml.benchmarks.cli vm deallocate -y
+```
+
+### 14.3 Parallel Pool Workflow (Recommended)
+
+For full 154-task evaluations, use multiple VMs:
+
+```bash
+# 1. Create pool (provisions N Azure VMs with Docker + WAA)
+uv run python -m openadapt_ml.benchmarks.cli pool-create --workers 5
+
+# 2. Wait for all workers to be ready (Windows boot + WAA server startup)
+uv run python -m openadapt_ml.benchmarks.cli pool-wait
+
+# 3. Run benchmark across all workers
+#    Tasks are distributed using WAA's native --worker_id/--num_workers
+uv run python -m openadapt_ml.benchmarks.cli pool-run --tasks 154
+
+# 4. Monitor progress
+uv run python -m openadapt_ml.benchmarks.cli pool-status
+uv run python -m openadapt_ml.benchmarks.cli pool-logs
+
+# 5. Cleanup (delete all VMs - IMPORTANT to stop billing!)
+uv run python -m openadapt_ml.benchmarks.cli pool-cleanup
+```
+
+### 14.4 VNC Access to Workers
+
+View what each Windows VM is doing:
+
+```bash
+# Get worker IPs
+uv run python -m openadapt_ml.benchmarks.cli pool-status
+
+# Set up SSH tunnels (tunnels are created automatically, but you can also do this manually)
+ssh -f -N -L 8006:localhost:8006 azureuser@<worker-0-ip>  # localhost:8006
+ssh -f -N -L 8007:localhost:8006 azureuser@<worker-1-ip>  # localhost:8007
+# etc.
+
+# Open in browser
+open http://localhost:8006  # Worker 0
+open http://localhost:8007  # Worker 1
+```
+
+### 14.5 Architecture
+
+```
+Local Machine
+├── openadapt-ml CLI (pool-create, pool-wait, pool-run)
+│   └── SSH tunnels to each worker
+│
+Azure (N VMs, Standard_D8ds_v5)
+├── waa-pool-00
+│   └── Docker
+│       └── windowsarena/winarena:latest
+│           └── QEMU (Windows 11)
+│               ├── WAA Flask server (port 5000)
+│               └── Navi agent (GPT-4o-mini)
+├── waa-pool-01
+│   └── ...
+└── waa-pool-N
+    └── ...
+```
+
+### 14.6 Cost Estimates
+
+| VM Size | vCPUs | RAM | Cost/hr | 5 VMs for 10hrs |
+|---------|-------|-----|---------|-----------------|
+| Standard_D8ds_v5 | 8 | 32GB | ~$0.38 | ~$19 |
+
+**Tips:**
+- Always run `pool-cleanup` when done to delete VMs and stop billing
+- Use `vm deallocate` (not delete) to pause billing but keep disk
+- Set `--auto-shutdown-hours 2` on `vm monitor` for safety
+- Prices vary by Azure region
+
+---
+
+## 15. Limitations & Notes
 
 - **Apple Silicon / bitsandbytes**:
   - Example configs are sized for CPU / Apple Silicon development runs; see
@@ -995,7 +1132,7 @@ For deeper architectural details, see [`docs/design.md`](docs/design.md).
 
 ---
 
-## 15. Roadmap
+## 16. Roadmap
 
 For the up-to-date, prioritized roadmap (including concrete implementation
 targets and agent-executable acceptance criteria), see