@hybridlabor-api/aos 4.2.0 → 4.4.0

package/mcps/windows-computer-use-mcp/docs/ocr.md

@@ -1,107 +0,0 @@
-# OCR (Optical Character Recognition)
-
-Windows Computer Use MCP uses [Tesseract OCR](https://github.com/tesseract-ocr/tesseract) via the `pytesseract` Python wrapper for extracting text from screenshots and window captures.
-
-## How it's used
-
-| Tool | Function | OCR Use |
-|------|----------|---------|
-| `automation_visual` | `extract_text` | OCR a window or screen region |
-| `automation_assert` | `assert_text` | Verify text appears in a window |
-| `get_window_state` | `use_ocr=True` | Enrich UIA element tree with OCR text |
-| `get_desktop_state` | `use_ocr=True` | OCR-enrich full desktop tree |
-| `api/v1/ocr/*` | REST endpoints | OCR via HTTP API |
-| Adaptive cascade | OCR fallback | Find elements when UIA selectors fail |
-
-## About Tesseract
-
-[Tesseract](https://github.com/tesseract-ocr/tesseract) is the most widely used open-source OCR engine. Originally developed by HP in the 1980s, it was open-sourced in 2005 and later maintained by Google. Version 4.0 (2018) introduced an LSTM-based neural network classifier, and version 5.x (2021+) continues improvements.
-
-- **GitHub:** https://github.com/tesseract-ocr/tesseract
-- **Current version:** 5.5.0 (as of 2026)
-- **License:** Apache 2.0
-- **Language:** C++
-- **Training data:** `tessdata` — language-specific `.traineddata` files with character models and dictionaries
-
-## Known limitations
-
-Discovered during testing of this repo:
-
-| Issue | Detail |
-|-------|--------|
-| **ASCII art** | Widely separated symbol groups (e.g. `(__)` then spaces then `(__)`) confuse page layout analysis. The characters themselves (`(`, `)`, `\`, `/`) are read correctly in normal text flow. |
-| **Minimum font size** | Characters smaller than ~10pt (x-height < 20px) degrade recognition. Notepad default ~9pt is marginal. Enlarge with `Ctrl+Shift+>` or use larger UI text. |
-| **Consolas `/` and `\`** | These characters render similarly to letters in monospace fonts. `/` is often read as `[` or `1`, `\` as `V` or `]`. |
-| **Keyboard shortcut `Ctrl`** | Frequently read as `Ctr1` (digit 1 replaces lowercase l). This is standard for OCR — most commercial engines have the same issue with small type. |
-| **`$` sign** | Sometimes read as `@` or `S` depending on font. |
-
-## Setup
-
-### Automatic (recommended)
-
-```powershell
-just install-tesseract        # interactive dialog
-.\scripts\install-tesseract.ps1  # silent
-```
-
-The NSIS installer also offers Tesseract as an optional checkbox during setup.
-
-### Manual
-
-Download from [UB-Mannheim/tesseract](https://github.com/UB-Mannheim/tesseract/releases) and run the installer. Ensure `C:\Program Files\Tesseract-OCR` is in your PATH.
-
-Tesseract 5.x requires the `eng` traineddata file, which is included in the installer (~5 MB). Additional languages can be downloaded from [tesseract-ocr/tessdata](https://github.com/tesseract-ocr/tessdata).
-
-## Verification
-
-```powershell
-tesseract --version
-tesseract --list-langs
-```
-
-Or from Python:
-
-```python
-import pytesseract
-print(pytesseract.get_tesseract_version())
-```
-
-## Testing
-
-OCR screenshots and results are saved to `ocr_scans/ocr_result_<timestamp>.txt` with corresponding PNG files. The autonomous demo (`just demo-autonomous`) runs a full pipeline: type text → screenshot → OCR → display → save.
-
-## Competition
-
-| Engine | License | Quality | Speed | Notes |
-|--------|---------|---------|-------|-------|
-| **Tesseract 5.x** | Apache 2.0 | Good | Fast | Default, free, self-hosted |
-| **Windows Media OCR** | Proprietary | Good | Fast | Built into Windows, COM API, no install |
-| **Azure AI Vision** | Commercial | Excellent | Fast | Cloud-based, pay-per-call |
-| **Google Cloud Vision** | Commercial | Excellent | Fast | Cloud-based, pay-per-call |
-| **AWS Textract** | Commercial | Excellent | Fast | Cloud-based, pay-per-page |
-| **EasyOCR** | Apache 2.0 | Good | Moderate | Python, supports 80+ languages |
-| **PaddleOCR** | Apache 2.0 | Good | Fast | Chinese-focused but multi-language |
-| **Surya OCR** | MIT | Very Good | Moderate | Modern architecture, good for dense text |
-
-### Windows Media OCR
-
-Windows 10/11 includes a built-in OCR engine via `Windows.Media.Ocr` — the same engine used by Snipping Tool and Game Bar. Available on every Windows 10/11 system with zero additional install.
-
-**Integrated via `winrt-*` PyPI packages:** `winrt-windows-media-ocr`, `winrt-windows-storage`, `winrt-windows-graphics-imaging`, `winrt-windows-foundation`
-
-Use by passing `ocr_provider="windowsmedia"` to `extract_text`:
-
-```python
-automation_visual(extract_text, window_handle=hwnd, ocr_provider="windowsmedia")
-```
-
-Or set the env var `WINDOWS_COMPUTER_USE_MCP_OCR_PROVIDER=windowsmedia` for all OCR operations.
-
-**Comparison with Tesseract:** Both engines produce nearly identical results on prose text. Windows Media OCR is slightly faster for UI text and requires no language data files. For ASCII art and unusual symbol arrangements, both engines produce similar garbage — it's a fundamental OCR limitation, not engine-specific.
-
-## Future
-
-- ~~Windows Media OCR integration~~ (blocked on WinRT async interop)
-- Preprocessing pipeline: adaptive binarization, deskew, resolution upscaling
-- OCR confidence scoring in `extract_text` results
-- Configurable per-application OCR presets (e.g. terminal fonts vs prose)

package/mcps/windows-computer-use-mcp/docs/py-stack.md

@@ -1,47 +0,0 @@
-# Python Stack
-
-Every `py-` module in this project and what it does.
-
-## Core UI Automation
-
-| Package | What it does here |
-|---------|-------------------|
-| **[pywinauto](https://github.com/pywinauto/pywinauto)** | Walks UIA and Win32 control trees, finds elements by automation ID / title / class, clicks, reads text, checks state. Backs `automation_windows`, `automation_elements`. |
-| **pywin32** | Win32 COM and low-level API bindings. Used for session management, process enumeration, DPI helpers, and raw `SetCursorPos`/`mouse_event` alongside pywinauto. |
-| **pygetwindow** | Lightweight window enumeration by title — fallback discovery before pywinauto attaches to the HWND. |
-| **pyautogui** | Screen-level pointer and screenshot fallback. Used mainly for keyboard injection paths where pywinauto's `type_keys` has focus issues. |
-
-## Input Simulation
-
-| Package | What it does here |
-|---------|-------------------|
-| **pynput** | Cross-platform keyboard/mouse listener and controller. Powers `automation_keyboard` — normal typing, modifier combos, and the opt-in `global_keylogger` for debugging shortcut focus issues. |
-| **pyperclip** | Clipboard read/write for `automation_system`. Used to paste long paths in file dialogs (`automation_dialog`) and for copy/paste automation flows. |
-
-## Vision and OCR
-
-| Package | What it does here |
-|---------|-------------------|
-| **opencv-python-headless** | Screenshot capture, template matching, camera index probe. Backs `automation_visual` (`find_image`, `screenshot`) and the `automation_face` biometric preview pipeline. |
-| **pytesseract** | Tesseract OCR wrapper for `automation_visual` `extract_text`. Requires Tesseract 5.x installed on the host. Auto-install via `just install-tesseract` or NSIS installer checkbox. **Known limitation:** reads prose, punctuation, and all symbol characters correctly in normal text flow. ASCII art with widely separated character groups may produce segmentation errors (isolated groups confuse page layout analysis). For best results, ensure text is at least 10pt (default Notepad size may need enlargement via Ctrl+Shift+>). OCR results and screenshots are saved to `ocr_scans/` for review. |
-
-## Optional: Face Recognition
-
-| Package | What it does here |
-|---------|-------------------|
-| **face_recognition** | Webcam capture and face matching for `automation_face` (opt-in, `WINDOWS_COMPUTER_USE_MCP_ENABLE_FACE=1`). Not installed by default. |
-| **dlib** | ML backend for face_recognition. Heavy dependency — requires CMake to build. |
-
-## Infrastructure Dependencies
-
-These aren't `py-*` packages but are critical to the stack:
-
-| Package | Role |
-|---------|------|
-| **FastMCP** | MCP 3.2 framework — tool registration, transport (stdio + HTTP), Prefab cards, sampling |
-| **FastAPI + uvicorn** | HTTP REST layer — `/api/v1/*`, `/mcp` streamable HTTP, health checks |
-| **Pillow** | Image format handling — screenshot encoding (PNG/JPEG), thumbnail generation |
-| **numpy** | Array operations under opencv, face_recognition, and image diff/hash functions in `automation_assert` |
-| **httpx** | Async HTTP client — bridge calls to fleet MCP servers and LLM proxies |
-| **psutil** | Process listing, system resource monitoring in `automation_system` |
-| **Pydantic** | Request/response models for all portmanteau tools |

package/mcps/windows-computer-use-mcp/tests/README.md

@@ -1,155 +0,0 @@
-# Windows Computer Use Test Suite
-
-Comprehensive test harness for Windows Computer Use server.
-
-## E2E (LibreOffice Calc — Cua parity)
-
-Requires LibreOffice `soffice` (same install as **libreoffice-mcp**). Default pytest **excludes** e2e (`-m "not e2e"`).
-
-```powershell
-uv run pytest -m e2e -v --no-cov
-```
-
-Optional: `$env:PYWINAUTO_E2E_SOFFICE = "C:\Program Files\LibreOffice\program\soffice.exe"`
-
-Tests live under `tests/e2e/` and set `WINDOWS_COMPUTER_USE_MCP_LOOSE_UIA=1` for LibreOffice UIA nodes.
-
-## Test Structure
-
-```
-tests/
-├── conftest.py              # Shared fixtures and configuration
-├── test_basic_tools.py      # Tests for basic tools (health_check, get_help)
-├── test_window_tools.py     # Tests for window management tools
-├── test_mouse_tools.py      # Tests for mouse control tools
-├── test_system_tools.py     # Tests for system-level tools
-├── test_app_integration.py  # Integration tests for app initialization
-├── test_face_recognition.py # Tests for face recognition (existing)
-└── test_images/             # Test images for face recognition
-```
-
-## Running Tests
-
-### Run all tests
-```bash
-pytest
-```
-
-### Run with coverage
-```bash
-pytest --cov=src/windows_computer_use_mcp --cov-report=html
-```
-
-### Run specific test file
-```bash
-pytest tests/test_basic_tools.py
-```
-
-### Run specific test
-```bash
-pytest tests/test_basic_tools.py::TestHealthCheck::test_health_check_returns_healthy_status
-```
-
-### Run with markers
-```bash
-# Run only unit tests
-pytest -m unit
-
-# Skip slow tests
-pytest -m "not slow"
-
-# Run only Windows-specific tests
-pytest -m windows_only
-```
-
-## Test Categories
-
-### Unit Tests
-- Test individual tool functions in isolation
-- Use mocks for Windows API calls
-- Fast execution
-- No external dependencies
-
-### Integration Tests
-- Test tool interactions
-- Test app initialization
-- Test module imports
-- May require some system access
-
-### Windows-Specific Tests
-- Tests that require Windows OS
-- May need actual window access
-- Marked with `@pytest.mark.windows_only`
-
-## Fixtures
-
-Common fixtures available in `conftest.py`:
-
-- `app_instance`: FastMCP app instance
-- `mock_window`: Mock window object
-- `mock_element`: Mock UI element
-- `mock_pyautogui`: Mocked pyautogui
-- `mock_pywinauto`: Mocked pywinauto
-- `mock_pygetwindow`: Mocked pygetwindow
-- `temp_test_dir`: Temporary directory for test files
-- `sample_image_path`: Sample image file for testing
-
-## Writing New Tests
-
-1. **Use fixtures** from `conftest.py` when possible
-2. **Mock Windows APIs** - don't rely on actual windows being open
-3. **Test error cases** - ensure proper error handling
-4. **Use descriptive names** - `test_function_name_scenario`
-5. **Group related tests** in classes
-6. **Add markers** for test categorization
-
-### Example Test
-
-```python
-class TestMyTool:
-    """Tests for my_tool."""
-    
-    @patch('windows_computer_use_mcp.tools.my_module.some_dependency')
-    def test_my_tool_success(self, mock_dep, app_instance):
-        """Test my_tool with successful execution."""
-        from windows_computer_use_mcp.tools.my_module import my_tool
-        
-        mock_dep.return_value = "expected_result"
-        
-        result = my_tool(param="value")
-        
-        assert isinstance(result, dict)
-        assert "success" in result
-```
-
-## Coverage Goals
-
-- **Minimum**: 30% (enforced by pytest.ini)
-- **Target**: 70%+
-- **Current**: Check with `pytest --cov-report=term-missing`
-
-## Continuous Integration
-
-Tests run automatically on:
-- Push to main/master/develop branches
-- Pull requests
-- Multiple Python versions (3.10, 3.11, 3.12)
-- Windows environment
-
-## Troubleshooting
-
-### Import Errors
-- Ensure `src/` is in Python path
-- Check that all dependencies are installed
-- Verify `conftest.py` is in tests directory
-
-### Mock Issues
-- Ensure mocks are patched before imports
-- Check patch paths match actual import paths
-- Verify mock return values match expected types
-
-### Windows-Specific Failures
-- Some tests require Windows OS
-- Use `@pytest.mark.windows_only` marker
-- Skip on non-Windows: `pytest -m "not windows_only"`
-

package/mcps/windows-computer-use-mcp/tests/__init__.py

@@ -1 +0,0 @@
-# Tests package

package/mcps/windows-computer-use-mcp/tests/conftest.py

@@ -1,281 +0,0 @@
-"""Pytest configuration and shared fixtures for PyWinAuto MCP tests."""
-
-import sys
-from pathlib import Path
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-# Add src to path
-sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
-
-# Environment-aware CI vs local (mcp-central-docs: standards/testing-environment-aware.md)
-
-
-@pytest.fixture(scope="session")
-def app_instance():
-    """Get the FastMCP app instance."""
-    from windows_computer_use_mcp.app import app
-
-    return app
-
-
-@pytest.fixture
-def mock_window():
-    """Create a mock window object."""
-    window = MagicMock()
-    window.title = "Test Window"
-    window.handle = 12345
-    window.isVisible = True
-    window.isEnabled = True
-    window.rectangle = MagicMock()
-    window.rectangle.left = 100
-    window.rectangle.top = 100
-    window.rectangle.right = 800
-    window.rectangle.bottom = 600
-    window.rectangle.width = 700
-    window.rectangle.height = 500
-    window.class_name = "TestWindowClass"
-    window.process_id = 1234
-    window.process_name = "test.exe"
-    return window
-
-
-@pytest.fixture
-def mock_element():
-    """Create a mock UI element."""
-    element = MagicMock()
-    element.automation_id = "btnOK"
-    element.name = "OK Button"
-    element.control_type = "Button"
-    element.is_visible = True
-    element.is_enabled = True
-    element.rectangle = MagicMock()
-    element.rectangle.left = 200
-    element.rectangle.top = 300
-    element.rectangle.right = 300
-    element.rectangle.bottom = 350
-    element.rectangle.width = 100
-    element.rectangle.height = 50
-    element.text = "OK"
-    return element
-
-
-@pytest.fixture
-def mock_desktop():
-    """Create a mock Desktop object."""
-    desktop = MagicMock()
-    return desktop
-
-
-@pytest.fixture
-def mock_application():
-    """Create a mock Application object."""
-    app = MagicMock()
-    app.window.return_value = MagicMock()
-    return app
-
-
-@pytest.fixture
-def mock_pyautogui():
-    """Mock pyautogui for testing across portmanteau modules."""
-    import sys
-
-    mock_instance = MagicMock()
-    # Patch pyautogui at the sys.modules level so local imports inside functions resolve
-    old_pyautogui = sys.modules.get("pyautogui")
-    sys.modules["pyautogui"] = mock_instance
-
-    # Configure common mock behavior
-    mock_instance.position.return_value = (500, 500)
-    mock_instance.size.return_value = (1920, 1080)
-    mock_instance.click.return_value = None
-    mock_instance.moveTo.return_value = None
-    mock_instance.scroll.return_value = None
-    mock_instance.write.return_value = None
-    mock_instance.typewrite.return_value = None
-    mock_instance.press.return_value = None
-    mock_instance.hotkey.return_value = None
-    mock_instance.doubleClick.return_value = None
-    mock_instance.rightClick.return_value = None
-    mock_instance.dragTo.return_value = None
-
-    yield mock_instance
-
-    # Restore original pyautogui
-    if old_pyautogui is not None:
-        sys.modules["pyautogui"] = old_pyautogui
-    else:
-        sys.modules.pop("pyautogui", None)
-
-
-@pytest.fixture
-def mock_desktop_uia():
-    """Mock pywinauto.Desktop for UIA-based element discovery."""
-    # Patch all locations where Desktop is instantiated
-    with (
-        patch("windows_computer_use_mcp.tools.portmanteau_elements.Desktop") as mock_elem_desktop,
-        patch("windows_computer_use_mcp.tools.portmanteau_windows.Desktop") as mock_win_desktop,
-        patch("windows_computer_use_mcp.tools.portmanteau_visual.Desktop") as mock_vis_desktop,
-        patch("windows_computer_use_mcp.tools.utils.Desktop") as mock_utils_desktop,
-        patch("windows_computer_use_mcp.desktop_state.walker.Desktop") as mock_walker_desktop,
-    ):
-        mock_instance = MagicMock()
-        mock_elem_desktop.return_value = mock_instance
-        mock_win_desktop.return_value = mock_instance
-        mock_vis_desktop.return_value = mock_instance
-        mock_utils_desktop.return_value = mock_instance
-        mock_walker_desktop.return_value = mock_instance
-
-        yield mock_instance
-
-
-@pytest.fixture
-def mock_pywinauto():
-    """Mock pywinauto for testing."""
-    # Patch in the new portmanteau locations
-    patches = [
-        patch("windows_computer_use_mcp.tools.portmanteau_windows.Application"),
-        patch("windows_computer_use_mcp.tools.portmanteau_windows.findwindows"),
-    ]
-
-    mock_objs = [p.start() for p in patches]
-    mock_app, mock_find = mock_objs
-
-    mock_window = MagicMock()
-    mock_window.wrapper_object.return_value = MagicMock()
-    mock_app.return_value = mock_window
-
-    yield {"Application": mock_app, "findwindows": mock_find}
-
-    for p in patches:
-        p.stop()
-
-
-@pytest.fixture
-def mock_pygetwindow():
-    """Mock pygetwindow for testing."""
-    with patch("windows_computer_use_mcp.tools.portmanteau_system.gw") as mock:
-        mock_window = MagicMock()
-        mock_window.title = "Test Window"
-        mock_window._hWnd = 12345
-        mock_window.isActive = True
-        mock.getWindowsWithTitle.return_value = [mock_window]
-        mock.getAllWindows.return_value = [mock_window]
-        yield mock
-
-
-@pytest.fixture
-def mock_psutil():
-    """Mock psutil for testing."""
-    with patch("windows_computer_use_mcp.tools.portmanteau_system.psutil") as mock:
-        mock_process = MagicMock()
-        mock_process.name.return_value = "test.exe"
-        mock_process.pid = 1234
-        mock.process_iter.return_value = [mock_process]
-        yield mock
-
-
-@pytest.fixture
-def temp_test_dir(tmp_path):
-    """Create a temporary directory for test files."""
-    return tmp_path
-
-
-@pytest.fixture(autouse=True)
-def mock_approval():
-    """Mock approval_state.is_approved to always return True for tests."""
-    with patch("windows_computer_use_mcp.app.approval_state.is_approved") as mock:
-        mock.return_value = True
-        yield mock
-
-
-@pytest.fixture
-def sample_image_path(tmp_path):
-    """Create a sample image file for testing."""
-    from PIL import Image
-
-    img = Image.new("RGB", (100, 100), color="red")
-    img_path = tmp_path / "test_image.png"
-    img.save(img_path)
-    return str(img_path)
-
-
-@pytest.fixture
-def mock_ocr():
-    """Mock OCR functionality."""
-    with patch("windows_computer_use_mcp.tools.portmanteau_visual.pytesseract") as mock:
-        mock.image_to_string.return_value = "Sample OCR Text"
-        mock.image_to_data.return_value = []
-        yield mock
-
-
-@pytest.fixture
-def mock_face_recognition():
-    """Mock face recognition functionality."""
-    with patch("windows_computer_use_mcp.tools.portmanteau_face.face_recognition") as mock:
-        mock.face_locations.return_value = [(100, 200, 300, 400)]
-        mock.face_encodings.return_value = [[0.1] * 128]
-        mock.compare_faces.return_value = [True]
-        mock.face_distance.return_value = [0.3]
-        yield mock
-
-
-@pytest.fixture
-def mock_cv2():
-    """Mock OpenCV for testing."""
-    with patch("windows_computer_use_mcp.tools.portmanteau_visual.cv2") as mock:
-        mock.imread.return_value = None
-        mock.imwrite.return_value = True
-        mock.VideoCapture.return_value = MagicMock()
-        yield mock
-
-
-@pytest.fixture
-def mock_pil():
-    """Mock PIL/Pillow for testing."""
-    with (
-        patch("windows_computer_use_mcp.tools.portmanteau_visual.Image") as mock_image,
-        patch("windows_computer_use_mcp.tools.portmanteau_visual.ImageGrab") as mock_grab,
-    ):
-        mock_img = MagicMock()
-        mock_img.size = (1920, 1080)
-        mock_img.save.return_value = None
-        mock_image.open.return_value = mock_img
-        mock_image.new.return_value = mock_img
-        mock_grab.grab.return_value = mock_img
-
-
-@pytest.fixture
-def verify_result():
-    """Helper fixture to verify ToolResult objects."""
-
-    def _verify(result, expected_status="success", expected_keys=None, message_contains=None):
-        """
-        Verify ToolResult contents.
-
-        Args:
-            result: The ToolResult (or dict) to verify.
-            expected_status: "success" or "error".
-            expected_keys: List of keys that must be present in result.data.
-            message_contains: Substring that must be in the message.
-        """
-        # Convert Pydantic model to dict if necessary
-        res = result if isinstance(result, dict) else result.model_dump()
-
-        assert res["status"] == expected_status, f"Expected {expected_status}, got {res['status']}"
-
-        if message_contains:
-            assert message_contains.lower() in res["message"].lower()
-
-        if expected_status == "success":
-            assert "data" in res
-            if expected_keys and res["data"]:
-                for key in expected_keys:
-                    assert key in res["data"], f"Missing expected key '{key}' in result.data"
-        elif expected_status == "error":
-            assert res.get("recovery_tip") is not None, "Error result must include a recovery_tip"
-
-        return res
-
-    return _verify