code-puppy-core-plugins 0.0.76__tar.gz → 0.0.78__tar.gz

{code_puppy_core_plugins-0.0.76 → code_puppy_core_plugins-0.0.78}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: code-puppy-core-plugins
-Version: 0.0.76
+Version: 0.0.78
 Summary: Official plugin bundle for Code Puppy
 Project-URL: Repository, https://github.com/mpfaffenberger/code_puppy_core_plugins
 Project-URL: Homepage, https://github.com/mpfaffenberger/code_puppy_core_plugins
@@ -34,6 +34,13 @@ Requires-Dist: tree-sitter-scala>=0.26.2
 Requires-Dist: tree-sitter-swift>=0.7.3
 Requires-Dist: tree-sitter-typescript>=0.23.2
 Requires-Dist: tree-sitter<0.26,>=0.25.2
+Provides-Extra: computer-use
+Requires-Dist: pillow>=11; extra == 'computer-use'
+Requires-Dist: pyobjc-framework-applicationservices>=10.3; (sys_platform == 'darwin') and extra == 'computer-use'
+Requires-Dist: pyobjc-framework-cocoa>=10.3; (sys_platform == 'darwin') and extra == 'computer-use'
+Requires-Dist: pyobjc-framework-quartz>=10.3; (sys_platform == 'darwin') and extra == 'computer-use'
+Requires-Dist: pywinauto<0.7,>=0.6.9; (sys_platform == 'win32') and extra == 'computer-use'
+Requires-Dist: windows-capture<3,>=2.0.1; (sys_platform == 'win32') and extra == 'computer-use'
 Description-Content-Type: text/markdown
 
 # Code Puppy Core Plugins

code_puppy_core_plugins-0.0.78/code_puppy_core_plugins/computer_use/FOCUS_INVESTIGATION.md

@@ -0,0 +1,65 @@
+# Foreground activation investigation (2026-10-04)
+
+## Report and reproduction
+
+The interactive acceptance report for initial plugin commit `2bf27fc` recorded
+successful early actions, then repeated "Target lost focus" errors while fresh
+state was being requested. Background captures still worked. The recorded primary
+fixture HWND no longer existed when this investigation began, so it was not reused.
+
+A new diagnostic, `tests/live_windows_focus.py`, created two owned native fixtures
+and exercised activation on the same COM worker used by the backend. Repeated runs
+reproduced the failure: activating the first window succeeded; activating the
+second with Win32 returned false and left the first window foreground.
+
+Observed in the final instrumented run (these HWNDs are historical, not reusable):
+
+| Operation | Observation |
+|---|---|
+| Raw Win32 request for HWND 108005828 | returned false; HWND 6097114 remained foreground |
+| Patched backend requests state for HWND 108005828 | actual UIA fallback callback recorded `[108005828]` |
+| Independent foreground verification | HWND 108005828 was foreground |
+| State generation | valid new revision, UIA nodes, WGC PNG |
+| Additional alternating state requests | all six succeeded |
+
+This proves the original activation path can fail and the tested UIA recovery
+works. It does not establish which internal Windows eligibility condition caused
+every failure in the original interactive session.
+
+## Fix and safety
+
+- Try the ordinary Win32 request and allow a bounded foreground-confirmation wait.
+- If still not foreground, request the target root's advertised UIA SetFocus.
+  Call the COM method, not pywinauto's wrapper that may inject input.
+- Independently verify the actual foreground HWND. A successful COM return alone
+  is insufficient. If activation cannot be confirmed, refuse the operation.
+- Recheck consent, pause, cancellation, desktop availability, process identity,
+  and policy throughout the activation attempt. Existing pre-input checks remain.
+- Report activation refusal separately from losing focus during an action, with
+  useful HWND and request-result diagnostics and a manual recovery instruction.
+- No synthetic Alt keys, input-queue attachment, changed system settings, elevation,
+  or relaxed foreground checks were added. Providers without root focus support
+  can still require manual focus.
+
+## Validation
+
+- 204 targeted computer-use tests passed, up from the original 187.
+- New cases cover Win32 denial, UIA recovery, misleading success responses,
+  unsupported focus, cancellation, policy revocation, and process-identity changes.
+- Real two-window diagnosis confirmed the fallback actually ran and recovered.
+- The native harness passed on primary `(180,180)`, secondary `(2050,180)`, and
+  portrait `(-1300,180)` placements after strengthening its assertions:
+  - fractional scroll changes native SCROLLINFO position;
+  - pixel drag leaves a nonempty native EDIT selection;
+  - the complete occluded capture matches an unfocused, unobscured reference PNG.
+- The harness still is not an LLM-driven acceptance session. Remaining entries in
+  the user's CHECKLIST.md must be resumed after restarting the development instance.
+  Live macOS validation remains outstanding.
+
+The focus diagnostic requires existing consent and never enables it. It prints
+exact PNG paths as evidence, closes only its own verified fixture HWNDs, and does
+not delete unrelated screenshots. Run from the plugin checkout:
+
+```powershell
+.venv\Scripts\python.exe -X utf8 tests\live_windows_focus.py
+```

code_puppy_core_plugins-0.0.78/code_puppy_core_plugins/computer_use/PARITY.md

@@ -0,0 +1,80 @@
+# Cross-platform implementation and verification
+
+## Architecture
+
+`backend.py` selects macOS or Windows behind the existing twelve tool names.
+`macos_backend.py` retains the original macOS implementation. `windows_backend.py`
+implements the same dispatcher methods with UIA, WGC, and typed Win32 input.
+Commands, policy, revision storage, terminal previews, tool-result images, batch
+execution, and UI-settling logic are shared. Windows UIA objects and entire requests
+stay on a single COM worker, never arbitrary event-loop/thread-pool threads.
+
+The plugin keeps its existing package/entry-point name `computer_use`. Neither
+platform needs a `windows_desktop` plugin or alternate tool schemas. The main
+Code Puppy repository change adds Windows dependencies to its existing extra;
+the companion core-plugins repository contains the implementation.
+
+## Capability matrix
+
+| Existing capability | Windows implementation | Verification |
+|---|---|---|
+| Twelve tools / opt-in registration | Same registrars and Pydantic-AI schemas | Automated, both platform gates |
+| Revisioned application state | HWND identity, UIA tree, WGC screenshot | Live native fixture |
+| Accessibility node IDs | Bounded UIA control-view traversal | Live + bounds/truncation tests |
+| Element click | Invoke / SelectionItem / Toggle patterns | Live button/checkbox; mock pattern cases |
+| Set control value | Value / RangeValue, read-only and password checks | Live EDIT + mock cases |
+| Advertised control actions | Invoke, toggle, selection, expand/collapse, focus | Live invoke/focus; remaining mocked |
+| Exact contextual text selection | TextPattern ranges, prefix/suffix, caret endpoints | Live RichEdit selection after supplementary Unicode; caret/suffix mocked |
+| Keyboard and Unicode typing | Balanced SendInput events; literal text | Live shared batch |
+| Fractional-page scrolling | UIA ScrollPattern view size/extent | Live vertical half-page; horizontal mocked |
+| Pixel click and drag | DWM bounds, per-thread DPI awareness, SendInput | Live on all three monitors |
+| Background window capture | WGC by HWND, no activation or desktop crop | Live with an opaque blue occluder |
+| Guarded batches / UI settling | Shared batch engine and accessibility fingerprint | Live + failure/wait/cancellation tests |
+| PNG attachment to model | Shared BinaryContent/ToolReturn | Live tools + schema tests |
+| Inline terminal preview | Existing shared terminal helper | Existing mocked terminal tests; not live on Windows Terminal |
+| Consent, pause, process restrictions | Shared policy plus Windows security process rules | Automated; live temporary-policy pause |
+| Password-value redaction | UIA protected nodes/descendants, no semantic password operations | Live password EDIT + mock descendant cases |
+
+## Results for this development change
+
+- 220 targeted `test_computer_use*.py` tests passed on Windows after the
+  [foreground activation fix](FOCUS_INVESTIGATION.md), coordinate-metadata
+  correction, and checkpoint-based batch focus-loss fix (initial baseline: 187;
+  foreground-fix baseline: 204; coordinate-fix baseline: 209).
+- The batch focus-loss fix is verified by mocked regressions, including the
+  serialized Windows runtime, and a subsequent focused public-session retest on
+  `219210a`. The owned sink took focus, the batch stopped, and the sentinel stayed
+  at zero. Cleanup and final owner pause were verified. The original failure is
+  retained in the acceptance evidence; earlier live results below are prior
+  evidence, not complete campaign reruns on this change.
+- The existing macOS backend's 21 original methods were compared by Python AST
+  against the upstream baseline: unchanged. A small `invalidate_state` method was
+  added for shared failed-batch cleanup. Imports/formatting and module location changed.
+- Existing macOS action, activation, accessibility, policy, image, and settling
+  regressions passed with their mocked native APIs.
+- The disposable live fixture passed at `(180,180)`, `(2050,180)`, and `(-1300,180)`:
+  primary landscape, secondary landscape, and left-hand portrait monitor.
+- Both the modified Code Puppy and core-plugins packages built as wheels.
+- No tests enabled persistent user consent; no existing application documents
+  were used. The follow-up focus diagnostic honors already granted consent.
+
+## What this does not prove
+
+This is implementation of the shared capability set, **not a claim that every
+Windows application exposes the same accessibility features as every Mac app**.
+A control without UIA Value/Text/Scroll patterns returns unsupported. In particular,
+Windows semantic scrolling currently needs ScrollPattern, while the macOS backend
+can inject pixel scroll events. The model can use supported keyboard/pixel actions
+where appropriate, but the plugin does not silently pretend those are semantic
+operations. Full application-by-application equivalence is not established.
+
+Live tests covered native Win32 controls and the current three-monitor hardware.
+Electron/Chromium apps, mixed-DPI configurations different from this machine,
+RDP, ARM64, other Python versions, GPU/driver variations, and real macOS native
+execution still need environment-specific validation. WGC rejects minimized,
+protected, or mismatched-geometry captures rather than falling back to the desktop.
+
+No live macOS host was available. Shared batch execution now runs off the event
+loop and invalidates failed batches; this is tested with mocks but should receive
+an actual Mac smoke test before publishing a coordinated release. An LLM-driven
+chat session after installation/consent also remains a separate end-to-end check.

{code_puppy_core_plugins-0.0.76 → code_puppy_core_plugins-0.0.78}/code_puppy_core_plugins/computer_use/README.md

@@ -1,10 +1,16 @@
-# macOS Computer Use
+# Computer Use (macOS and Windows)
 
-This opt-in plugin lets Code Puppy inspect and operate the current macOS desktop
-through Apple's Accessibility APIs. It is macOS-only and does not use a virtual
-machine or copy any Codex implementation.
+One opt-in plugin, the same twelve `computer_*` tools, and the same `/computer-use`
+commands on both platforms. macOS uses Accessibility, Quartz, and ScreenCaptureKit;
+Windows uses UI Automation, Win32 input, and HWND-targeted Windows Graphics Capture.
+No virtual machine, duplicate plugin, or alternate Windows tool API is required.
 
-## Install
+**Windows installation, examples, limitations, and live tests:** [WINDOWS.md](WINDOWS.md).
+**Implementation and verification matrix:** [PARITY.md](PARITY.md).
+
+The macOS setup and platform-specific behavior below remain supported.
+
+## macOS installation
 
 ```bash
 pip install "code-puppy[computer-use]"
@@ -51,14 +57,28 @@ application is in front.
    element's advertised actions.
 3. Run one mutation, or one guarded batch of at most 20 mutations.
 4. Fetch fresh state. A completed batch does this automatically after waiting
-   for the accessibility tree to stabilize.
+   for the accessibility tree to stabilize. Check `ui_settle.settled`: a timeout
+   is reported, not a guarantee that animations or network activity finished.
 
-Before every mutation, the target process is activated and verified as the
+For a batch `perform_action` step, use `action_name` for the advertised AX/UIA
+operation because `action` identifies the step itself, for example:
+`{"action": "perform_action", "element_id": 7, "action_name": "AXPress"}`.
+A failed batch invalidates the revision; the failing action may have partially
+executed, so inspect before retrying.
+
+On macOS, before every mutation, the target process is activated and verified as the
 frontmost application. This is required because ScreenCaptureKit can capture a
 background window, while Quartz sends global clicks and keystrokes to whichever
 application is actually frontmost. Use a guarded batch for click-then-type
 workflows so focus is retained and the result is verified in one control loop.
 
+On Windows, standalone mutations and batch entry may activate the target. Once a
+batch starts, validation only checks foreground ownership: detected focus loss
+aborts the batch instead of reactivating the target. This applies during waits,
+between actions, and through final state refresh. Already executed steps are not
+rolled back. These are checkpoint checks, not continuous focus monitoring or
+instant interruption of a blocking native call. See [WINDOWS.md](WINDOWS.md).
+
 Pixel clicks and drags use window-local screenshot pixels. The backend converts
 them to global Quartz points using the captured window bounds, so Retina scaling,
 negative display origins, and windows on secondary displays remain consistent.
@@ -72,7 +92,9 @@ and cursor placement before or after the match.
 - Persisted user consent is required before tools are exposed to the model;
   disabling the feature removes them again and blocks future captures/actions.
 - State revisions and element IDs are single-use and expire after 120 seconds.
-- Physical keyboard and pointer input do not interrupt YOLO-mode execution.
+- Ordinary physical keyboard/pointer input is not an explicit cancellation signal.
+  Windows batches additionally abort on focus loss detected at guard checkpoints.
+  Use the dedicated emergency-stop mechanism to request a stop deliberately.
 - Every capture and mutation re-checks the application policy; the emergency
   stop therefore also interrupts a batch at its next action boundary.
 - Password-field values are never returned in snapshots.
@@ -100,8 +122,8 @@ xcode-select --install
 ## Test
 
 ```bash
-uv run ruff check code_puppy/plugins/computer_use tests/plugins
-uv run pytest tests/plugins/test_computer_use_*.py -q --no-cov
+uv run ruff check code_puppy_core_plugins/computer_use
+uv run pytest tests/test_computer_use_*.py -q --no-cov
 ```
 
 For a live Ghostty smoke test, launch Code Puppy from Ghostty and ask for

code_puppy_core_plugins-0.0.78/code_puppy_core_plugins/computer_use/WINDOWS.md

@@ -0,0 +1,240 @@
+# Windows Computer Use
+
+This is the Windows backend of the existing `computer_use` plugin, not a second
+plugin. It exposes the same twelve tools and `/computer-use` commands as macOS.
+The earlier standalone `windows_desktop` prototype is not needed.
+
+## Requirements
+
+- Windows 10 (1903 or later) or Windows 11 with an unlocked interactive desktop
+  and Windows Graphics Capture support. Native wheels must be available for your
+  Python/architecture; the live tests used CPython 3.12 on Windows x64.
+- The `computer-use` extra: pywinauto 0.6.9, windows-capture 2.x, and their dependencies.
+  pywinauto supplies pywin32/comtypes; Pillow encodes screenshots.
+- A vision-capable model for screenshots/pixel actions.
+- Standard privileges. Do not elevate Code Puppy to bypass application protections.
+
+## Install this source change safely
+
+The implementation now lives in the upstream **code_puppy_core_plugins** package.
+Do **not** copy this directory into `~/.code_puppy/plugins`: that would register
+another copy of the existing plugin, potentially duplicating hooks and commands.
+
+For source testing, replace these placeholders with your checkout locations and
+try an isolated environment first (PowerShell):
+
+```powershell
+uv run --isolated --no-project --python 3.12 `
+  --with "C:\path\to\code_puppy" `
+  --with "C:\path\to\code_puppy_core_plugins[computer-use]" `
+  code-puppy
+```
+
+This uses both modified source packages without replacing your normal Code Puppy
+installation. `--isolated` isolates Python dependencies, **not** your Code Puppy
+configuration: the normal user configuration/plugins and policy still apply.
+If the prototype is installed, disable it with `/windows-desktop disable` before
+using this implementation, and move its directory out of the plugin folder while
+Code Puppy is closed. No prototype consent is silently migrated.
+
+Once a plugin release containing this change is published, install its Windows
+dependencies into the same environment as Code Puppy:
+
+```powershell
+python -m pip install "code-puppy-core-plugins[computer-use]"
+```
+
+The currently published `code-puppy[computer-use]` extra only supplies macOS
+libraries; it does not yet install Windows dependencies. A companion core
+packaging update can delegate to the plugin extra. Until this plugin change is
+published, use the source checkout above rather than expecting the existing
+published plugin version to contain it.
+
+Inside the new session:
+
+```text
+/computer-use status
+/computer-use enable
+```
+
+Enable consent yourself, then restart the isolated session to refresh tools.
+Use `/computer-use disable` to revoke consent immediately. The setting is stored
+in the normal `computer_use_policy.json`; existing macOS policy schema is retained.
+Windows writes a protected DACL allowing the current user and LocalSystem, rather
+than pretending POSIX chmod bits enforce Windows permissions.
+
+## Workflow and API
+
+1. Call `computer_get_app_state(app_name="notepad.exe")`.
+2. Inspect the screenshot and UIA `nodes`. Use only each node's advertised actions.
+3. Use the returned `state_revision` for one mutation or a guarded batch.
+4. Inspect fresh state after every standalone mutation. Successful batches return it.
+
+### Batch focus-loss safety
+
+A new standalone mutation or batch may activate its explicitly targeted window.
+After Windows batch entry, focus checks never automatically reactivate it. If
+another window is foreground at a guard checkpoint, the batch stops, invalidates
+its revision, and reports completed steps. Pending sentinel actions must not run.
+This includes wait polling (up to 50ms sleeps), action boundaries, and the final
+settling/state-refresh phase. Even final refresh cannot steal focus back.
+
+This is checkpoint-based detection: a transient loss and return between checks
+can go unobserved, and a blocking UIA call cannot necessarily be interrupted.
+Completed actions are not rolled back. A provider action that opens another
+window can therefore abort the remainder of its batch; inspect fresh state and
+explicitly target that window in a new request instead of retrying blindly.
+Explicit cancellation, consent/policy, emergency stop, expiry, process identity,
+and geometry guards remain in force. Standalone activation and the macOS backend
+retain their existing behavior.
+
+The focus-loss fix has mocked regression coverage, including the serialized UIA
+runtime path, plus a focused public-session retest on `219210a`: the owned sink
+took focus, the batch stopped without reactivation, and the sentinel stayed at
+zero. Cleanup and final owner pause were verified. The original failure remains
+historical evidence. This closes that regression, not all deferred acceptance
+coverage or universal application compatibility.
+
+`app_name` accepts an exact executable basename (`notepad.exe`), basename without
+extension (`notepad`), exact window title, full executable path, or `hwnd:NUMBER`.
+An ambiguous match returns matching titles and `hwnd:` selectors instead of guessing
+which document to change. State returns an HWND selector as `application` so a title
+change does not cause subsequent batch verification to target another window.
+
+All existing tools are present:
+
+- `computer_get_app_state`, `computer_snapshot`, `computer_screenshot`
+- `computer_click`, `computer_set_value`, `computer_perform_action`
+- `computer_select_text`, `computer_press_key`, `computer_type_text`
+- `computer_scroll`, `computer_drag`, `computer_use_batch`
+
+Examples:
+
+```json
+{"state_revision":"...","element_id":7,"value":"Updated text"}
+```
+
+```json
+{"state_revision":"...","element_id":9,"text":"dog","prefix":"blue ","mode":"cursor_after"}
+```
+
+```json
+{"state_revision":"...","steps":[
+  {"action":"perform_action","element_id":7,"action_name":"UIASetFocus"},
+  {"action":"press_key","key":"a","modifiers":["control"]},
+  {"action":"type_text","text":"Literal Unicode text"}
+]}
+```
+
+The batch discriminator is `action`; `action_name` specifies the UIA/AX operation
+for a `perform_action` step. Use `text`, `cursor_before`, or `cursor_after` for text
+selection. Exact ambiguous matches require prefix/suffix context. Selection uses
+UIA text ranges/endpoints, including text after supplementary Unicode characters.
+
+`control`/`ctrl`, `shift`, `option`/`alt`, and `command`/`win` are accepted modifiers.
+**command means the Windows key**, not Ctrl. Use `control` for Windows shortcuts.
+Newline/tab in typed text send Enter/Tab; these can submit a form. The clipboard
+is not modified. Secure-attention and lock-screen shortcuts are blocked.
+
+## Multiple displays and DPI
+
+Screenshots and x/y action coordinates use original **window-local physical pixels**.
+Native input maps those pixels to global Windows physical coordinates. Portrait
+screens, negative monitor origins, and different display scaling are handled via
+per-thread DPI awareness and DWM visible-frame bounds. Windows display settings are
+never changed. Moving/resizing a window requires fresh state.
+
+`action_coordinate_system` and `screenshot_coordinate_system` both describe these
+**window-local** physical pixels. `bounds_coordinate_system` separately identifies
+UIA node bounds and `window_bounds_points` as **global** Windows physical pixels
+(the legacy `window_bounds_points` field name is retained for API compatibility).
+For a window at `(180,180)`, a global control center `(288,292)` corresponds to
+local action coordinates `(108,112)`, not `(288,292)`. Prefer semantic element IDs.
+Earlier development builds incorrectly labeled action coordinates as global;
+that metadata bug is corrected without changing actual click/drag mapping.
+
+The live fixture was tested on the user's two 1920x1080 landscape monitors and the
+1440x2560 portrait monitor left of the primary screen, including real pixel clicks
+and drags. These tests cover the current configuration, not every GPU/scaling/RDP
+combination. Capture rejects mismatched frame geometry rather than guessing.
+
+## Safety, semantics, and limits
+
+- Consent is off by default. `/computer-use pause` stops subsequent operations;
+  `/computer-use resume` requires the user to confirm it is safe to continue.
+- Holding **Pause/Break** or moving to the primary display's top-left 2x2 pixels
+  is an additional Windows emergency stop. It takes effect at the next check.
+  A blocking provider/OS call must return before cooperative cancellation can act.
+- Deny an executable with `/computer-use deny notepad.exe`; allow removes an
+  explicit denial. Hardcoded security-process restrictions cannot be removed.
+  Locked/UAC/non-default desktops are rejected. This is not a security sandbox.
+- UIA password nodes and protected descendants have names/help/values redacted;
+  semantic text operations on them are refused. **Screenshots are not redacted.**
+  PNGs/metadata go to the configured model and may be retained in chat/session data.
+- App state activates the target before inspecting it, as macOS does. Explicit
+  `computer_screenshot(app_name=...)` captures the HWND in the background without
+  activating it or including an overlapping window. Minimized/protected/uncapturable
+  windows can fail. There is no desktop-crop or PrintWindow fallback.
+- Semantic click/value/text operations require the relevant UIA patterns. Custom
+  controls and some legacy EDIT providers may not implement them. Unsupported is
+  returned honestly, not replaced by guessed typing. Pixel clicks/drags and keyboard
+  input remain available where the app permits them.
+- Fractional-page scrolling uses UIA ScrollPattern view size and extent. A window
+  with no scroll provider returns unsupported; wheel notches are not mislabeled as
+  pixels/pages. Horizontal support also depends on the provider.
+- The tree examines at most 2,000 nodes and returns 1-500 nodes, with truncation
+  metadata. A stability fingerprint is not proof that the user's task completed.
+- Revisions are single-use and expire after 120 seconds. Batches stop on failure;
+  actions are not transactional and cannot undo already delivered input.
+- Screenshots use unique temporary PNG files, like the macOS backend. Explicit
+  output paths are never overwritten. WGC startup/frame/cleanup waits are bounded;
+  a wedged native driver may retain a daemon cleanup worker until process exit.
+- Inline previews use the same best-effort terminal helper as macOS. Ordinary
+  Windows Terminal may not support its protocols; the model still receives the PNG.
+
+## Diagnosing background-capture focus changes
+
+Explicit Windows screenshots include `foreground_observation.before_capture` and
+`foreground_observation.after_capture`: HWND samples taken immediately around the
+native HWND capture. They do not include subsequent image presentation, terminal
+UI work, compaction, or a later verification shell call. Equal samples do not rule
+out a transient focus change, and different samples do not establish causality.
+
+If a separate shell check reports the controller foreground afterward, inspect
+these samples before blaming capture. The screenshot path does not activate or
+restore a foreground window; it must not steal focus back from user activity.
+The interactive acceptance report's controller-focus observation remains unresolved
+until measured with this instrumentation in that session.
+
+## Foreground activation troubleshooting
+
+Windows can refuse a plain `SetForegroundWindow` request even for a visible,
+allowed target. The backend first tries that API, then requests real UIA
+`SetFocus` on the target when available. It independently checks the foreground
+HWND before proceeding and rechecks policy, cancellation, and process identity.
+It never injects Alt keys, attaches input queues, changes foreground-lock settings,
+or elevates privileges to force activation.
+
+If both methods fail, the error includes the requested/current HWND, the Win32
+request result, and whether UIA was attempted. Manually focus the target and get
+fresh state; do not forge revisions or disable focus checks. This recovery is
+provider-dependent, not a promise that Windows will always allow activation.
+
+See [FOCUS_INVESTIGATION.md](FOCUS_INVESTIGATION.md) for the reproduced failure and
+verification. Existing sessions must restart to load the source fix.
+
+## Tests
+
+The checkout's `.venv` is a test environment; normal installation is not altered.
+
+```powershell
+.venv\Scripts\python.exe -c "import glob,pytest; raise SystemExit(pytest.main([*glob.glob('tests/test_computer_use*.py'),'-q','--no-cov']))"
+.venv\Scripts\python.exe tests\live_windows_computer_use.py --x 180 --y 180
+.venv\Scripts\python.exe tests\live_windows_computer_use.py --x 2050 --y 180
+.venv\Scripts\python.exe tests\live_windows_computer_use.py --x -1300 --y 180
+```
+
+Only use coordinates inside your own active monitor layout. The live test creates
+native EDIT, password EDIT, RichEdit, button, checkbox, and solid-color occluder
+fixtures. It uses temporary consent/screenshots and closes only its own processes.
+It does not open or modify existing documents or enable persistent user consent.

{code_puppy_core_plugins-0.0.76 → code_puppy_core_plugins-0.0.78}/code_puppy_core_plugins/computer_use/accessibility.py

@@ -3,7 +3,8 @@
 from __future__ import annotations
 
 from collections import Counter
-from typing import Any, Callable
+from collections.abc import Callable
+from typing import Any
 
 MAX_SCAN_NODES = 4_000
 # Electron embeds its web accessibility tree beneath a deep stack of generic

code_puppy_core_plugins-0.0.78/code_puppy_core_plugins/computer_use/backend.py

@@ -0,0 +1,28 @@
+"""Platform selection behind the stable computer_* tool contract.
+
+Native frameworks are lazy: importing the plugin never captures the desktop,
+initializes COM, or prompts for accessibility permissions.
+"""
+
+from __future__ import annotations
+
+import sys
+
+from .backend_types import ComputerUseError
+from .macos_backend import MacOSBackend
+
+__all__ = ["ComputerUseError", "MacOSBackend", "backend", "create_backend"]
+
+
+def create_backend(platform: str | None = None):
+    platform = platform or sys.platform
+    if platform == "win32":
+        from .windows_runtime import WindowsRuntime
+
+        return WindowsRuntime()
+    # Keep the importable macOS backend for existing callers and non-native unit
+    # tests. Registration separately gates tools to the two supported platforms.
+    return MacOSBackend()
+
+
+backend = create_backend()

code_puppy_core_plugins-0.0.78/code_puppy_core_plugins/computer_use/batch.py

@@ -0,0 +1,93 @@
+"""Shared, synchronous batch engine; the async adapter offloads the entire batch."""
+
+from __future__ import annotations
+
+import math
+import time
+from contextlib import ExitStack
+
+from .backend_types import ComputerUseError
+
+MAX_BATCH_STEPS = 20
+ACTIONS = {
+    "set_value": "set_value",
+    "perform_action": "perform_action",
+    "select_text": "select_text",
+    "press_key": "press_key",
+    "type_text": "type_text",
+    "scroll": "scroll_pages",
+    "drag": "drag_pixel",
+}
+
+
+def run_batch(backend, revision, steps, settler):
+    if not 1 <= len(steps) <= MAX_BATCH_STEPS:
+        return {"success": False, "error": "A batch must contain 1 to 20 steps."}
+    completed = []
+    initial = None
+    scope = ExitStack()
+    try:
+        batch_scope = getattr(backend, "batch_scope", None)
+        if batch_scope is not None:
+            scope.enter_context(batch_scope(revision))
+        initial = backend.require_state(revision)
+        for index, step in enumerate(steps):
+            action = str(step.get("action", ""))
+            kwargs = {key: value for key, value in step.items() if key != "action"}
+            try:
+                # Even wait steps honor expiry and emergency pause.
+                backend.require_state(revision)
+                if action == "wait":
+                    seconds = float(kwargs.get("seconds", 1))
+                    if not math.isfinite(seconds) or not 0 <= seconds <= 10:
+                        raise ComputerUseError(
+                            "Wait seconds must be finite and within 0..10"
+                        )
+                    deadline = time.monotonic() + seconds
+                    while time.monotonic() < deadline:
+                        backend.require_state(revision)
+                        time.sleep(min(0.05, max(0, deadline - time.monotonic())))
+                    result = {"success": True, "seconds": seconds}
+                elif action == "click":
+                    if "element_id" in kwargs and ("x" in kwargs or "y" in kwargs):
+                        raise ComputerUseError("Provide element_id or x/y, not both")
+                    method = (
+                        backend.click if "element_id" in kwargs else backend.click_pixel
+                    )
+                    result = method(revision, consume=False, **kwargs)
+                elif action in ACTIONS:
+                    if action == "perform_action":
+                        # The outer 'action' is the batch discriminator.
+                        kwargs["action"] = kwargs.pop("action_name")
+                    result = getattr(backend, ACTIONS[action])(
+                        revision, consume=False, **kwargs
+                    )
+                else:
+                    raise ComputerUseError(f"Unknown action: {action}")
+            except Exception as exc:  # noqa: BLE001 - stop the batch on any provider failure.
+                result = {"success": False, "error": str(exc)}
+            completed.append({"index": index, "action": action, "result": result})
+            if not result.get("success"):
+                backend.invalidate_state()
+                return {
+                    "success": False,
+                    "completed_steps": completed,
+                    "warning": "A failed action may have partially executed; inspect before retrying.",
+                }
+        backend.require_state(revision, consume=True)
+        settle = settler(backend.snapshot, initial.application)
+        updated = backend.get_app_state(initial.application, 100)
+        if not updated.get("success"):
+            return {
+                "success": True,
+                "completed_steps": completed,
+                "updated_state_error": updated.get("error"),
+            }
+        updated["completed_steps"] = completed
+        updated["ui_settle"] = settle
+        return updated
+    except Exception as exc:  # noqa: BLE001 - stop the batch on any provider failure.
+        backend.invalidate_state()
+        return {"success": False, "error": str(exc), "completed_steps": completed}
+    finally:
+        scope.close()

{code_puppy_core_plugins-0.0.76 → code_puppy_core_plugins-0.0.78}/code_puppy_core_plugins/computer_use/commands.py

@@ -4,8 +4,10 @@ from __future__ import annotations
 
 import shlex
 
+from code_puppy.i18n import t
 from code_puppy.messaging import emit_error, emit_info, emit_success, emit_warning
 
+from .backend_types import ComputerUseError
 from .policy import policy_store
 
 
@@ -13,12 +15,20 @@ def command_help() -> list[tuple[str, str]]:
     return [
         (
             "computer-use",
-            "Enable, disable, pause, or configure macOS Computer Use",
+            t("plugin.computer-use.command.help"),
         )
     ]
 
 
 def handle_command(command: str, name: str) -> bool | None:
+    try:
+        return _handle_command(command, name)
+    except (ComputerUseError, OSError) as exc:
+        emit_error(str(exc))
+        return True
+
+
+def _handle_command(command: str, name: str) -> bool | None:
     if name != "computer-use":
         return None
     try:
@@ -46,7 +56,7 @@ def handle_command(command: str, name: str) -> bool | None:
         emit_success("Computer Use resumed.")
     elif subcommand in {"allow", "deny"}:
         if len(tokens) < 3:
-            emit_error("Usage: /computer-use allow|deny BUNDLE_ID")
+            emit_error(t("plugin.computer-use.command.policy_usage"))
             return True
         bundle_id = tokens[2]
         if subcommand == "deny":