@ycheung5/pi-computer-use 0.0.0-stage → 0.1.1

package/GLOSSARY.md

@@ -0,0 +1,38 @@
+# Pi Computer Use
+
+Computer use lets a Pi agent observe and operate Windows desktop applications and web pages.
+
+## Language
+
+**Native Windows host**:
+An environment where Pi runs directly on Windows and controls that Windows desktop.
+
+**WSL2 host**:
+An environment where Pi runs inside Windows Subsystem for Linux 2 and controls the associated Windows desktop.
+
+**Window lifetime**:
+The period from a native window's creation until its destruction. A replacement window has a different lifetime even when the operating system reuses its numeric handle.
+
+**Window reference**:
+An observed identity bound to one window lifetime. A reference to an ended lifetime does not identify its replacement.
+
+**Feature parity**:
+A comparison of the capabilities and observable behavior of two systems, as used in the historical DSH comparison records.
+
+**Capability coverage**:
+The set of agreed Pi desktop, browser, settings, indicator and stopping behaviors exercised by named checks, including supported cases and explicit limitations.
+
+**Desktop ownership**:
+Exclusive permission for one Pi session to operate a Windows desktop during a computer-use operation, shared across native Windows and WSL2 hosts.
+
+**Exclusive control**:
+A period in which the agent operates the desktop and ordinary physical keyboard and mouse input is suppressed, while emergency stopping remains available.
+
+**Human handback**:
+A pause in automation during which exclusive control has ended and the user can interact normally. Resuming requires an explicit user continuation.
+
+**Computer-use operation**:
+A session's related computer-use activity across model turns until processing settles or the operation is stopped. Individual tool calls are steps within the operation.
+
+**Emergency stop**:
+A user request to stop active input and reject further actions in the stopped operation; it cannot undo actions already delivered to an application.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Pi Computer Use contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,166 @@
-# Temporary Holding Version
+# Pi Computer Use
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+![Pi Computer Use: give Pi a Windows desktop, from native Windows or WSL2.](assets/readme-banner.svg)
+
+[![Checks](https://github.com/ycheung5/pi-computer-use/actions/workflows/checks.yml/badge.svg)](https://github.com/ycheung5/pi-computer-use/actions/workflows/checks.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+![Windows 11 + WSL2](https://img.shields.io/badge/Windows_11_%2B_WSL2-0078D4)
+
+**Let your Pi agent work with Windows apps and browser pages.** Read controls, take screenshots, enter text and complete tasks from native Windows or WSL2.
+
+A standalone Pi extension with **25 computer-use tools**, a blue control border and explicit ways to stop or take back control. It uses your existing Pi model setup.
+
+[Install](#install) · [Example tasks](#try-it) · [Capabilities](#what-pi-can-do) · [Documentation](docs/README.md)
+
+> **Tested baseline:** Windows 11 x64, WSL2 Ubuntu and Pi SDK/CLI 1.4.0. See the [release status and known limits](docs/validation.md#first-release-preparation) for the exact accepted scope.
+
+## What Pi can do
+
+| Capability | What it enables |
+| --- | --- |
+| Windows apps | Discover windows and controls, inspect their state, invoke buttons and manage windows. |
+| Browser pages | Work with forms, tabs, navigation, dialogs and file transfers through a browser protocol channel. |
+| Visual tasks | Capture screenshots and read visible text with Windows OCR. |
+| Desktop input | Move, click, drag, scroll, type Unicode text and send keyboard shortcuts. |
+| Clipboard workflows | Paste text with preservation and cleanup checks; refuse unsupported preservation cases. |
+| Control and preferences | Show a blue border during exclusive control, stop active work and choose which tools Pi loads. |
+
+For ordinary app requests, Pi is guided to use Windows defaults and reuse an existing window with a separate tab or workspace when the app supports one. Dedicated browser automation uses a separate profile when that workspace is explicitly selected. [Workspace policy](docs/windows.md#default-apps-and-workspaces)
+
+## Install
+
+Use your existing Pi installation. Another computer-use extension is not required.
+
+### From npm
+
+Install [@ycheung5/pi-computer-use](https://www.npmjs.com/package/@ycheung5/pi-computer-use) from npm:
+
+```sh
+pi install npm:@ycheung5/pi-computer-use
+```
+
+Use the scoped name above. The unscoped `pi-computer-use` package belongs to a different publisher.
+
+### Build and try this checkout
+
+With Node 22.19+ and npm:
+
+```sh
+git clone https://github.com/ycheung5/pi-computer-use.git
+cd pi-computer-use
+npm ci --ignore-scripts
+npm run build
+pi -e .
+```
+
+To retain the built checkout as a local Pi package, run `pi install .` from that directory. Source checkouts do not include compiled output; build before registering the package.
+
+### Install a built archive
+
+For a prebuilt `ycheung5-pi-computer-use-0.1.1.tgz` produced by `npm pack`:
+
+```sh
+mkdir pi-computer-use
+tar -xzf ycheung5-pi-computer-use-0.1.1.tgz -C pi-computer-use --strip-components=1
+cd pi-computer-use
+npm install --omit=dev --omit=peer --ignore-scripts
+pi install .
+```
+
+Pi supplies its SDK and TypeBox peers. The archive contains compiled code, so this path needs no TypeScript compiler. Keep the extracted directory while Pi references it.
+
+### Requirements
+
+- **Windows 11 x64**, directly or through **WSL2 Ubuntu**, with an interactive desktop. WSL needs Windows executable interop, `cmd.exe` and `wslpath`.
+- **Pi** with your configured model. Tested SDK/CLI: **1.4.0**. A vision-capable model is needed to interpret screenshots; credentials are not bundled.
+- **Windows PowerShell 5.1** and the built-in .NET Framework compiler/desktop assemblies. A separate .NET SDK or Windows Node installation is not required.
+- **Chrome or Edge** for supported browser workflows. Explicitly selected installed Firefox-family browsers are experimental.
+- **Windows OCR language support** for the text you want to recognize.
+
+The first computer-use call downloads a pinned, checksum-verified private Windows Node runtime from nodejs.org into `%LOCALAPPDATA%\pi-computer-use\runtime`. Later calls reuse the worker. Initial setup needs HTTPS access to nodejs.org and does not change your global Node installation.
+
+## Try it
+
+Start with a read-only connection check:
+
+```text
+Call windows_computer_use_health, then list the open windows.
+Do not activate a window or send any input.
+```
+
+Then give Pi a goal. These are example requests, not promises of support for every app:
+
+```text
+Use Notepad to create a separate test document. Keep any existing documents
+unchanged. Type "Hello from Pi — 中文 café 😀" and verify the editor text.
+Leave the test document open and unsaved.
+```
+
+```text
+Open https://example.com in my Windows default browser. Reuse an existing
+window with a new tab if possible, then tell me the page title.
+```
+
+```text
+Find my Downloads window and summarize its visible controls.
+Do not click, type or activate it.
+```
+
+## Stay in control
+
+A **blue gradient border** marks exclusive desktop control. During that period, Windows hooks suppress ordinary physical input so it does not compete with the agent.
+
+- Press **Esc** when physical-input protection is armed, or use **`/computer-stop`** in Pi, to interrupt active work.
+- When a task needs your input, the agent should release control and wait for your explicit continuation.
+- Timed-out or cancelled input is not automatically replayed. Unconfirmed cleanup blocks dependent automation.
+
+Windows can silently remove timed-out hooks. Suppression and physical stop keys cannot be guaranteed after that OS behavior; detected protection loss stops automation. Read the [control and stopping contract](docs/stopping.md) before relying on unattended desktop work.
+
+## Make it yours
+
+```text
+/computer-settings show
+/computer-settings edit
+/computer-settings set {"toolPreset":"lazy"}
+```
+
+Settings are shared across projects and native Windows/WSL2 Pi in `%LOCALAPPDATA%\pi-computer-use\settings.json`. Choose full, lite, lazy or custom tool loading, plus browser and indicator preferences. The local border remains required during exclusive control. [Settings guide](docs/settings.md)
+
+## Tested scope and known limits
+
+The input-timing update passed **214 portable tests**, with one opt-in historical-reference skip, plus native compilation and built-distribution checks. Named native/browser agent workflows and human control/display checks have separate [acceptance receipts](docs/validation.md#capability-coverage). CI is not a claim that every application or desktop state works.
+
+- Second-computer WSL installation and desktop actions were user-confirmed. Fresh native-Windows installation, broken-prerequisite handling and shutdown in that clean environment remain accepted, unverified first-release test limitations.
+- Display mapping was exercised at **100%, 150% and 175%**, including a changed negative-origin layout. Other configurations, including 200%, remain unverified.
+- Windows may refuse foreground activation or access to elevated/protected interfaces. The extension reports refusal rather than bypassing Windows protections.
+- Explicit Unicode SendInput defaults to a **50 ms per-unit delay**; explicit native pacing and per-call deadlines are [configurable](docs/text.md#sendinput-pacing-and-deadlines). Clipboard preservation has format-dependent limits; installed Firefox support is experimental.
+- Arm64, genuine same-HWND/PID reuse, verified UIPI denial and additional recorder/remote/capture variants remain unverified.
+
+See the [release scope](docs/validation.md#first-release-preparation) for the accepted limits and their evidence.
+
+## Develop and verify
+
+```sh
+npm run check
+npm run test:package
+```
+
+Build with `npm run build`; individual checks are `npm test`, `npm run typecheck` and `npm run lint`. Format with `npm run format` or apply configured fixes with `npm run lint:fix`. Packaging verifies compiled/native assets, licenses, metadata and documentation links.
+
+Live Windows and paid-model runners are opt-in. Read their [prerequisites and evidence scopes](docs/validation.md#reproduce-automated-checks) before running them; serialize live WSL checks with `flock -w 120 /tmp/pi-cu-live-tests.lock`. Historical DSH comparisons are optional developer utilities, not release prerequisites.
+
+[Documentation index](docs/README.md) · [Requirements](docs/spec.md) · [Implementation plan](docs/implementation-plan.md) · [Report a bug](https://github.com/ycheung5/pi-computer-use/issues)
+
+## Troubleshooting
+
+| Symptom | Check |
+| --- | --- |
+| `WORKER_ASSET_MISSING` | Build the checkout or use a built package. |
+| Runtime download or checksum failure | Allow nodejs.org HTTPS. Remove only the indicated corrupt private-runtime cache before retrying on a trusted connection. |
+| `WSL_BOOTSTRAP_FAILED` | Check Windows interop, `cmd.exe` and `wslpath`. A sandbox `UtilBindVsockAnyPort` failure is an execution restriction. |
+| Native compiler or OCR unavailable | Check Windows Framework/desktop components and installed OCR languages. |
+| Desktop busy or stopped | Finish or stop the owning operation, then provide new user input. Do not replay uncertain actions through another tool. |
+
+## License
+
+[MIT](LICENSE). Created by **ycheung5**. Bundled notices: [Node](assets/node-LICENSE.txt) and [Puppeteer](assets/puppeteer-LICENSE.txt). Installed dependencies retain their own licenses.

package/assets/accessibility.cs

@@ -0,0 +1,208 @@
+using System;
+using System.Collections.Generic;
+using System.Diagnostics;
+using System.Runtime.InteropServices;
+using System.Text;
+using System.Threading;
+using System.Windows;
+using System.Windows.Automation;
+
+partial class WindowsBridge {
+  [DllImport("user32.dll")] static extern bool EnumChildWindows(IntPtr parent, EnumWindow callback, IntPtr parameter);
+  [DllImport("user32.dll")] static extern IntPtr GetAncestor(IntPtr handle, uint flags);
+  [DllImport("user32.dll")] static extern bool SetForegroundWindow(IntPtr handle);
+  sealed class ObservedElement {
+    public AutomationElement Element;
+    public IntPtr Window;
+    public uint Owner;
+    public DateTime Seen;
+    public string Runtime,Reference;
+  }
+  static readonly Dictionary<string, ObservedElement> observed = new Dictionary<string, ObservedElement>();
+  static readonly Dictionary<string,string> elementKeys=new Dictionary<string,string>();
+  static bool Option(Dictionary<string,object> args, string key, bool fallback) { return args.ContainsKey(key) ? Convert.ToBoolean(args[key]) : fallback; }
+  static string TextOption(Dictionary<string,object> args, string key, string fallback) { return args.ContainsKey(key) ? Convert.ToString(args[key]) : fallback; }
+  static int NumberOption(Dictionary<string,object> args, string key, int fallback) { return args.ContainsKey(key) ? Convert.ToInt32(args[key]) : fallback; }
+  static string WindowTitle(IntPtr handle) { StringBuilder value=new StringBuilder(2048); GetWindowText(handle,value,value.Capacity); return value.ToString(); }
+  static uint WindowOwner(IntPtr handle) { uint owner; GetWindowThreadProcessId(handle,out owner); return owner; }
+  static IntPtr ResolveObservationWindow(Dictionary<string,object> args) {
+    WindowReference bound=ResolveWindowReference(args);
+    long explicitHandle=bound!=null?bound.Window.ToInt64():args.ContainsKey("nativeWindowHandle") ? Convert.ToInt64(args["nativeWindowHandle"]) : 0;
+    uint owner=args.ContainsKey("processId") ? Convert.ToUInt32(args["processId"]) : 0;
+    string title=TextOption(args,"windowTitle","");
+    IntPtr handle=IntPtr.Zero;
+    if(explicitHandle!=0) {
+      handle=new IntPtr(explicitHandle);
+      if(!IsWindow(handle) || (owner!=0 && WindowOwner(handle)!=owner) || (title.Length>0 && WindowTitle(handle).IndexOf(title,StringComparison.OrdinalIgnoreCase)<0)) throw new InvalidOperationException("TARGET_IDENTITY_CHANGED: window no longer matches the requested owner/target; observe again");
+    } else if(owner!=0 || title.Length>0) {
+      List<IntPtr> candidates=new List<IntPtr>();
+      EnumWindows(delegate(IntPtr candidate,IntPtr ignored) {
+        if(!IsWindowVisible(candidate)) return true;
+        if(owner!=0 && WindowOwner(candidate)!=owner) return true;
+        if(title.Length>0 && WindowTitle(candidate).IndexOf(title,StringComparison.OrdinalIgnoreCase)<0) return true;
+        candidates.Add(candidate); return true;
+      },IntPtr.Zero);
+      if(candidates.Count==0) throw new InvalidOperationException("TARGET_NOT_FOUND: no window matches the requested target");
+      if(candidates.Count>1) throw new InvalidOperationException("AMBIGUOUS_TARGET: multiple windows match; specify nativeWindowHandle");
+      handle=candidates[0];
+    } else handle=GetForegroundWindow();
+    if(handle==IntPtr.Zero || !IsWindow(handle)) throw new InvalidOperationException("TARGET_NOT_FOUND: no usable foreground window");
+    CheckPointerTitle(args,handle);
+    UIntPtr ignoredResult;
+    bool responsive=SendMessageTimeout(handle,0,UIntPtr.Zero,IntPtr.Zero,3,100,out ignoredResult)!=IntPtr.Zero;
+    if(!responsive) throw new InvalidOperationException(Marshal.GetLastWin32Error()==5 ? "TARGET_ACCESS_DENIED: Windows refused access" : "TARGET_UNRESPONSIVE: avoid repeated UIA reads; use native discovery/capture");
+    if(Option(args,"activate",false) && GetForegroundWindow()!=handle) {
+      uint targetOwner=WindowOwner(handle);Dictionary<string,object> diagnostic=new Dictionary<string,object>();
+      CheckWindowReference(handle);
+      if(!ManagementActivate(handle,targetOwner,diagnostic)) {diagnostic["foregroundAtRefusal"]=ManagementForegroundRelationship(GetForegroundWindow(),handle,targetOwner);throw new ActivationRefusedException(diagnostic);}
+    }
+    CheckWindowReference(handle);return handle;
+  }
+  static Dictionary<string,object> UiaBox(System.Windows.Rect bounds) {
+    return new Dictionary<string,object> { {"x",bounds.X},{"y",bounds.Y},{"width",bounds.Width},{"height",bounds.Height},{"centerX",bounds.X+bounds.Width/2},{"centerY",bounds.Y+bounds.Height/2} };
+  }
+  static string RuntimeId(AutomationElement element) { return "uia:rt:"+String.Join("-",element.GetRuntimeId()); }
+  static void PruneElements() {
+    List<string> remove=new List<string>(); DateTime cutoff=DateTime.UtcNow.AddMinutes(-5);
+    foreach(KeyValuePair<string,ObservedElement> item in observed) if(item.Value.Seen<cutoff || !IsWindow(item.Value.Window) || WindowOwner(item.Value.Window)!=item.Value.Owner) remove.Add(item.Key);
+    foreach(string key in remove) {ObservedElement item=observed[key];elementKeys.Remove(item.Reference+":"+item.Runtime);observed.Remove(key);}
+    if(observed.Count>20000) {observed.Clear();elementKeys.Clear();}
+  }
+  static IntPtr ElementWindow(AutomationElement element,IntPtr window) {
+    if(window!=IntPtr.Zero)return window;
+    IntPtr own=new IntPtr(element.Current.NativeWindowHandle);
+    if(own!=IntPtr.Zero)return GetAncestor(own,2);
+    AutomationElement parent=element;
+    for(int level=0;level<128 && parent!=null;level++){int hwnd=parent.Current.NativeWindowHandle;if(hwnd!=0)return GetAncestor(new IntPtr(hwnd),2);parent=TreeWalker.RawViewWalker.GetParent(parent);}
+    return IntPtr.Zero;
+  }
+  static Dictionary<string,object> DescribeElement(AutomationElement element, IntPtr window, bool full, int depth, bool bind=true) {
+    AutomationElement.AutomationElementInformation current=element.Current;
+    string runtime=RuntimeId(element),id=null;window=ElementWindow(element,window);
+    var bindingMetadata=new Dictionary<string,object>();WindowReference binding=bind?ObserveWindowReference(window,bindingMetadata):null;
+    if(binding!=null) {
+      string key=binding.Id+":"+runtime;ObservedElement existing;
+      if(elementKeys.TryGetValue(key,out id) && observed.TryGetValue(id,out existing))existing.Seen=DateTime.UtcNow;
+      else {id="u:"+referenceEpoch+":"+(++elementSequence).ToString("x");elementKeys[key]=id;observed[id]=new ObservedElement {Element=element,Window=window,Owner=binding.Owner,Seen=DateTime.UtcNow,Runtime=runtime,Reference=binding.Id};}
+    }
+    ControlType controlType=current.ControlType;
+    bool unknownType=controlType==null || String.IsNullOrEmpty(controlType.ProgrammaticName);
+    Dictionary<string,object> info=new Dictionary<string,object> { {"name",current.Name ?? ""},{"controlType",unknownType ? "Unknown" : controlType.ProgrammaticName.Replace("ControlType.","")},{"depth",depth},{"isEnabled",current.IsEnabled},{"isOffscreen",current.IsOffscreen},{"hasKeyboardFocus",current.HasKeyboardFocus},{"isPassword",current.IsPassword} };
+    if(id!=null)info["id"]=id;
+    if(binding==null && bindingMetadata.ContainsKey("windowRefUnavailable"))info["windowRefUnavailable"]=bindingMetadata["windowRefUnavailable"];
+    if(!current.BoundingRectangle.IsEmpty) info["boundingBox"]=UiaBox(current.BoundingRectangle);
+    if(unknownType) info["controlTypeUnavailable"]=true;
+    if(!String.IsNullOrEmpty(current.AutomationId)) info["automationId"]=current.AutomationId;
+    if(!String.IsNullOrEmpty(current.ClassName)) info["className"]=current.ClassName;
+    if(full) {
+      info["processId"]=current.ProcessId; info["nativeWindowHandle"]=current.NativeWindowHandle; info["runtimeId"]=element.GetRuntimeId(); info["localizedControlType"]=current.LocalizedControlType;
+      List<string> patterns=new List<string>(); foreach(AutomationPattern pattern in element.GetSupportedPatterns()) patterns.Add(pattern.ProgrammaticName.Replace("PatternIdentifiers.Pattern","")); info["patterns"]=patterns;
+      object value;
+      if(!current.IsPassword && element.TryGetCurrentPattern(ValuePattern.Pattern,out value)) { string text=((ValuePattern)value).Current.Value; if(!String.IsNullOrEmpty(text)) info["value"]=text.Length>2000 ? text.Substring(0,2000) : text; }
+    }
+    return info;
+  }
+  static AutomationElement ResolveObservedElement(string id, Dictionary<string,object> args, out IntPtr window) {
+    ObservedElement entry;
+    if(id.StartsWith("uia:hwnd:",StringComparison.Ordinal)) {
+      string[] parts=id.Split(':'); long handle; uint pid;
+      if(parts.Length!=5 || parts[3]!="pid" || !Int64.TryParse(parts[2],out handle) || !UInt32.TryParse(parts[4],out pid)) throw new InvalidOperationException("INVALID_ELEMENT_ID");
+      window=new IntPtr(handle);
+      if(!IsWindow(window) || WindowOwner(window)!=pid) throw new InvalidOperationException("STALE_ELEMENT: native window identity changed");
+      ValidateElementTarget(window,args); return AutomationElement.FromHandle(window);
+    }
+    if(!observed.TryGetValue(id,out entry) || entry.Seen<DateTime.UtcNow.AddMinutes(-5)) throw new InvalidOperationException("STALE_ELEMENT: element was not observed or has expired; observe again");
+    window=entry.Window;
+    if(!IsWindow(window) || WindowOwner(window)!=entry.Owner) { observed.Remove(id); throw new InvalidOperationException("STALE_ELEMENT: owning window identity changed"); }
+    WindowReference root=ResolveWindowReference(new Dictionary<string,object>{{"windowRef",entry.Reference}});
+    ValidateElementTarget(window,args);
+    try { if(RuntimeId(entry.Element)!=entry.Runtime) throw new InvalidOperationException("STALE_ELEMENT: runtime identity changed"); entry.Element.GetCurrentPropertyValue(AutomationElement.ProcessIdProperty); return entry.Element; }
+    catch(ElementNotAvailableException) { observed.Remove(id); throw new InvalidOperationException("STALE_ELEMENT: control no longer exists; observe again"); }
+  }
+  static void ValidateElementTarget(IntPtr window, Dictionary<string,object> args) {
+    if(args.ContainsKey("windowRef") || args.ContainsKey("nativeWindowHandle") || args.ContainsKey("processId") || args.ContainsKey("windowTitle")) if(ResolveObservationWindow(args)!=window) throw new InvalidOperationException("ELEMENT_WRONG_OWNER: element belongs to a different target window; observe the target window and element again; do not automatically replay the action");
+  }
+  sealed class Traversal {
+    public Stopwatch Watch=Stopwatch.StartNew(); public int Budget,Maximum,Depth,Count,UnknownControlTypes; public string Reason; public string Error; public bool Offscreen,Full,BrowserLayout; public TreeWalker Walker;
+    public bool CanRead() { if(Watch.ElapsedMilliseconds>=Budget) { if(Reason==null) Reason="treeTimeoutMs"; return false; } if(Count>=Maximum) { if(Reason==null) Reason="maxNodes"; return false; } return true; }
+  }
+  static Dictionary<string,object> ReadTree(AutomationElement element,IntPtr window,int depth,int physicalDepth,Traversal state) {
+    if(!state.CanRead()) return null;
+    if(physicalDepth>128) { state.Reason="providerDepth"; return null; }
+    Dictionary<string,object> node;
+    try { node=DescribeElement(element,window,state.Full,depth); }
+    catch(ElementNotAvailableException) { state.Reason="providerError"; state.Error="An element disappeared during observation"; return null; }
+    state.Count++;
+    if(node.ContainsKey("controlTypeUnavailable")) state.UnknownControlTypes++;
+    List<object> children=new List<object>();
+    AutomationElement child=state.Walker.GetFirstChild(element);
+    while(child!=null && state.CanRead()) {
+      bool transparent=state.BrowserLayout && child.Current.ControlType==ControlType.Pane && String.IsNullOrEmpty(child.Current.Name);
+      int childDepth=depth+(transparent?0:1);
+      if(childDepth>state.Depth) { if(state.Reason==null) state.Reason="maxDepth"; }
+      else { Dictionary<string,object> item=ReadTree(child,window,childDepth,physicalDepth+1,state); if(item!=null) children.Add(item); }
+      if(state.Reason=="maxNodes" || state.Reason=="treeTimeoutMs") break;
+      child=state.Walker.GetNextSibling(child);
+    }
+    if(children.Count>0) node["children"]=children;
+    return node;
+  }
+  static TreeWalker ObservationWalker(string view,bool includeOffscreen) {
+    Condition condition=view=="raw"?Automation.RawViewCondition:view=="content"?Automation.ContentViewCondition:Automation.ControlViewCondition;
+    // Filter the projected view, rather than prune visible descendants of an offscreen ancestor.
+    if(!includeOffscreen)condition=new AndCondition(condition,new PropertyCondition(AutomationElement.IsOffscreenProperty,false));
+    return new TreeWalker(condition);
+  }
+  static Dictionary<string,object> WarmBrowser(IntPtr window,Traversal state) {
+    List<IntPtr> renderers=new List<IntPtr>();
+    EnumChildWindows(window,delegate(IntPtr child,IntPtr ignored) { StringBuilder name=new StringBuilder(256); GetClassName(child,name,name.Capacity); if(name.ToString()=="Chrome_RenderWidgetHostHWND") renderers.Add(child); return true; },IntPtr.Zero);
+    if(renderers.Count==0) return null;
+    int ready=0; long start=state.Watch.ElapsedMilliseconds; long budget=Math.Min(1500,state.Budget/2);
+    do {
+      ready=0;
+      foreach(IntPtr renderer in renderers) {
+        if(state.Watch.ElapsedMilliseconds>=state.Budget) break;
+        try { AutomationElement root=TreeWalker.RawViewWalker.GetFirstChild(AutomationElement.FromHandle(renderer)); if(root!=null) { ControlType type=root.Current.ControlType; if(TreeWalker.RawViewWalker.GetFirstChild(root)!=null) ready++; } } catch(ElementNotAvailableException) {} catch(InvalidOperationException) {}
+      }
+      if(ready==renderers.Count || state.Watch.ElapsedMilliseconds-start>=budget) break;
+      Thread.Sleep(50);
+    } while(true);
+    return new Dictionary<string,object>{{"renderers",renderers.Count},{"ready",ready},{"waitedMs",state.Watch.ElapsedMilliseconds-start},{"state",ready==renderers.Count ? "ready" : ready>0 ? "partial" : "starting"}};
+  }
+  static void FindMatches(Dictionary<string,object> node,string query,string control,List<object> matches,int maximum) {
+    if(node==null || matches.Count>=maximum) return;
+    string text=""; foreach(string field in new string[]{"name","automationId","className","value","controlType"}) if(node.ContainsKey(field)) text+=" "+Convert.ToString(node[field]);
+    if(text.IndexOf(query,StringComparison.OrdinalIgnoreCase)>=0 && (control.Length==0 || String.Equals(Convert.ToString(node["controlType"]),control,StringComparison.OrdinalIgnoreCase))) { Dictionary<string,object> match=new Dictionary<string,object>(node); match.Remove("children"); matches.Add(match); }
+    if(node.ContainsKey("children")) foreach(object child in (List<object>)node["children"]) { FindMatches((Dictionary<string,object>)child,query,control,matches,maximum); if(matches.Count>=maximum) break; }
+  }
+  static object ObserveAccessibility(string action, Dictionary<string,object> args) {
+    if(args.ContainsKey("windowRef") && TextOption(args,"scope","active_window")=="desktop")throw new InvalidOperationException("WINDOW_REFERENCE_CONFLICT: desktop observation cannot use a window reference");
+    PruneElements(); bool full=TextOption(args,"detailLevel","compact")=="full" || action=="find";
+    if(action=="element_info") {
+      IntPtr owner; AutomationElement element;
+      if(args.ContainsKey("elementId")) element=ResolveObservedElement(Convert.ToString(args["elementId"]),args,out owner);
+      else {element=AutomationElement.FromPoint(new System.Windows.Point(Convert.ToInt32(args["x"]),Convert.ToInt32(args["y"])));owner=ElementWindow(element,IntPtr.Zero);if(args.ContainsKey("windowRef") || args.ContainsKey("nativeWindowHandle") || args.ContainsKey("processId") || args.ContainsKey("windowTitle")){IntPtr target=ResolveObservationWindow(args);if(owner!=target)throw new InvalidOperationException("ELEMENT_WRONG_OWNER: point is outside target; observe again");}}
+      var infoMetadata=new Dictionary<string,object>();WindowReference infoBinding=ObserveWindowReference(owner,infoMetadata);if(infoBinding!=null)currentReference=infoBinding;
+      Dictionary<string,object> result=Result(true,action); result["element"]=DescribeElement(element,owner,full,0,infoBinding!=null);foreach(var pair in infoMetadata){result[pair.Key]=pair.Value;if(pair.Key=="windowRefUnavailable")((Dictionary<string,object>)result["element"])[pair.Key]=pair.Value;}if(infoBinding!=null)result["windowRef"]=infoBinding.Id; if(((Dictionary<string,object>)result["element"]).ContainsKey("controlTypeUnavailable")) result["conclusive"]=false; return result;
+    }
+    bool desktop=TextOption(args,"scope","active_window")=="desktop";
+    IntPtr window=desktop ? IntPtr.Zero : ResolveObservationWindow(args);
+    var observationMetadata=new Dictionary<string,object>();WindowReference observationBinding=ObserveWindowReference(window,observationMetadata);if(observationBinding!=null){currentReference=observationBinding;observationMetadata["windowRef"]=observationBinding.Id;}
+    if(window!=IntPtr.Zero){RememberPointerOrigin(window);RememberPointerTitle(args,window);}
+    bool includeOffscreen=Option(args,"includeOffscreen",false);
+    Traversal state=new Traversal { Budget=NumberOption(args,"treeTimeoutMs",5000),Maximum=NumberOption(args,"maxNodes",500),Depth=NumberOption(args,"maxDepth",6),Offscreen=includeOffscreen,Full=full,Walker=ObservationWalker(TextOption(args,"viewMode","control"),includeOffscreen) };
+    if(window!=IntPtr.Zero) { StringBuilder name=new StringBuilder(256); GetClassName(window,name,name.Capacity); state.BrowserLayout=name.ToString().StartsWith("Chrome_",StringComparison.Ordinal) || name.ToString().StartsWith("Mozilla",StringComparison.Ordinal); }
+    Dictionary<string,object> web=window==IntPtr.Zero ? null : WarmBrowser(window,state);
+    Dictionary<string,object> root=null;
+    try { AutomationElement element=desktop ? AutomationElement.RootElement : AutomationElement.FromHandle(window); root=ReadTree(element,window,0,0,state); }
+    catch(Exception error) { state.Reason="providerError"; state.Error=error.Message; }
+    Dictionary<string,object> output=Result(root!=null,action); output["scope"]=desktop?"desktop":"active_window"; output["tree"]=root; output["nodeCount"]=state.Count; output["truncated"]=state.Reason!=null || state.UnknownControlTypes>0; output["durationMs"]=state.Watch.ElapsedMilliseconds; output["treeTimeoutMs"]=state.Budget; output["viewMode"]=TextOption(args,"viewMode","control"); output["includeOffscreen"]=state.Offscreen; output["detailLevel"]=TextOption(args,"detailLevel","compact");
+    if(window!=IntPtr.Zero) {CheckWindowReference(window);foreach(var pair in observationMetadata)output[pair.Key]=pair.Value; output["nativeWindowHandle"]=window.ToInt64(); output["processId"]=WindowOwner(window); }
+    if(web!=null) output["webContent"]=web;
+    if(state.UnknownControlTypes>0) { output["unknownControlTypes"]=state.UnknownControlTypes; output["conclusive"]=false; output["truncationReason"]="controlTypeUnavailable"; }
+    if(state.Reason!=null) { output["truncationReason"]=state.Reason; output["conclusive"]=false; }
+    if(state.Error!=null) { output["providerError"]=state.Error; if(root==null) output["error"]="UIA_PROVIDER_ERROR: "+state.Error; }
+    if(action=="find") { List<object> matches=new List<object>(); FindMatches(root,TextOption(args,"query",""),TextOption(args,"controlType",""),matches,NumberOption(args,"maxResults",25)); output.Remove("tree"); output["matches"]=matches; output["query"]=TextOption(args,"query",""); if(matches.Count==0 && Option(args,"ocrFallback",true) && TextOption(args,"controlType","").Length==0 && (state.Count<=20 || state.Reason!=null || state.UnknownControlTypes>0)) output["ocrFallbackPending"]=true; }
+    return output;
+  }
+}

package/assets/browser-indicators.cs

@@ -0,0 +1,37 @@
+using System;
+using System.Collections.Generic;
+using System.Diagnostics;
+using System.Runtime.InteropServices;
+
+partial class WindowsBridge {
+  [DllImport("dwmapi.dll",EntryPoint="DwmGetWindowAttribute")] static extern int GetBrowserBorder(IntPtr window,int attribute,out uint value,int bytes);
+  [DllImport("dwmapi.dll")] static extern int DwmSetWindowAttribute(IntPtr window,int attribute,ref uint value,int bytes);
+  class BrowserBorder { public IntPtr Window;public uint Process,Original; }
+  static readonly Dictionary<long,BrowserBorder> browserBorders=new Dictionary<long,BrowserBorder>();
+  static object MarkBrowserFrame(Dictionary<string,object> args) {
+    uint owner=Convert.ToUInt32(args["processId"]);bool clear=Option(args,"clear",false);int applied=0,refused=0;
+    using(Process process=Process.GetProcessById((int)owner)) {
+      IntPtr identity=process.Handle;
+      if(process.HasExited)throw new InvalidOperationException("BROWSER_PROCESS_EXITED");
+      if(clear) {
+        foreach(BrowserBorder entry in new List<BrowserBorder>(browserBorders.Values))if(entry.Process==owner) {
+          if(IsWindow(entry.Window) && WindowOwner(entry.Window)==owner) {uint original=entry.Original;if(DwmSetWindowAttribute(entry.Window,34,ref original,4)==0)applied++;else refused++;}
+          browserBorders.Remove(entry.Window.ToInt64());
+        }
+      } else EnumWindows(delegate(IntPtr window,IntPtr unused) {
+        if(!IsWindowVisible(window) || WindowOwner(window)!=owner)return true;
+        uint original;BrowserBorder entry;
+        if(!browserBorders.TryGetValue(window.ToInt64(),out entry)) {
+          if(GetBrowserBorder(window,34,out original,4)!=0)original=0xFFFFFFFF;
+          entry=new BrowserBorder {Window=window,Process=owner,Original=original};
+        }
+        uint blue=0x00EB841C;
+        if(!process.HasExited && WindowOwner(window)==owner && DwmSetWindowAttribute(window,34,ref blue,4)==0) {browserBorders[window.ToInt64()]=entry;applied++;}else refused++;
+        return true;
+      },IntPtr.Zero);
+    }
+    Dictionary<string,object> result=Result(true,"browser_frame");result["applied"]=applied;result["refused"]=refused;
+    if(applied==0)result["warning"]="No owned visible browser frame accepted DWM border color; verify visual presentation manually";
+    return result;
+  }
+}