@agenthippo-ai/agenthippo 0.1.0

package/out/base-prompts/skills/desktop-automation/SKILL.md

@@ -0,0 +1,166 @@
+---
+name: desktop-automation
+description: Drive desktop applications on Windows and macOS using accessibility APIs, keyboard input, screenshots, and coordinate fallback. Use when you need to control an already-open app window, collect visual evidence, debug UI state, or automate UI that browser-native tools cannot attach to.
+---
+
+# Desktop Automation
+
+Use this skill when the task is about a real desktop window rather than a browser-native or code-only workflow.
+
+Prefer browser-native tools such as `agent-browser` for websites in a managed or CDP-enabled browser. Use this skill when the user cares about the active desktop app/window, when browser tools cannot attach, or when visual evidence from the actual desktop matters.
+
+## Control Ladder
+
+1. Use the platform accessibility tree first.
+2. Use keyboard shortcuts and text input for common app flows.
+3. Capture screenshots before and after important actions, defaulting to window-targeted capture by window ID/handle when available.
+4. Use coordinate clicks only after accessibility lookup fails or the app exposes poor metadata.
+5. Verify the outcome with screenshots plus an independent signal when possible.
+
+Independent signals can be a window title, URL, file existence, process state, port state, app log, clipboard value, or exported data.
+
+## Browser Defaults (When Using Desktop Automation)
+
+When automating an already-open desktop browser window:
+
+1. Open a **new tab by default** (`Ctrl+T` on Windows/Linux, `Cmd+T` on macOS) unless the user explicitly asks to reuse the current tab.
+2. Navigate to the requested site in the new tab.
+3. After the site loads, interact with **in-page controls** (search fields, buttons, menus) via accessibility first.
+4. If the user says to type in a site search/input box, entering text in the browser address bar does **not** satisfy the request.
+5. Capture before/after screenshots that show the page state at each key milestone.
+
+## Windows
+
+Use Windows UI Automation through PowerShell.
+
+Helper module:
+
+```powershell
+Set-ExecutionPolicy -Scope Process Bypass -Force
+Import-Module '<skill_dir>\scripts\windows-desktop-uia.psm1' -Force
+```
+
+General workflow:
+
+1. Find the top-level window by title or process.
+2. Bring it to the foreground.
+3. Enumerate descendants and inspect `Name`, `ControlType`, bounds, and supported patterns.
+4. Invoke controls by accessibility pattern when possible.
+5. Use `SendKeys` for standard shortcuts and text entry.
+6. Capture screenshots for proof and debugging, defaulting to window-targeted capture by window handle; use full-screen capture only when window-targeted capture is unavailable.
+
+Useful primitives:
+- `Get-WindowAutomationElementByName`
+- `Get-WindowDescendants`
+- `Find-WindowElementByName`
+- `Invoke-UIElement`
+- `Focus-ProcessWindow`
+- `Save-FullScreen`
+- `Click-ScreenPoint`
+
+## macOS
+
+Use Accessibility plus AppleScript/JXA and screenshots.
+
+Preflight:
+
+```bash
+osascript -e 'tell application "System Events" to get name of first process whose frontmost is true'
+```
+
+If this fails with an accessibility/privacy error, the terminal or IDE running the command needs permission in:
+
+```text
+System Settings > Privacy & Security > Accessibility
+```
+
+General workflow:
+
+1. Bring the app forward with AppleScript.
+2. Inspect visible windows and UI elements through `System Events`.
+3. Prefer menu items, buttons, text fields, and keyboard shortcuts over coordinates.
+4. Use `screencapture` before and after actions, defaulting to window-targeted capture by window id (`-l <windowid>`); use full-screen capture only when a reliable window id is unavailable.
+5. Use coordinate clicks only as a fallback, usually through AppleScript, `cliclick`, or another available local tool.
+
+Bring an app forward:
+
+```bash
+osascript -e 'tell application "Safari" to activate'
+```
+
+List windows:
+
+```bash
+osascript -e 'tell application "System Events" to tell process "Safari" to get name of windows'
+```
+
+Type text into the focused control:
+
+```bash
+osascript -e 'tell application "System Events" to keystroke "hello world"'
+```
+
+Press a shortcut:
+
+```bash
+osascript -e 'tell application "System Events" to keystroke "l" using command down'
+```
+
+Take a screenshot:
+
+```bash
+screencapture -x /tmp/desktop-before.png
+```
+
+## Screenshot Debugging
+
+When the accessibility tree and the visible screen disagree, pause and gather evidence:
+
+1. Capture a screenshot.
+2. Dump visible controls with names, types, bounds, and offscreen state.
+3. Compare the screenshot against the accessibility dump.
+4. Decide whether the problem is stale state, wrong focus, offscreen UI, a transient overlay, or missing accessibility metadata.
+
+Common interpretations:
+- Screenshot shows target, accessibility dump does not: use coordinate fallback, but keep looking for a nearby stable control.
+- Accessibility dump shows target, screenshot does not: refocus or wait; the tree may be stale or the element may be offscreen.
+- Both show target but action fails: try supported patterns first, then keyboard, then center-point click.
+- Text typed into the wrong place: restore focus, select the intended field, and paste literal text from the clipboard instead of sending special characters directly.
+
+## Coordinate Fallback
+
+Coordinates are temporary scaffolding. Derive them from a screenshot plus window/control bounds, then verify immediately.
+
+Rules:
+- click near the visual center of the target
+- avoid borders, splitters, scrollbars, and label edges
+- recompute coordinates after window movement, zoom changes, scrolling, or layout changes
+- save a before and after screenshot
+- replace the coordinate with an accessibility selector once you find a stable one
+
+## Recording
+
+Use recording for longer flows or when the user will review the interaction.
+
+Windows:
+
+```powershell
+ffmpeg -f gdigrab -framerate 15 -i desktop -c:v mpeg4 .\desktop-run.avi
+```
+
+macOS:
+
+```bash
+screencapture -v /tmp/desktop-run.mov
+```
+
+If OBS is already configured, it is fine to use OBS instead. Always keep screenshots for key states even when recording video.
+
+## App-Specific Recipes
+
+Keep the core skill general. Put product-specific tricks in references.
+
+Current reference:
+- `references/agenthippo-ui-recipes.md` documents AgentHippo IDE workflows, status menus, command palette behavior, and lessons from real Windows testing.
+
+Read that reference only when the task is specifically about AgentHippo IDE or when you need an example of a complete desktop automation workflow.

package/out/base-prompts/skills/desktop-automation/references/agenthippo-ui-recipes.md

@@ -0,0 +1,441 @@
+# AgentHippo Windows UI Recipes
+
+This reference captures the concrete tools, scripts, commands, and reliability tricks that were used during real Agent Anywhere and Spotlight validation on Windows.
+
+Use it with `windows-desktop-uia.psm1`.
+
+## Real IDE launch
+
+Launch the real IDE, not just the test host:
+
+```powershell
+D:\dev\agenthippo-vscode\scripts\code.bat
+```
+
+Compile before UI testing:
+
+```powershell
+npm run compile
+```
+
+Extension-only compile:
+
+```powershell
+C:\Users\trung\AppData\Roaming\nodejs\node.exe D:\dev\agenthippo-vscode\node_modules\typescript\lib\tsc.js -p D:\dev\agenthippo-vscode\extensions\agentide
+```
+
+## Import pattern
+
+Unsigned module imports need a per-process execution policy bypass:
+
+```powershell
+Set-ExecutionPolicy -Scope Process Bypass -Force
+Import-Module 'D:\dev\agenthippo-vscode\extensions\agentide\base-prompts\skills\desktop-automation\scripts\windows-desktop-uia.psm1' -Force
+```
+
+Without that bypass, local module import may fail before any UI interaction starts.
+
+## Core helper functions
+
+The helper module exposes these tested functions:
+
+- `Get-AgentHippoProcess`
+- `Get-AgentHippoWindow`
+- `Get-AgentHippoRect`
+- `Focus-AgentHippo`
+- `Save-FullScreen`
+- `Minimize-InterferingBrowsers`
+- `Get-AgentDescendants`
+- `Find-AgentElementByName`
+- `Invoke-AgentElement`
+- `Invoke-AgentButtonByName`
+- `Invoke-AgentStatusButton`
+- `Invoke-AgentCommand`
+- `Get-AgentAnywhereNotificationWindow`
+- `Invoke-AgentAnywhereNotificationButton`
+- `Click-ScreenPoint`
+
+## Proven command-palette flow
+
+This is the most reliable command pattern that worked in the live IDE:
+
+```powershell
+Invoke-AgentCommand -Query 'Agent Anywhere Status' -TargetName 'Agent Hippo: Agent Anywhere Status'
+```
+
+Other real command labels that appeared in the live IDE:
+
+- `Agent Hippo: Install Agent Anywhere (Gateway), recently used`
+- `Agent Hippo: Agent Anywhere Status`
+- `Agent Hippo: Connect Agent Anywhere`
+- `Agent Hippo: Uninstall Agent Anywhere (OpenClaw CLI)`
+- `Agent Hippo: Detect Agent Anywhere Gateway`
+- `Agent Hippo: Disconnect Agent Anywhere`
+
+## Robust manual command fallback
+
+When `Invoke-AgentCommand` misses, use this sequence instead of guessing:
+
+1. `Focus-AgentHippo`
+2. Send `{ESC}{ESC}`
+3. Send `^+p`
+4. Send `^a`
+5. Type `>query`
+6. Wait about `700-900ms`
+7. Enumerate `Get-AgentDescendants`
+8. Find a `ControlType.ListItem` whose name starts with the desired command label
+9. Invoke it with `Invoke-AgentElement`
+
+This worked better than blind Enter presses because the visible selection can lag behind the typed query.
+
+## Button and notification recipes
+
+### Agent Anywhere status
+
+Preferred pattern:
+
+1. Open status
+2. Inspect the notification window
+3. Click the exact named button
+
+Example:
+
+```powershell
+Invoke-AgentCommand -Query 'Agent Anywhere Status' -TargetName 'Agent Hippo: Agent Anywhere Status'
+Invoke-AgentAnywhereNotificationButton -Name 'Restart'
+```
+
+Buttons observed in live runs:
+
+When running:
+- `Web UI`
+- `Config`
+- `Restart`
+- `Reset auth`
+- `Uninstall`
+
+When stopped:
+- `Connect`
+- `Config`
+- `Uninstall`
+
+### Important safety note
+
+Do not click `Reset auth` on a live profile unless the user wants the gateway auth token rotated in `C:\Users\trung\.openclaw\openclaw.json`.
+
+## Typing into the AgentHippo UI
+
+Chat typing was successfully done with focused coordinate click plus `SendKeys`.
+
+Real example that worked:
+
+```powershell
+Focus-AgentHippo
+Click-ScreenPoint -X 1960 -Y 1190 -DelayMs 500
+[System.Windows.Forms.SendKeys]::SendWait('hello test')
+Start-Sleep -Milliseconds 700
+Save-FullScreen -Path 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\214-after-typing-local-chat.png'
+```
+
+Takeaways:
+- first get the app to the foreground
+- click only after the input area is visibly ready
+- capture a screenshot immediately after typing
+- prefer UIA focus if the input control is easy to identify; use coordinate clicks only when the input is visually obvious but UIA names are not stable
+
+New chat button examples:
+
+UIA route:
+
+```powershell
+Invoke-AgentButtonByName -Name 'New Chat (Ctrl+N)'
+```
+
+Coordinate fallback:
+
+```powershell
+Focus-AgentHippo
+Click-ScreenPoint -X 2009 -Y 125 -DelayMs 1500
+```
+
+## Screenshot practice
+
+Use full-screen captures at each important milestone:
+
+```powershell
+Save-FullScreen -Path 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\before.png'
+```
+
+Recommended sequence:
+- before action
+- immediately after action
+- after settle if state changes asynchronously
+
+Artifacts from the real runs live under:
+
+- Screens: `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens`
+- Alternate captures: `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screenshots`
+
+## Debugging UI state mismatches
+
+Use this when UIA says one thing but the live screen suggests something else.
+
+Examples from real work:
+- stale status card content after connect or restart
+- a visible notification that disappears between two scripts
+- focus landing in the wrong app after a browser steals foreground
+- a control that is clearly visible but has no usable UIA name
+
+Recommended workflow:
+
+1. Save a screenshot immediately.
+2. Dump the AgentHippo window bounds.
+3. Dump on-screen UIA elements with names and rectangles.
+4. Compare screenshot versus bounds.
+5. Only then choose the next action.
+
+### Window bounds probe
+
+This tells you where the live app window sits on the desktop:
+
+```powershell
+Focus-AgentHippo
+Get-AgentHippoRect | Format-List
+```
+
+The earlier ad hoc artifact script for this was:
+
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\rects.ps1`
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\uia-window.ps1`
+
+### On-screen UIA dump
+
+This is the most useful dump when you need to compare visible controls against the screenshot:
+
+```powershell
+Focus-AgentHippo
+$rect = Get-AgentHippoRect
+$all = Get-AgentDescendants
+$items = @()
+for ($i = 0; $i -lt $all.Count; $i++) {
+  $el = $all.Item($i)
+  $x = $el.Current.BoundingRectangle.Left
+  $y = $el.Current.BoundingRectangle.Top
+  if ($el.Current.Name -and $x -ge $rect.Left -and $x -le $rect.Right -and $y -ge $rect.Top -and $y -le $rect.Bottom) {
+    $items += [pscustomobject]@{
+      Name = $el.Current.Name
+      Type = $el.Current.ControlType.ProgrammaticName
+      X = $x
+      Y = $y
+      W = $el.Current.BoundingRectangle.Width
+      H = $el.Current.BoundingRectangle.Height
+      Offscreen = $el.Current.IsOffscreen
+    }
+  }
+}
+$items | Where-Object { $_.Type -in @('ControlType.Button','ControlType.ListItem','ControlType.Edit','ControlType.Text') } |
+  Sort-Object Y, X |
+  ConvertTo-Json -Depth 3 |
+  Set-Content 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\uia-elements-onscreen.json'
+```
+
+This was based on the real artifact script:
+
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-onscreen.ps1`
+
+### How to interpret a mismatch
+
+Use this decision table:
+
+- Screenshot shows the target, dump does not:
+  - likely custom-drawn control or poor accessibility metadata
+  - use coordinate fallback, but keep looking for a nearby accessible anchor
+
+- Dump shows the control, screenshot does not:
+  - likely stale window, wrong focus, or offscreen element
+  - refocus and recapture before clicking anything
+
+- Both show the control but the action fails:
+  - likely timing issue, transient overlay, or wrong click point
+  - slow down and click the center of the visible target
+
+- Screenshot and dump disagree on window location:
+  - likely multiple monitors or wrong foreground window
+  - re-run `Focus-AgentHippo`, then recapture
+
+## Deriving fallback click coordinates from screenshots
+
+Coordinate clicks were a last resort, but they were sometimes necessary. The key is to derive them deliberately.
+
+### Step-by-step method
+
+1. Bring AgentHippo to the foreground:
+
+```powershell
+Focus-AgentHippo
+```
+
+2. Capture a before screenshot:
+
+```powershell
+Save-FullScreen -Path 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\before-target.png'
+```
+
+3. Record the app bounds:
+
+```powershell
+$rect = Get-AgentHippoRect
+$rect | Format-List
+```
+
+4. Inspect the screenshot and note the target's approximate center relative to the app window.
+
+5. If useful, dump nearby accessible controls to get neighboring bounds:
+
+```powershell
+$all = Get-AgentDescendants
+$all | ForEach-Object {
+  if ($_.Current.Name) {
+    [pscustomobject]@{
+      Name = $_.Current.Name
+      Type = $_.Current.ControlType.ProgrammaticName
+      X = $_.Current.BoundingRectangle.Left
+      Y = $_.Current.BoundingRectangle.Top
+      W = $_.Current.BoundingRectangle.Width
+      H = $_.Current.BoundingRectangle.Height
+    }
+  }
+} | Sort-Object Y, X
+```
+
+6. Choose a click point near the visual center of the target.
+
+7. Click and capture immediately:
+
+```powershell
+Click-ScreenPoint -X 2009 -Y 125 -DelayMs 1500
+Save-FullScreen -Path 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\after-target-click.png'
+```
+
+### Specific lessons from real AgentHippo runs
+
+- For chat input, clicking low and centered in the input area worked better than clicking near placeholder text.
+- For title-bar or toolbar actions like `New Chat`, using the center of the icon/button area was more reliable than clicking on adjacent whitespace.
+- For ephemeral menus, the coordinate often had to be used in the same script that opened the menu.
+- If Chrome or Edge was visible, focus theft caused false coordinate failures; minimizing browsers first improved consistency.
+
+### Valuable artifact scripts
+
+These are worth reading when adapting the method:
+
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\new-chat-click.ps1`
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\type-local-chat.ps1`
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\rects.ps1`
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-onscreen.ps1`
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-interesting.ps1`
+
+### Coordinates are disposable
+
+Treat coordinates as:
+- useful for immediate progress
+- valid only for the current window size/layout/state
+- something to replace later with a better UIA target if possible
+
+Do not promote a coordinate click to the primary strategy unless repeated UIA inspection shows there is no reliable accessible target.
+
+## Screen recording
+
+Long UI flows were captured to:
+
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\video\agent-anywhere-e2e.avi`
+
+The existing recording logs show a working `ffmpeg` desktop capture using `gdigrab` at `15 fps` over the whole desktop. A good reusable command shape is:
+
+```powershell
+ffmpeg -f gdigrab -framerate 15 -i desktop -c:v mpeg4 D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\video\agent-anywhere-e2e.avi
+```
+
+Evidence for this comes from:
+
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\ffmpeg-stderr.log`
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\ffmpeg-stdout.log`
+
+If OBS is already configured and scriptable on the machine, it is fine to use OBS instead, but `ffmpeg + gdigrab` is the simpler fallback.
+
+## UIA inspection and discovery
+
+When a dialog is unfamiliar, dump the interesting descendants instead of guessing. This pattern was useful repeatedly:
+
+```powershell
+Focus-AgentHippo
+$all = Get-AgentDescendants
+$items = @()
+for ($i = 0; $i -lt $all.Count; $i++) {
+  $el = $all.Item($i)
+  if ($el.Current.Name) {
+    $items += [pscustomobject]@{
+      Name = $el.Current.Name
+      Type = $el.Current.ControlType.ProgrammaticName
+      X = $el.Current.BoundingRectangle.Left
+      Y = $el.Current.BoundingRectangle.Top
+      W = $el.Current.BoundingRectangle.Width
+      H = $el.Current.BoundingRectangle.Height
+      Offscreen = $el.Current.IsOffscreen
+    }
+  }
+}
+$items | Sort-Object Y, X | ConvertTo-Json -Depth 3 | Set-Content 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\uia-elements.json'
+```
+
+Real discovery scripts in the artifact folder:
+
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia.ps1`
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-interesting.ps1`
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-interesting-2.ps1`
+- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-onscreen.ps1`
+
+## Focus and timing tricks that mattered
+
+- Minimize browsers before fine-grained UI work:
+
+```powershell
+Minimize-InterferingBrowsers
+```
+
+- Many actions need a short settle delay:
+  - `150-300ms` after focus changes
+  - `700-900ms` after typing palette queries
+  - `1200-1800ms` after command invocation
+
+- If a notification vanishes quickly, open-and-click in one script instead of opening status in one step and clicking later.
+
+- If the command palette or dialog seems stale, send `{ESC}{ESC}` first.
+
+## Validation habits from the real Agent Anywhere work
+
+A script succeeding is not enough. Pair UI interaction with at least one independent check:
+
+- screenshot
+- port check
+- log tail
+- JSON status snapshot
+- browser window title
+
+Examples from the real runs:
+
+- Browser title check after `Web UI`:
+  - `OpenClaw Control - Google Chrome`
+- Agent Anywhere screenshots:
+  - `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\649-install-manualinvoke-final.png`
+  - `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\683-after-connect-hidden-launcher.png`
+  - `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\695-after-restart-combined.png`
+  - `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\703-after-webui-combined.png`
+
+## Known live-workflow lessons
+
+1. Prefer UIA by label over coordinates.
+2. Use coordinates only as a bridge to discover a better target.
+3. Keep visual evidence for each important step.
+4. Refresh state before diagnosing a bug if the UI looks self-contradictory.
+5. Be careful with live-profile actions that rewrite auth or config.
+6. Build small temporary scripts in `.artifacts` during exploration, then fold the stable lessons back into the reusable skill.