@agenthippo-ai/agenthippo 1.110.9 → 1.110.19

package/out/base-prompts/skills/desktop-automation/SKILL.md

@@ -1,166 +0,0 @@
----
-name: desktop-automation
-description: Drive desktop applications on Windows and macOS using accessibility APIs, keyboard input, screenshots, and coordinate fallback. Use when you need to control an already-open app window, collect visual evidence, debug UI state, or automate UI that browser-native tools cannot attach to.
----
-
-# Desktop Automation
-
-Use this skill when the task is about a real desktop window rather than a browser-native or code-only workflow.
-
-Prefer browser-native tools such as `agent-browser` for websites in a managed or CDP-enabled browser. Use this skill when the user cares about the active desktop app/window, when browser tools cannot attach, or when visual evidence from the actual desktop matters.
-
-## Control Ladder
-
-1. Use the platform accessibility tree first.
-2. Use keyboard shortcuts and text input for common app flows.
-3. Capture screenshots before and after important actions, defaulting to window-targeted capture by window ID/handle when available.
-4. Use coordinate clicks only after accessibility lookup fails or the app exposes poor metadata.
-5. Verify the outcome with screenshots plus an independent signal when possible.
-
-Independent signals can be a window title, URL, file existence, process state, port state, app log, clipboard value, or exported data.
-
-## Browser Defaults (When Using Desktop Automation)
-
-When automating an already-open desktop browser window:
-
-1. Open a **new tab by default** (`Ctrl+T` on Windows/Linux, `Cmd+T` on macOS) unless the user explicitly asks to reuse the current tab.
-2. Navigate to the requested site in the new tab.
-3. After the site loads, interact with **in-page controls** (search fields, buttons, menus) via accessibility first.
-4. If the user says to type in a site search/input box, entering text in the browser address bar does **not** satisfy the request.
-5. Capture before/after screenshots that show the page state at each key milestone.
-
-## Windows
-
-Use Windows UI Automation through PowerShell.
-
-Helper module:
-
-```powershell
-Set-ExecutionPolicy -Scope Process Bypass -Force
-Import-Module '<skill_dir>\scripts\windows-desktop-uia.psm1' -Force
-```
-
-General workflow:
-
-1. Find the top-level window by title or process.
-2. Bring it to the foreground.
-3. Enumerate descendants and inspect `Name`, `ControlType`, bounds, and supported patterns.
-4. Invoke controls by accessibility pattern when possible.
-5. Use `SendKeys` for standard shortcuts and text entry.
-6. Capture screenshots for proof and debugging, defaulting to window-targeted capture by window handle; use full-screen capture only when window-targeted capture is unavailable.
-
-Useful primitives:
-- `Get-WindowAutomationElementByName`
-- `Get-WindowDescendants`
-- `Find-WindowElementByName`
-- `Invoke-UIElement`
-- `Focus-ProcessWindow`
-- `Save-FullScreen`
-- `Click-ScreenPoint`
-
-## macOS
-
-Use Accessibility plus AppleScript/JXA and screenshots.
-
-Preflight:
-
-```bash
-osascript -e 'tell application "System Events" to get name of first process whose frontmost is true'
-```
-
-If this fails with an accessibility/privacy error, the terminal or IDE running the command needs permission in:
-
-```text
-System Settings > Privacy & Security > Accessibility
-```
-
-General workflow:
-
-1. Bring the app forward with AppleScript.
-2. Inspect visible windows and UI elements through `System Events`.
-3. Prefer menu items, buttons, text fields, and keyboard shortcuts over coordinates.
-4. Use `screencapture` before and after actions, defaulting to window-targeted capture by window id (`-l <windowid>`); use full-screen capture only when a reliable window id is unavailable.
-5. Use coordinate clicks only as a fallback, usually through AppleScript, `cliclick`, or another available local tool.
-
-Bring an app forward:
-
-```bash
-osascript -e 'tell application "Safari" to activate'
-```
-
-List windows:
-
-```bash
-osascript -e 'tell application "System Events" to tell process "Safari" to get name of windows'
-```
-
-Type text into the focused control:
-
-```bash
-osascript -e 'tell application "System Events" to keystroke "hello world"'
-```
-
-Press a shortcut:
-
-```bash
-osascript -e 'tell application "System Events" to keystroke "l" using command down'
-```
-
-Take a screenshot:
-
-```bash
-screencapture -x /tmp/desktop-before.png
-```
-
-## Screenshot Debugging
-
-When the accessibility tree and the visible screen disagree, pause and gather evidence:
-
-1. Capture a screenshot.
-2. Dump visible controls with names, types, bounds, and offscreen state.
-3. Compare the screenshot against the accessibility dump.
-4. Decide whether the problem is stale state, wrong focus, offscreen UI, a transient overlay, or missing accessibility metadata.
-
-Common interpretations:
-- Screenshot shows target, accessibility dump does not: use coordinate fallback, but keep looking for a nearby stable control.
-- Accessibility dump shows target, screenshot does not: refocus or wait; the tree may be stale or the element may be offscreen.
-- Both show target but action fails: try supported patterns first, then keyboard, then center-point click.
-- Text typed into the wrong place: restore focus, select the intended field, and paste literal text from the clipboard instead of sending special characters directly.
-
-## Coordinate Fallback
-
-Coordinates are temporary scaffolding. Derive them from a screenshot plus window/control bounds, then verify immediately.
-
-Rules:
-- click near the visual center of the target
-- avoid borders, splitters, scrollbars, and label edges
-- recompute coordinates after window movement, zoom changes, scrolling, or layout changes
-- save a before and after screenshot
-- replace the coordinate with an accessibility selector once you find a stable one
-
-## Recording
-
-Use recording for longer flows or when the user will review the interaction.
-
-Windows:
-
-```powershell
-ffmpeg -f gdigrab -framerate 15 -i desktop -c:v mpeg4 .\desktop-run.avi
-```
-
-macOS:
-
-```bash
-screencapture -v /tmp/desktop-run.mov
-```
-
-If OBS is already configured, it is fine to use OBS instead. Always keep screenshots for key states even when recording video.
-
-## App-Specific Recipes
-
-Keep the core skill general. Put product-specific tricks in references.
-
-Current reference:
-- `references/agenthippo-ui-recipes.md` documents AgentHippo IDE workflows, status menus, command palette behavior, and lessons from real Windows testing.
-
-Read that reference only when the task is specifically about AgentHippo IDE or when you need an example of a complete desktop automation workflow.

package/out/base-prompts/skills/desktop-automation/references/agenthippo-ui-recipes.md

@@ -1,441 +0,0 @@
-# AgentHippo Windows UI Recipes
-
-This reference captures the concrete tools, scripts, commands, and reliability tricks that were used during real Agent Anywhere and Spotlight validation on Windows.
-
-Use it with `windows-desktop-uia.psm1`.
-
-## Real IDE launch
-
-Launch the real IDE, not just the test host:
-
-```powershell
-D:\dev\agenthippo-vscode\scripts\code.bat
-```
-
-Compile before UI testing:
-
-```powershell
-npm run compile
-```
-
-Extension-only compile:
-
-```powershell
-C:\Users\trung\AppData\Roaming\nodejs\node.exe D:\dev\agenthippo-vscode\node_modules\typescript\lib\tsc.js -p D:\dev\agenthippo-vscode\extensions\agentide
-```
-
-## Import pattern
-
-Unsigned module imports need a per-process execution policy bypass:
-
-```powershell
-Set-ExecutionPolicy -Scope Process Bypass -Force
-Import-Module 'D:\dev\agenthippo-vscode\extensions\agentide\base-prompts\skills\desktop-automation\scripts\windows-desktop-uia.psm1' -Force
-```
-
-Without that bypass, local module import may fail before any UI interaction starts.
-
-## Core helper functions
-
-The helper module exposes these tested functions:
-
-- `Get-AgentHippoProcess`
-- `Get-AgentHippoWindow`
-- `Get-AgentHippoRect`
-- `Focus-AgentHippo`
-- `Save-FullScreen`
-- `Minimize-InterferingBrowsers`
-- `Get-AgentDescendants`
-- `Find-AgentElementByName`
-- `Invoke-AgentElement`
-- `Invoke-AgentButtonByName`
-- `Invoke-AgentStatusButton`
-- `Invoke-AgentCommand`
-- `Get-AgentAnywhereNotificationWindow`
-- `Invoke-AgentAnywhereNotificationButton`
-- `Click-ScreenPoint`
-
-## Proven command-palette flow
-
-This is the most reliable command pattern that worked in the live IDE:
-
-```powershell
-Invoke-AgentCommand -Query 'Agent Anywhere Status' -TargetName 'Agent Hippo: Agent Anywhere Status'
-```
-
-Other real command labels that appeared in the live IDE:
-
-- `Agent Hippo: Install Agent Anywhere (Gateway), recently used`
-- `Agent Hippo: Agent Anywhere Status`
-- `Agent Hippo: Connect Agent Anywhere`
-- `Agent Hippo: Uninstall Agent Anywhere (OpenClaw CLI)`
-- `Agent Hippo: Detect Agent Anywhere Gateway`
-- `Agent Hippo: Disconnect Agent Anywhere`
-
-## Robust manual command fallback
-
-When `Invoke-AgentCommand` misses, use this sequence instead of guessing:
-
-1. `Focus-AgentHippo`
-2. Send `{ESC}{ESC}`
-3. Send `^+p`
-4. Send `^a`
-5. Type `>query`
-6. Wait about `700-900ms`
-7. Enumerate `Get-AgentDescendants`
-8. Find a `ControlType.ListItem` whose name starts with the desired command label
-9. Invoke it with `Invoke-AgentElement`
-
-This worked better than blind Enter presses because the visible selection can lag behind the typed query.
-
-## Button and notification recipes
-
-### Agent Anywhere status
-
-Preferred pattern:
-
-1. Open status
-2. Inspect the notification window
-3. Click the exact named button
-
-Example:
-
-```powershell
-Invoke-AgentCommand -Query 'Agent Anywhere Status' -TargetName 'Agent Hippo: Agent Anywhere Status'
-Invoke-AgentAnywhereNotificationButton -Name 'Restart'
-```
-
-Buttons observed in live runs:
-
-When running:
-- `Web UI`
-- `Config`
-- `Restart`
-- `Reset auth`
-- `Uninstall`
-
-When stopped:
-- `Connect`
-- `Config`
-- `Uninstall`
-
-### Important safety note
-
-Do not click `Reset auth` on a live profile unless the user wants the gateway auth token rotated in `C:\Users\trung\.openclaw\openclaw.json`.
-
-## Typing into the AgentHippo UI
-
-Chat typing was successfully done with focused coordinate click plus `SendKeys`.
-
-Real example that worked:
-
-```powershell
-Focus-AgentHippo
-Click-ScreenPoint -X 1960 -Y 1190 -DelayMs 500
-[System.Windows.Forms.SendKeys]::SendWait('hello test')
-Start-Sleep -Milliseconds 700
-Save-FullScreen -Path 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\214-after-typing-local-chat.png'
-```
-
-Takeaways:
-- first get the app to the foreground
-- click only after the input area is visibly ready
-- capture a screenshot immediately after typing
-- prefer UIA focus if the input control is easy to identify; use coordinate clicks only when the input is visually obvious but UIA names are not stable
-
-New chat button examples:
-
-UIA route:
-
-```powershell
-Invoke-AgentButtonByName -Name 'New Chat (Ctrl+N)'
-```
-
-Coordinate fallback:
-
-```powershell
-Focus-AgentHippo
-Click-ScreenPoint -X 2009 -Y 125 -DelayMs 1500
-```
-
-## Screenshot practice
-
-Use full-screen captures at each important milestone:
-
-```powershell
-Save-FullScreen -Path 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\before.png'
-```
-
-Recommended sequence:
-- before action
-- immediately after action
-- after settle if state changes asynchronously
-
-Artifacts from the real runs live under:
-
-- Screens: `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens`
-- Alternate captures: `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screenshots`
-
-## Debugging UI state mismatches
-
-Use this when UIA says one thing but the live screen suggests something else.
-
-Examples from real work:
-- stale status card content after connect or restart
-- a visible notification that disappears between two scripts
-- focus landing in the wrong app after a browser steals foreground
-- a control that is clearly visible but has no usable UIA name
-
-Recommended workflow:
-
-1. Save a screenshot immediately.
-2. Dump the AgentHippo window bounds.
-3. Dump on-screen UIA elements with names and rectangles.
-4. Compare screenshot versus bounds.
-5. Only then choose the next action.
-
-### Window bounds probe
-
-This tells you where the live app window sits on the desktop:
-
-```powershell
-Focus-AgentHippo
-Get-AgentHippoRect | Format-List
-```
-
-The earlier ad hoc artifact script for this was:
-
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\rects.ps1`
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\uia-window.ps1`
-
-### On-screen UIA dump
-
-This is the most useful dump when you need to compare visible controls against the screenshot:
-
-```powershell
-Focus-AgentHippo
-$rect = Get-AgentHippoRect
-$all = Get-AgentDescendants
-$items = @()
-for ($i = 0; $i -lt $all.Count; $i++) {
-  $el = $all.Item($i)
-  $x = $el.Current.BoundingRectangle.Left
-  $y = $el.Current.BoundingRectangle.Top
-  if ($el.Current.Name -and $x -ge $rect.Left -and $x -le $rect.Right -and $y -ge $rect.Top -and $y -le $rect.Bottom) {
-    $items += [pscustomobject]@{
-      Name = $el.Current.Name
-      Type = $el.Current.ControlType.ProgrammaticName
-      X = $x
-      Y = $y
-      W = $el.Current.BoundingRectangle.Width
-      H = $el.Current.BoundingRectangle.Height
-      Offscreen = $el.Current.IsOffscreen
-    }
-  }
-}
-$items | Where-Object { $_.Type -in @('ControlType.Button','ControlType.ListItem','ControlType.Edit','ControlType.Text') } |
-  Sort-Object Y, X |
-  ConvertTo-Json -Depth 3 |
-  Set-Content 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\uia-elements-onscreen.json'
-```
-
-This was based on the real artifact script:
-
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-onscreen.ps1`
-
-### How to interpret a mismatch
-
-Use this decision table:
-
-- Screenshot shows the target, dump does not:
-  - likely custom-drawn control or poor accessibility metadata
-  - use coordinate fallback, but keep looking for a nearby accessible anchor
-
-- Dump shows the control, screenshot does not:
-  - likely stale window, wrong focus, or offscreen element
-  - refocus and recapture before clicking anything
-
-- Both show the control but the action fails:
-  - likely timing issue, transient overlay, or wrong click point
-  - slow down and click the center of the visible target
-
-- Screenshot and dump disagree on window location:
-  - likely multiple monitors or wrong foreground window
-  - re-run `Focus-AgentHippo`, then recapture
-
-## Deriving fallback click coordinates from screenshots
-
-Coordinate clicks were a last resort, but they were sometimes necessary. The key is to derive them deliberately.
-
-### Step-by-step method
-
-1. Bring AgentHippo to the foreground:
-
-```powershell
-Focus-AgentHippo
-```
-
-2. Capture a before screenshot:
-
-```powershell
-Save-FullScreen -Path 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\before-target.png'
-```
-
-3. Record the app bounds:
-
-```powershell
-$rect = Get-AgentHippoRect
-$rect | Format-List
-```
-
-4. Inspect the screenshot and note the target's approximate center relative to the app window.
-
-5. If useful, dump nearby accessible controls to get neighboring bounds:
-
-```powershell
-$all = Get-AgentDescendants
-$all | ForEach-Object {
-  if ($_.Current.Name) {
-    [pscustomobject]@{
-      Name = $_.Current.Name
-      Type = $_.Current.ControlType.ProgrammaticName
-      X = $_.Current.BoundingRectangle.Left
-      Y = $_.Current.BoundingRectangle.Top
-      W = $_.Current.BoundingRectangle.Width
-      H = $_.Current.BoundingRectangle.Height
-    }
-  }
-} | Sort-Object Y, X
-```
-
-6. Choose a click point near the visual center of the target.
-
-7. Click and capture immediately:
-
-```powershell
-Click-ScreenPoint -X 2009 -Y 125 -DelayMs 1500
-Save-FullScreen -Path 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\after-target-click.png'
-```
-
-### Specific lessons from real AgentHippo runs
-
-- For chat input, clicking low and centered in the input area worked better than clicking near placeholder text.
-- For title-bar or toolbar actions like `New Chat`, using the center of the icon/button area was more reliable than clicking on adjacent whitespace.
-- For ephemeral menus, the coordinate often had to be used in the same script that opened the menu.
-- If Chrome or Edge was visible, focus theft caused false coordinate failures; minimizing browsers first improved consistency.
-
-### Valuable artifact scripts
-
-These are worth reading when adapting the method:
-
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\new-chat-click.ps1`
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\type-local-chat.ps1`
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\rects.ps1`
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-onscreen.ps1`
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-interesting.ps1`
-
-### Coordinates are disposable
-
-Treat coordinates as:
-- useful for immediate progress
-- valid only for the current window size/layout/state
-- something to replace later with a better UIA target if possible
-
-Do not promote a coordinate click to the primary strategy unless repeated UIA inspection shows there is no reliable accessible target.
-
-## Screen recording
-
-Long UI flows were captured to:
-
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\video\agent-anywhere-e2e.avi`
-
-The existing recording logs show a working `ffmpeg` desktop capture using `gdigrab` at `15 fps` over the whole desktop. A good reusable command shape is:
-
-```powershell
-ffmpeg -f gdigrab -framerate 15 -i desktop -c:v mpeg4 D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\video\agent-anywhere-e2e.avi
-```
-
-Evidence for this comes from:
-
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\ffmpeg-stderr.log`
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\ffmpeg-stdout.log`
-
-If OBS is already configured and scriptable on the machine, it is fine to use OBS instead, but `ffmpeg + gdigrab` is the simpler fallback.
-
-## UIA inspection and discovery
-
-When a dialog is unfamiliar, dump the interesting descendants instead of guessing. This pattern was useful repeatedly:
-
-```powershell
-Focus-AgentHippo
-$all = Get-AgentDescendants
-$items = @()
-for ($i = 0; $i -lt $all.Count; $i++) {
-  $el = $all.Item($i)
-  if ($el.Current.Name) {
-    $items += [pscustomobject]@{
-      Name = $el.Current.Name
-      Type = $el.Current.ControlType.ProgrammaticName
-      X = $el.Current.BoundingRectangle.Left
-      Y = $el.Current.BoundingRectangle.Top
-      W = $el.Current.BoundingRectangle.Width
-      H = $el.Current.BoundingRectangle.Height
-      Offscreen = $el.Current.IsOffscreen
-    }
-  }
-}
-$items | Sort-Object Y, X | ConvertTo-Json -Depth 3 | Set-Content 'D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\uia-elements.json'
-```
-
-Real discovery scripts in the artifact folder:
-
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia.ps1`
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-interesting.ps1`
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-interesting-2.ps1`
-- `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\logs\dump-uia-onscreen.ps1`
-
-## Focus and timing tricks that mattered
-
-- Minimize browsers before fine-grained UI work:
-
-```powershell
-Minimize-InterferingBrowsers
-```
-
-- Many actions need a short settle delay:
-  - `150-300ms` after focus changes
-  - `700-900ms` after typing palette queries
-  - `1200-1800ms` after command invocation
-
-- If a notification vanishes quickly, open-and-click in one script instead of opening status in one step and clicking later.
-
-- If the command palette or dialog seems stale, send `{ESC}{ESC}` first.
-
-## Validation habits from the real Agent Anywhere work
-
-A script succeeding is not enough. Pair UI interaction with at least one independent check:
-
-- screenshot
-- port check
-- log tail
-- JSON status snapshot
-- browser window title
-
-Examples from the real runs:
-
-- Browser title check after `Web UI`:
-  - `OpenClaw Control - Google Chrome`
-- Agent Anywhere screenshots:
-  - `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\649-install-manualinvoke-final.png`
-  - `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\683-after-connect-hidden-launcher.png`
-  - `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\695-after-restart-combined.png`
-  - `D:\dev\agenthippo-vscode\.artifacts\agent-anywhere-e2e\screens\703-after-webui-combined.png`
-
-## Known live-workflow lessons
-
-1. Prefer UIA by label over coordinates.
-2. Use coordinates only as a bridge to discover a better target.
-3. Keep visual evidence for each important step.
-4. Refresh state before diagnosing a bug if the UI looks self-contradictory.
-5. Be careful with live-profile actions that rewrite auth or config.
-6. Build small temporary scripts in `.artifacts` during exploration, then fold the stable lessons back into the reusable skill.