mobridge-mcp 0.9.2 → 0.10.0

package/skills/mobridge-surfaces/SKILL.md

@@ -0,0 +1,106 @@
+---
+name: mobridge-surfaces
+description: Use when a screen has no usable nodes - Flutter, Compose, WebView, canvas, games, unlabeled icons or ads
+version: 1.0.0
+---
+
+# MoBridge UI surfaces
+
+Accessibility only sees what the app exposes. Different UI toolkits expose different amounts. Pick the playbook by
+what `observe` returned, not by guessing the framework.
+
+## When to use
+
+- `observe` returns few or no nodes, or only a single container.
+- A query gives `NODE_NOT_FOUND` for something clearly visible.
+- The result says `NEEDS_VISION`, or `treeEvidence.state` stays `UNCHANGED` after a visible change.
+- The target is an icon without a label, a game, a canvas, a video, or an ad.
+
+## Workflow
+
+1. Run `mobridge screen observe --actionable` and classify the surface with the table below.
+2. Semantic surface: act by query (apply `mobridge-core`). No usable nodes: capture, choose a point, `act at`.
+3. Verify with a new read; vision taps are never verified by the tree.
+4. After two misses on the same point stop and change approach (apply `mobridge-debugging`).
+
+## Pick the playbook
+
+| What you see in `observe` | Surface | Playbook |
+|---|---|---|
+| Nodes with `text`, `contentDescription`, `resourceId` | Native views / Compose with semantics | Query by text, label or resource id |
+| Compose, labels present but no `resourceId` | Compose | Query by text or label; scope with `--within` |
+| Flutter: nodes with labels only, ids absent | Flutter | Query by label; no resource ids exist |
+| One big `WebView` node, little inside | WebView | Use `inspect`, scroll the WebView, fall back to capture |
+| One node covering the screen | Canvas / game / custom view | Capture, then `act at` |
+| Icon buttons without text | Unlabeled icons | `capture --regions`, choose a region, `act at` |
+| Ad or video overlay | Ad / media | Capture, `act at` with `VISUAL_DYNAMIC` |
+
+## Semantic surfaces (native, Compose, Flutter)
+
+```bash
+mobridge screen observe --actionable
+mobridge screen act click --text "Continue" --verify auto
+mobridge screen act click --resource-id "com.example:id/save" --verify auto
+```
+
+- Prefer the most specific stable selector: resource id, then label, then text. Text changes with language.
+- Lists: scope with `--within` or scroll first. A match inside a nested scroll host needs `axis`
+  (`VERTICAL` / `HORIZONTAL`) or it fails with `AMBIGUOUS_MATCH`.
+- Flutter and Compose may rebuild nodes each frame. Re-resolve from the original query, never from an old NodeRef.
+- If the tree stays `UNCHANGED` after a visible change, the surface is not in the tree: switch to the vision path.
+
+## WebView
+
+- WebView content may be absent from the Accessibility tree. `treeEvidence` can say `UNCHANGED` even when the page
+  changed; do not read that as failure.
+- Scroll by gesture on the WebView node, then capture to read the result.
+- Use a short wait after navigation, then capture; do not trust a tree-based verify alone.
+
+```bash
+mobridge screen observe
+mobridge screen capture
+```
+
+## Vision path (canvas, games, unlabeled icons, ads)
+
+Rule: never guess coordinates. Every `act at` uses a point read from a fresh capture.
+
+1. Capture. The image stays in daemon memory (about 30 s) under a `captureId`.
+2. For an icon without a node, list neutral regions:
+
+```bash
+mobridge screen capture --regions
+```
+
+3. For a known colour: `mobridge screen capture --color "#FF0000"` returns matches. Only exactly one match can be
+   tapped through a pipe.
+4. Look at the image, choose the point, act on it:
+
+```bash
+mobridge screen act at --capture <captureId> --x 540 --y 1200
+```
+
+5. Verify with a new capture or `observe`. An old `captureId` fails with `STALE_CAPTURE`; capture again.
+
+Small text or icons: `mobridge screen capture --mode detail --roi x,y,w,h` returns native pixels for one region.
+Dynamic UI (video, ads, animation): the daemon applies `VISUAL_DYNAMIC` so a changing screen does not invalidate your
+point; keep the delay between capture and tap short.
+
+## Ads
+
+There is no ad command. An ad is just a surface: capture, find the close control (often a small corner icon),
+choose its point from the image, `act at`, then observe. If the ad is a full-screen interstitial in the app's own
+Activity, `app.open` may report the wrong screen; observe before continuing. Never loop on taps: after two misses stop
+and ask the person.
+
+## Blocked pixels
+
+`CAPTURE_BLOCKED_SECURE_WINDOW` means the app marks the window secure (banking, password fields, DRM). No capture is
+possible and none should be attempted by another route. Use semantic reads only, or ask the person to act.
+
+## Verification
+
+- After a vision tap, always verify with a new read; vision taps are `unverified` by design.
+- Keep your own count: if the same point failed twice, change approach instead of retrying.
+
+References: `references/tree-evidence.md`.

package/skills/mobridge-surfaces/references/tree-evidence.md

@@ -0,0 +1,26 @@
+# Tree evidence
+
+`observation.treeEvidence.state` compares the Accessibility tree before and after an action.
+
+| State | Meaning | What to do |
+|---|---|---|
+| `CHANGED` | The tree differs | Normal for navigation and edits |
+| `UNCHANGED` | The tree is the same | On a native screen the action probably did nothing. On WebView, canvas or games it proves nothing: capture. |
+| `UNKNOWN` | Not comparable | Observe, then decide |
+
+`screenChanged` is the same tree comparison. It is never a pixel comparison.
+
+## `stuck`
+
+`stuck {repeatCount, hint}` appears after the same action hit the same target three times with the tree unchanged.
+Stop, observe, and change approach: another selector, a scroll, a capture, or ask the person.
+
+## Partial changes
+
+A BACK that changes only part of the tree may still be a real navigation. Do not retry BACK blindly; use an explicit
+verify (a node that must appear) or tap the screen's own back control.
+
+## Surfaces that hide from the tree
+
+Canvas, games, video and parts of WebView are not in the tree. For them the only evidence is pixels: capture before
+and after, and compare what you see.