@bdayadev/flutter-ultra-mcp 0.0.0 → 1.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (58) hide show
  1. package/.claude-plugin/marketplace.json +35 -0
  2. package/.claude-plugin/plugin.json +14 -14
  3. package/.mcp.json +67 -67
  4. package/CODE_OF_CONDUCT.md +83 -83
  5. package/CONTRIBUTING.md +108 -108
  6. package/LICENSE +201 -201
  7. package/README.md +77 -77
  8. package/SECURITY.md +19 -19
  9. package/dart/ultra_flutter/CHANGELOG.md +34 -34
  10. package/dart/ultra_flutter/EXTENSIONS.md +61 -61
  11. package/dart/ultra_flutter/README.md +109 -109
  12. package/dart/ultra_flutter/pubspec.yaml +37 -37
  13. package/dart/ultra_flutter_devtools/README.md +7 -7
  14. package/dart/ultra_flutter_devtools/pubspec.yaml +27 -27
  15. package/docs/UPSTREAM-PATROL-PRS.md +5 -5
  16. package/docs/UPSTREAM-SENTRY-PR.md +62 -62
  17. package/docs/discovery-empirics.md +435 -435
  18. package/examples/counter-app/README.md +24 -24
  19. package/examples/counter-app/pubspec.yaml +23 -23
  20. package/examples/oidc-app/README.md +48 -48
  21. package/examples/oidc-app/pubspec.yaml +24 -24
  22. package/package.json +82 -82
  23. package/packages/flutter-ultra-browser/README.md +29 -29
  24. package/packages/flutter-ultra-browser/package.json +39 -39
  25. package/packages/flutter-ultra-build/README.md +60 -60
  26. package/packages/flutter-ultra-build/package.json +38 -38
  27. package/packages/flutter-ultra-devtools/README.md +7 -7
  28. package/packages/flutter-ultra-devtools/package.json +36 -36
  29. package/packages/flutter-ultra-gesture/README.md +51 -51
  30. package/packages/flutter-ultra-gesture/package.json +58 -58
  31. package/packages/flutter-ultra-native-desktop/README.md +131 -131
  32. package/packages/flutter-ultra-native-desktop/package.json +81 -81
  33. package/packages/flutter-ultra-native-mobile/README.md +103 -103
  34. package/packages/flutter-ultra-native-mobile/package.json +72 -72
  35. package/packages/flutter-ultra-patrol/README.md +40 -40
  36. package/packages/flutter-ultra-patrol/package.json +38 -38
  37. package/packages/flutter-ultra-runtime/README.md +63 -63
  38. package/packages/flutter-ultra-runtime/package.json +69 -69
  39. package/shared/contracts/package.json +31 -31
  40. package/shared/device-router/README.md +51 -51
  41. package/shared/device-router/package.json +62 -62
  42. package/shared/keyring/README.md +7 -7
  43. package/shared/keyring/package.json +24 -24
  44. package/shared/mcp-runtime/README.md +116 -116
  45. package/shared/mcp-runtime/package.json +58 -58
  46. package/shared/state-store/README.md +66 -66
  47. package/shared/state-store/package.json +60 -60
  48. package/shared/vm-service-client/README.md +135 -135
  49. package/shared/vm-service-client/package.json +62 -62
  50. package/skills/_internal-on-tool-failure/SKILL.md +13 -13
  51. package/skills/_internal-session-bootstrap/SKILL.md +18 -18
  52. package/skills/debug/SKILL.md +20 -20
  53. package/skills/devtools/SKILL.md +21 -21
  54. package/skills/drive/SKILL.md +20 -20
  55. package/skills/scaffold/SKILL.md +21 -21
  56. package/skills/setup/SKILL.md +26 -26
  57. package/skills/test/SKILL.md +19 -19
  58. package/skills/tour/SKILL.md +22 -22
@@ -1,131 +1,131 @@
1
- # @flutter-ultra/flutter-ultra-native-desktop
2
-
3
- Cross-platform MCP server for **native desktop UI automation** — drives the
4
- OS-level a11y tree on macOS (AXUIElement via a Swift sidecar), Windows (UIA
5
- via a FlaUI C# sidecar), and Linux (AT-SPI 2 via a PyGObject sidecar). Used
6
- by Claude Code skills to introspect dialogs, click buttons, type text, and
7
- capture window screenshots in apps that Flutter's VM service can't reach.
8
-
9
- ## Architecture
10
-
11
- A single platform sidecar process is spawned per MCP-server lifetime. On
12
- crash the cached entry is dropped and the next tool invocation respawns it.
13
- Tool handlers stay platform-agnostic — the `DesktopBackend` interface
14
- (`src/types.ts`) hides every OS-specific quirk. `src/index.ts` switches on
15
- `process.platform` and selects `MacDesktopBackend`, `WinDesktopBackend`, or
16
- `LinuxDesktopBackend` accordingly; the registry registers tools only when
17
- the backend reports `helperPresent && permissionGranted`.
18
-
19
- ## Tool surface (9 tools, plan §5.6)
20
-
21
- | Tool | Purpose |
22
- | ----------------------- | ------------------------------------------------------------------------ |
23
- | `list_windows` | Visible top-level windows, optionally filtered by process / title. |
24
- | `dump_window_tree` | Full a11y tree for a window, depth-bounded. |
25
- | `desktop_query` | XPath subset: `//role`, `//role[@name="X"]`, `//*[@label~="X"]`. |
26
- | `desktop_click` | Click by element id (preferred) or screen coordinates. |
27
- | `desktop_type` | Type text; optionally clear field first; optional element focus. |
28
- | `desktop_screenshot` | PNG (base64) — window or full screen. |
29
- | `select_file_in_dialog` | Type a path into the frontmost file dialog + click Open/Save. |
30
- | `confirm_dialog` | Click a dialog button by intent (allow/deny/ok/cancel/yes/no/open/save). |
31
- | `wait_for_window` | Poll for a window matching title regex / process; configurable timeout. |
32
-
33
- Zero tools register when the per-OS sidecar is missing or the a11y bus is
34
- unreachable (AC-ND4). Startup logs explain why so users can self-remediate.
35
-
36
- ## Linux path — AT-SPI via PyGObject
37
-
38
- The Linux backend invokes `python3 -u -m atspi_bridge` from the package's
39
- `sidecars/linux-atspi/` directory. The Python sidecar:
40
-
41
- - Wraps `gi.repository.Atspi` for window enumeration, accessible-tree
42
- introspection, and action invocation (`Action.do_action` for click,
43
- `EditableText.insert_text` for type).
44
- - Shells out to `grim` on Wayland or `scrot`/`import` on X11 for
45
- screenshots. Both branches required because Wayland sandboxing makes
46
- X11 screenshot APIs unusable.
47
- - Shells out to `xdotool` (X11) or `ydotool` (Wayland) for
48
- cursor-coordinate input synthesis when AT-SPI alone can't reach a
49
- widget (Flutter Linux desktop is the common case — its custom-painted
50
- widgets don't expose the EditableText interface).
51
-
52
- ### Distro support
53
-
54
- | Distro family | Install command |
55
- | ---------------------------- | ------------------------------------------------------------------ |
56
- | Debian / Ubuntu / Mint / Pop | `sudo apt-get install -y python3-gi gir1.2-atspi-2.0 at-spi2-core` |
57
- | Fedora / RHEL / Rocky / Alma | `sudo dnf install -y python3-gobject atspi at-spi2-core` |
58
- | Arch / Manjaro / EndeavourOS | `sudo pacman -S --needed python-gobject at-spi2-core` |
59
- | openSUSE (Leap & Tumbleweed) | `sudo zypper install -y python3-gobject typelib-1_0-Atspi-2_0` |
60
- | Alpine | `sudo apk add py3-gobject3 at-spi2-core` |
61
-
62
- Use the exported `detectLocalDistro()` helper to print the exact command
63
- for the running host.
64
-
65
- ### Wayland caveat
66
-
67
- AT-SPI works fully on **X11**. On **Wayland** the coverage varies by
68
- toolkit:
69
-
70
- | Toolkit | Coverage |
71
- | ---------------------------------------------- | --------------------------------------------------------------- |
72
- | GTK 3 / GTK 4 | Fully exposed |
73
- | Qt 5 / Qt 6 with `QT_ACCESSIBILITY=1` | Fully exposed |
74
- | Electron with `--force-renderer-accessibility` | Fully exposed |
75
- | Flutter Linux desktop | Active window only (flutter/flutter#107016) — use ultra_flutter |
76
-
77
- When `XDG_SESSION_TYPE=wayland` the backend sets `capabilities.waylandLimited
78
- = true` so the registry can present a one-shot warning. For Flutter Linux
79
- apps prefer the in-app `ultra_flutter` binding via
80
- `@flutter-ultra/flutter-ultra-gesture` / `@flutter-ultra/flutter-ultra-runtime`.
81
-
82
- ### Headless / minimal compositors
83
-
84
- Sway, river, and hyprland do NOT auto-spawn `at-spi-dbus-bus`. Enable it
85
- explicitly:
86
-
87
- ```bash
88
- systemctl --user enable --now at-spi-dbus-bus
89
- ```
90
-
91
- GNOME, KDE, XFCE, MATE, and Cinnamon auto-spawn it on session start.
92
-
93
- ## Device router placeholder
94
-
95
- The TS server consumes the `Device` interface from `src/device/types.ts`.
96
- Today only `LocalDevice` ships; SSH and WSL `Device` implementations land
97
- post-wave-3 in `@flutter-ultra/device-router`. The Python sidecar runs
98
- inside the target Linux environment (local host, WSL distro, or remote
99
- SSH-Linux); the TS server stays on the MCP host and pipes JSON-RPC across
100
- stdio. No backend code changes when the router lands — the platform
101
- switch in `src/index.ts` keeps using `new LocalDevice()` until the router
102
- swaps it for `routerDevice.select(deviceId)`.
103
-
104
- ## Development
105
-
106
- ```bash
107
- # Repo root
108
- npm ci
109
- npm run -w @flutter-ultra/flutter-ultra-native-desktop build
110
- npm run -w @flutter-ultra/flutter-ultra-native-desktop test
111
- npm run -w @flutter-ultra/flutter-ultra-native-desktop typecheck
112
-
113
- # Python sidecar tests (Linux only)
114
- cd packages/flutter-ultra-native-desktop/sidecars/linux-atspi
115
- sudo apt-get install -y python3-gi gir1.2-atspi-2.0 at-spi2-core
116
- pip install pytest
117
- PYTHONPATH=. pytest tests/
118
- ```
119
-
120
- ## CI
121
-
122
- | Workflow | Purpose |
123
- | ------------------------------------- | ------------------------------------------------------------------- |
124
- | `.github/workflows/ci.yml` | Cross-platform TS unit tests (ubuntu/macos/windows × Node 20/22) |
125
- | `.github/workflows/sidecar-macos.yml` | Build the Swift helper on macos-latest + TCC probe smoke |
126
- | `.github/workflows/sidecar-linux.yml` | Python sidecar pytest matrix + Xvfb-driven AT-SPI integration smoke |
127
-
128
- ## Status
129
-
130
- Wave 3 complete. macOS path merged via PR #17 (worker-J). Linux path
131
- shipped in this PR (worker-K). Windows path in flight (worker-I).
1
+ # @flutter-ultra/flutter-ultra-native-desktop
2
+
3
+ Cross-platform MCP server for **native desktop UI automation** — drives the
4
+ OS-level a11y tree on macOS (AXUIElement via a Swift sidecar), Windows (UIA
5
+ via a FlaUI C# sidecar), and Linux (AT-SPI 2 via a PyGObject sidecar). Used
6
+ by Claude Code skills to introspect dialogs, click buttons, type text, and
7
+ capture window screenshots in apps that Flutter's VM service can't reach.
8
+
9
+ ## Architecture
10
+
11
+ A single platform sidecar process is spawned per MCP-server lifetime. On
12
+ crash the cached entry is dropped and the next tool invocation respawns it.
13
+ Tool handlers stay platform-agnostic — the `DesktopBackend` interface
14
+ (`src/types.ts`) hides every OS-specific quirk. `src/index.ts` switches on
15
+ `process.platform` and selects `MacDesktopBackend`, `WinDesktopBackend`, or
16
+ `LinuxDesktopBackend` accordingly; the registry registers tools only when
17
+ the backend reports `helperPresent && permissionGranted`.
18
+
19
+ ## Tool surface (9 tools, plan §5.6)
20
+
21
+ | Tool | Purpose |
22
+ | ----------------------- | ------------------------------------------------------------------------ |
23
+ | `list_windows` | Visible top-level windows, optionally filtered by process / title. |
24
+ | `dump_window_tree` | Full a11y tree for a window, depth-bounded. |
25
+ | `desktop_query` | XPath subset: `//role`, `//role[@name="X"]`, `//*[@label~="X"]`. |
26
+ | `desktop_click` | Click by element id (preferred) or screen coordinates. |
27
+ | `desktop_type` | Type text; optionally clear field first; optional element focus. |
28
+ | `desktop_screenshot` | PNG (base64) — window or full screen. |
29
+ | `select_file_in_dialog` | Type a path into the frontmost file dialog + click Open/Save. |
30
+ | `confirm_dialog` | Click a dialog button by intent (allow/deny/ok/cancel/yes/no/open/save). |
31
+ | `wait_for_window` | Poll for a window matching title regex / process; configurable timeout. |
32
+
33
+ Zero tools register when the per-OS sidecar is missing or the a11y bus is
34
+ unreachable (AC-ND4). Startup logs explain why so users can self-remediate.
35
+
36
+ ## Linux path — AT-SPI via PyGObject
37
+
38
+ The Linux backend invokes `python3 -u -m atspi_bridge` from the package's
39
+ `sidecars/linux-atspi/` directory. The Python sidecar:
40
+
41
+ - Wraps `gi.repository.Atspi` for window enumeration, accessible-tree
42
+ introspection, and action invocation (`Action.do_action` for click,
43
+ `EditableText.insert_text` for type).
44
+ - Shells out to `grim` on Wayland or `scrot`/`import` on X11 for
45
+ screenshots. Both branches required because Wayland sandboxing makes
46
+ X11 screenshot APIs unusable.
47
+ - Shells out to `xdotool` (X11) or `ydotool` (Wayland) for
48
+ cursor-coordinate input synthesis when AT-SPI alone can't reach a
49
+ widget (Flutter Linux desktop is the common case — its custom-painted
50
+ widgets don't expose the EditableText interface).
51
+
52
+ ### Distro support
53
+
54
+ | Distro family | Install command |
55
+ | ---------------------------- | ------------------------------------------------------------------ |
56
+ | Debian / Ubuntu / Mint / Pop | `sudo apt-get install -y python3-gi gir1.2-atspi-2.0 at-spi2-core` |
57
+ | Fedora / RHEL / Rocky / Alma | `sudo dnf install -y python3-gobject atspi at-spi2-core` |
58
+ | Arch / Manjaro / EndeavourOS | `sudo pacman -S --needed python-gobject at-spi2-core` |
59
+ | openSUSE (Leap & Tumbleweed) | `sudo zypper install -y python3-gobject typelib-1_0-Atspi-2_0` |
60
+ | Alpine | `sudo apk add py3-gobject3 at-spi2-core` |
61
+
62
+ Use the exported `detectLocalDistro()` helper to print the exact command
63
+ for the running host.
64
+
65
+ ### Wayland caveat
66
+
67
+ AT-SPI works fully on **X11**. On **Wayland** the coverage varies by
68
+ toolkit:
69
+
70
+ | Toolkit | Coverage |
71
+ | ---------------------------------------------- | --------------------------------------------------------------- |
72
+ | GTK 3 / GTK 4 | Fully exposed |
73
+ | Qt 5 / Qt 6 with `QT_ACCESSIBILITY=1` | Fully exposed |
74
+ | Electron with `--force-renderer-accessibility` | Fully exposed |
75
+ | Flutter Linux desktop | Active window only (flutter/flutter#107016) — use ultra_flutter |
76
+
77
+ When `XDG_SESSION_TYPE=wayland` the backend sets `capabilities.waylandLimited
78
+ = true` so the registry can present a one-shot warning. For Flutter Linux
79
+ apps prefer the in-app `ultra_flutter` binding via
80
+ `@flutter-ultra/flutter-ultra-gesture` / `@flutter-ultra/flutter-ultra-runtime`.
81
+
82
+ ### Headless / minimal compositors
83
+
84
+ Sway, river, and hyprland do NOT auto-spawn `at-spi-dbus-bus`. Enable it
85
+ explicitly:
86
+
87
+ ```bash
88
+ systemctl --user enable --now at-spi-dbus-bus
89
+ ```
90
+
91
+ GNOME, KDE, XFCE, MATE, and Cinnamon auto-spawn it on session start.
92
+
93
+ ## Device router placeholder
94
+
95
+ The TS server consumes the `Device` interface from `src/device/types.ts`.
96
+ Today only `LocalDevice` ships; SSH and WSL `Device` implementations land
97
+ post-wave-3 in `@flutter-ultra/device-router`. The Python sidecar runs
98
+ inside the target Linux environment (local host, WSL distro, or remote
99
+ SSH-Linux); the TS server stays on the MCP host and pipes JSON-RPC across
100
+ stdio. No backend code changes when the router lands — the platform
101
+ switch in `src/index.ts` keeps using `new LocalDevice()` until the router
102
+ swaps it for `routerDevice.select(deviceId)`.
103
+
104
+ ## Development
105
+
106
+ ```bash
107
+ # Repo root
108
+ npm ci
109
+ npm run -w @flutter-ultra/flutter-ultra-native-desktop build
110
+ npm run -w @flutter-ultra/flutter-ultra-native-desktop test
111
+ npm run -w @flutter-ultra/flutter-ultra-native-desktop typecheck
112
+
113
+ # Python sidecar tests (Linux only)
114
+ cd packages/flutter-ultra-native-desktop/sidecars/linux-atspi
115
+ sudo apt-get install -y python3-gi gir1.2-atspi-2.0 at-spi2-core
116
+ pip install pytest
117
+ PYTHONPATH=. pytest tests/
118
+ ```
119
+
120
+ ## CI
121
+
122
+ | Workflow | Purpose |
123
+ | ------------------------------------- | ------------------------------------------------------------------- |
124
+ | `.github/workflows/ci.yml` | Cross-platform TS unit tests (ubuntu/macos/windows × Node 20/22) |
125
+ | `.github/workflows/sidecar-macos.yml` | Build the Swift helper on macos-latest + TCC probe smoke |
126
+ | `.github/workflows/sidecar-linux.yml` | Python sidecar pytest matrix + Xvfb-driven AT-SPI integration smoke |
127
+
128
+ ## Status
129
+
130
+ Wave 3 complete. macOS path merged via PR #17 (worker-J). Linux path
131
+ shipped in this PR (worker-K). Windows path in flight (worker-I).
@@ -1,81 +1,81 @@
1
- {
2
- "name": "@flutter-ultra/flutter-ultra-native-desktop",
3
- "version": "0.0.1",
4
- "private": false,
5
- "description": "MCP server for native desktop UIs: Windows UIA (FlaUI sidecar), macOS AXUIElement (Swift sidecar + TCC UX), Linux AT-SPI (PyGObject sidecar). Cross-OS feature detection registers ZERO tools per OS path when its sidecar is unavailable.",
6
- "license": "Apache-2.0",
7
- "homepage": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/tree/main/packages/flutter-ultra-native-desktop",
8
- "repository": {
9
- "type": "git",
10
- "url": "git+https://github.com/Bdaya-Dev/flutter-ultra-mcp.git",
11
- "directory": "packages/flutter-ultra-native-desktop"
12
- },
13
- "bugs": {
14
- "url": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/issues"
15
- },
16
- "keywords": [
17
- "mcp",
18
- "desktop",
19
- "axuielement",
20
- "accessibility",
21
- "macos",
22
- "tcc",
23
- "ui-automation",
24
- "at-spi",
25
- "atspi",
26
- "linux",
27
- "pygobject"
28
- ],
29
- "type": "module",
30
- "main": "dist/index.js",
31
- "module": "dist/index.js",
32
- "types": "dist/index.d.ts",
33
- "bin": {
34
- "flutter-ultra-native-desktop": "dist/bin.js"
35
- },
36
- "exports": {
37
- ".": {
38
- "types": "./dist/index.d.ts",
39
- "import": "./dist/index.js"
40
- }
41
- },
42
- "sideEffects": false,
43
- "files": [
44
- "dist",
45
- "README.md",
46
- "sidecars/macos-swift/Package.swift",
47
- "sidecars/macos-swift/Sources",
48
- "sidecars/macos-swift/bin",
49
- "sidecars/macos-swift/README.md",
50
- "sidecars/windows-flaui/FlutterUltraWinHelper",
51
- "sidecars/windows-flaui/README.md",
52
- "sidecars/linux-atspi/atspi_bridge",
53
- "sidecars/linux-atspi/pyproject.toml",
54
- "sidecars/linux-atspi/requirements.txt",
55
- "sidecars/linux-atspi/README.md"
56
- ],
57
- "scripts": {
58
- "build": "tsc -b",
59
- "build:sidecar:windows": "dotnet publish sidecars/windows-flaui/FlutterUltraWinHelper/FlutterUltraWinHelper.csproj -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true -p:PublishTrimmed=false -o sidecars/windows-flaui/bin",
60
- "lint": "eslint src",
61
- "test": "vitest run",
62
- "typecheck": "tsc -b --noEmit",
63
- "clean": "rimraf dist .turbo *.tsbuildinfo sidecars/windows-flaui/bin sidecars/windows-flaui/FlutterUltraWinHelper/bin sidecars/windows-flaui/FlutterUltraWinHelper/obj"
64
- },
65
- "dependencies": {
66
- "@flutter-ultra/mcp-runtime": "^0.0.1",
67
- "@modelcontextprotocol/sdk": "^1.29.0",
68
- "zod": "^3.23.8"
69
- },
70
- "devDependencies": {
71
- "rimraf": "^6.0.0",
72
- "typescript": "^5.6.0",
73
- "vitest": "^2.1.0"
74
- },
75
- "engines": {
76
- "node": ">=20"
77
- },
78
- "publishConfig": {
79
- "access": "public"
80
- }
81
- }
1
+ {
2
+ "name": "@flutter-ultra/flutter-ultra-native-desktop",
3
+ "version": "0.0.1",
4
+ "private": true,
5
+ "description": "MCP server for native desktop UIs: Windows UIA (FlaUI sidecar), macOS AXUIElement (Swift sidecar + TCC UX), Linux AT-SPI (PyGObject sidecar). Cross-OS feature detection registers ZERO tools per OS path when its sidecar is unavailable.",
6
+ "license": "Apache-2.0",
7
+ "homepage": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/tree/main/packages/flutter-ultra-native-desktop",
8
+ "repository": {
9
+ "type": "git",
10
+ "url": "git+https://github.com/Bdaya-Dev/flutter-ultra-mcp.git",
11
+ "directory": "packages/flutter-ultra-native-desktop"
12
+ },
13
+ "bugs": {
14
+ "url": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/issues"
15
+ },
16
+ "keywords": [
17
+ "mcp",
18
+ "desktop",
19
+ "axuielement",
20
+ "accessibility",
21
+ "macos",
22
+ "tcc",
23
+ "ui-automation",
24
+ "at-spi",
25
+ "atspi",
26
+ "linux",
27
+ "pygobject"
28
+ ],
29
+ "type": "module",
30
+ "main": "dist/index.js",
31
+ "module": "dist/index.js",
32
+ "types": "dist/index.d.ts",
33
+ "bin": {
34
+ "flutter-ultra-native-desktop": "dist/bin.js"
35
+ },
36
+ "exports": {
37
+ ".": {
38
+ "types": "./dist/index.d.ts",
39
+ "import": "./dist/index.js"
40
+ }
41
+ },
42
+ "sideEffects": false,
43
+ "files": [
44
+ "dist",
45
+ "README.md",
46
+ "sidecars/macos-swift/Package.swift",
47
+ "sidecars/macos-swift/Sources",
48
+ "sidecars/macos-swift/bin",
49
+ "sidecars/macos-swift/README.md",
50
+ "sidecars/windows-flaui/FlutterUltraWinHelper",
51
+ "sidecars/windows-flaui/README.md",
52
+ "sidecars/linux-atspi/atspi_bridge",
53
+ "sidecars/linux-atspi/pyproject.toml",
54
+ "sidecars/linux-atspi/requirements.txt",
55
+ "sidecars/linux-atspi/README.md"
56
+ ],
57
+ "scripts": {
58
+ "build": "tsc -b",
59
+ "build:sidecar:windows": "dotnet publish sidecars/windows-flaui/FlutterUltraWinHelper/FlutterUltraWinHelper.csproj -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true -p:PublishTrimmed=false -o sidecars/windows-flaui/bin",
60
+ "lint": "eslint src",
61
+ "test": "vitest run",
62
+ "typecheck": "tsc -b --noEmit",
63
+ "clean": "rimraf dist .turbo *.tsbuildinfo sidecars/windows-flaui/bin sidecars/windows-flaui/FlutterUltraWinHelper/bin sidecars/windows-flaui/FlutterUltraWinHelper/obj"
64
+ },
65
+ "dependencies": {
66
+ "@flutter-ultra/mcp-runtime": "^0.0.1",
67
+ "@modelcontextprotocol/sdk": "^1.29.0",
68
+ "zod": "^3.23.8"
69
+ },
70
+ "devDependencies": {
71
+ "rimraf": "^6.0.0",
72
+ "typescript": "^5.6.0",
73
+ "vitest": "^2.1.0"
74
+ },
75
+ "engines": {
76
+ "node": ">=20"
77
+ },
78
+ "publishConfig": {
79
+ "access": "public"
80
+ }
81
+ }
@@ -1,103 +1,103 @@
1
- # `@flutter-ultra/flutter-ultra-native-mobile`
2
-
3
- MCP server for **native mobile overlay automation**: Android UIAutomator via `adb`, iOS XCUITest via `xcrun simctl` (simulators, Mac only) and `go-ios` (physical devices). Owns the **Chrome Custom Tabs / SafariViewController OAuth** problem with the `solve_oauth_cct` composite tool — see [plan §5.5.1](../../docs/) for the full design.
4
-
5
- Part of the **Flutter Ultra MCP** plugin for Claude Code. The server runs as one of the eight MCP processes registered in `.mcp.json`.
6
-
7
- ## Tool catalogue
8
-
9
- | Tool | Class | Purpose |
10
- | ------------------------------------------------------------- | ----- | ------------------------------------------------------- |
11
- | `list_devices` | quick | Enumerate Android + iOS (sim + physical) devices |
12
- | `dump_a11y_tree` | long | Dump UIAutomator XML / XCUITest a11y as structured tree |
13
- | `wait_for_native_element` | long | Poll the a11y tree until a finder matches |
14
- | `native_tap` | quick | Tap by coordinates or finder (bounds-center) |
15
- | `native_type` | quick | Type into focused input |
16
- | `native_swipe` | quick | Swipe between two coordinates |
17
- | `native_back` / `_home` / `_app_switch` / `_open_settings` | quick | System keys / intents |
18
- | `native_pin_lock` | quick | Toggle Android lock-task (kiosk mode) |
19
- | `dismiss_permission_dialog` | quick | Smart allow/deny on runtime permission prompt |
20
- | `native_permission_grant` / `_deny` | quick | `pm grant` / `pm revoke` |
21
- | `take_device_screenshot` | quick | Base64 PNG via image content |
22
- | `set_device_orientation` | quick | Portrait / landscape |
23
- | `native_clipboard_set` / `_get` | quick | Device clipboard |
24
- | `start_device_logs` / `poll_device_logs` / `stop_device_logs` | quick | Split-tool logcat / oslog tail |
25
- | `solve_oauth_cct` | long | CCT/SVC OAuth via Playwright + deep-link dispatch |
26
-
27
- All long-running tools are wrapped by the shared `runWithWatchdog` so the 60-second MCP client ceiling never kills them.
28
-
29
- ## Architecture
30
-
31
- Every tool routes shell commands through a `Device` abstraction:
32
-
33
- ```text
34
- Tool handler ─► registry.get(deviceId) ─► AndroidDevice ─► adb -s <udid> ...
35
- ─► IosSimDevice ─► xcrun simctl ...
36
- ─► IosPhysicalDev ─► go-ios ...
37
- ```
38
-
39
- `AndroidDevice` / `IosSimDevice` / `IosPhysicalDevice` all implement the same `DeviceTransport` interface (`shell()`, `upload()`, `download()`, `isAlive()`, `dispose()`). A future remote-device adapter (WSL, SSH) drops in by implementing the same interface — no tool code changes.
40
-
41
- ```mermaid
42
- flowchart LR
43
- A[Tool handler] --> R[DeviceRegistry]
44
- R -->|cached| D[DeviceTransport]
45
- D --> Adb[AndroidDevice<br/>adb -s udid]
46
- D --> Sim[IosSimDevice<br/>xcrun simctl]
47
- D --> Phy[IosPhysicalDevice<br/>go-ios]
48
- D -.future.-> Ssh[SshDevice<br/>ssh user@host adb ...]
49
- D -.future.-> Wsl[WslDevice<br/>wsl -- adb ...]
50
- ```
51
-
52
- ## CCT OAuth bypass — `solve_oauth_cct`
53
-
54
- Chrome Custom Tabs and SafariViewController hide the OAuth flow behind a system-owned web view that no UI driver can introspect. We do **not** try to drive CCT directly. Instead:
55
-
56
- 1. Launch Playwright Chromium with the provider's `authorize` URL.
57
- 2. Submit credentials (optional `fillFlow`) — pause for human MFA when needed.
58
- 3. Intercept the redirect to the app scheme (`com.example.myapp://callback?code=…`) before Chromium drops it as `ERR_UNKNOWN_URL_SCHEME`.
59
- 4. Deliver the same URL into the app via `Device.shell()`:
60
- - Android: `adb shell am start -W -a android.intent.action.VIEW -d <url>`
61
- - iOS Sim: `xcrun simctl openurl <udid> <url>`
62
- - iOS physical: requires `idb` (out of scope for this server today).
63
-
64
- The Flutter app's deep-link handler cannot distinguish the dispatched intent from a real CCT close — the same `Intent.ACTION_VIEW` arrives with the same URL. PKCE state stays valid because the app initiated the flow (its `code_verifier` is in its own memory).
65
-
66
- ## Configuration
67
-
68
- | Env var | Default | Purpose |
69
- | ----------------------------------------- | ------------ | --------------------------------------- |
70
- | `FLUTTER_ULTRA_ADB` | `adb` | Path to the `adb` binary |
71
- | `FLUTTER_ULTRA_XCRUN` | `xcrun` | Path to `xcrun` |
72
- | `FLUTTER_ULTRA_GO_IOS_BIN` | `ios` | Path to the go-ios `ios` CLI |
73
- | `FLUTTER_ULTRA_TOOL_TIMEOUT_<NAME_UPPER>` | tool default | Override per-tool watchdog ceiling (ms) |
74
- | `FLUTTER_ULTRA_LOG_LEVEL` | `info` | `debug` / `info` / `warn` / `error` |
75
-
76
- ## Platform support
77
-
78
- | Tool | Linux | macOS | Windows |
79
- | ------------------------------------ | ----------------- | ----- | ------------------------------ |
80
- | Android tools | yes (adb on PATH) | yes | yes |
81
- | iOS Simulator tools | no | yes | no |
82
- | iOS physical (go-ios) | yes | yes | yes (depends on go-ios binary) |
83
- | `solve_oauth_cct` (Android dispatch) | yes | yes | yes |
84
- | `solve_oauth_cct` (iOS sim dispatch) | no | yes | no |
85
-
86
- iOS tools called on non-darwin hosts return a structured "unsupported" payload (per AC-NM3) — they do not crash the server.
87
-
88
- ## Acceptance criteria
89
-
90
- - **AC-NM1** — `dismiss_permission_dialog(intent='allow')` clears a runtime permission within 3s on the Android emulator.
91
- - **AC-NM2** — On iOS Simulator (Mac), `native_tap` on a SafariViewController OAuth login button completes the flow combined with `wait_for_url`.
92
- - **AC-NM3** — iOS tools cleanly return `unsupported` on win32 / linux.
93
- - **AC-NM4** — `solve_oauth_cct` end-to-end completes an OIDC OAuth flow against your identity provider on Android emulator within 60s.
94
-
95
- ## Development
96
-
97
- ```bash
98
- npm install # at repo root
99
- npm run -w @flutter-ultra/flutter-ultra-native-mobile build
100
- npm run -w @flutter-ultra/flutter-ultra-native-mobile test
101
- ```
102
-
103
- The unit-test suite focuses on the host-side parsers (`adb devices -l`, UIAutomator XML, simctl JSON) and the `Device` abstraction — full device-loop tests require an attached Android emulator and live in the cross-platform e2e workflow.
1
+ # `@flutter-ultra/flutter-ultra-native-mobile`
2
+
3
+ MCP server for **native mobile overlay automation**: Android UIAutomator via `adb`, iOS XCUITest via `xcrun simctl` (simulators, Mac only) and `go-ios` (physical devices). Owns the **Chrome Custom Tabs / SafariViewController OAuth** problem with the `solve_oauth_cct` composite tool — see [plan §5.5.1](../../docs/) for the full design.
4
+
5
+ Part of the **Flutter Ultra MCP** plugin for Claude Code. The server runs as one of the eight MCP processes registered in `.mcp.json`.
6
+
7
+ ## Tool catalogue
8
+
9
+ | Tool | Class | Purpose |
10
+ | ------------------------------------------------------------- | ----- | ------------------------------------------------------- |
11
+ | `list_devices` | quick | Enumerate Android + iOS (sim + physical) devices |
12
+ | `dump_a11y_tree` | long | Dump UIAutomator XML / XCUITest a11y as structured tree |
13
+ | `wait_for_native_element` | long | Poll the a11y tree until a finder matches |
14
+ | `native_tap` | quick | Tap by coordinates or finder (bounds-center) |
15
+ | `native_type` | quick | Type into focused input |
16
+ | `native_swipe` | quick | Swipe between two coordinates |
17
+ | `native_back` / `_home` / `_app_switch` / `_open_settings` | quick | System keys / intents |
18
+ | `native_pin_lock` | quick | Toggle Android lock-task (kiosk mode) |
19
+ | `dismiss_permission_dialog` | quick | Smart allow/deny on runtime permission prompt |
20
+ | `native_permission_grant` / `_deny` | quick | `pm grant` / `pm revoke` |
21
+ | `take_device_screenshot` | quick | Base64 PNG via image content |
22
+ | `set_device_orientation` | quick | Portrait / landscape |
23
+ | `native_clipboard_set` / `_get` | quick | Device clipboard |
24
+ | `start_device_logs` / `poll_device_logs` / `stop_device_logs` | quick | Split-tool logcat / oslog tail |
25
+ | `solve_oauth_cct` | long | CCT/SVC OAuth via Playwright + deep-link dispatch |
26
+
27
+ All long-running tools are wrapped by the shared `runWithWatchdog` so the 60-second MCP client ceiling never kills them.
28
+
29
+ ## Architecture
30
+
31
+ Every tool routes shell commands through a `Device` abstraction:
32
+
33
+ ```text
34
+ Tool handler ─► registry.get(deviceId) ─► AndroidDevice ─► adb -s <udid> ...
35
+ ─► IosSimDevice ─► xcrun simctl ...
36
+ ─► IosPhysicalDev ─► go-ios ...
37
+ ```
38
+
39
+ `AndroidDevice` / `IosSimDevice` / `IosPhysicalDevice` all implement the same `DeviceTransport` interface (`shell()`, `upload()`, `download()`, `isAlive()`, `dispose()`). A future remote-device adapter (WSL, SSH) drops in by implementing the same interface — no tool code changes.
40
+
41
+ ```mermaid
42
+ flowchart LR
43
+ A[Tool handler] --> R[DeviceRegistry]
44
+ R -->|cached| D[DeviceTransport]
45
+ D --> Adb[AndroidDevice<br/>adb -s udid]
46
+ D --> Sim[IosSimDevice<br/>xcrun simctl]
47
+ D --> Phy[IosPhysicalDevice<br/>go-ios]
48
+ D -.future.-> Ssh[SshDevice<br/>ssh user@host adb ...]
49
+ D -.future.-> Wsl[WslDevice<br/>wsl -- adb ...]
50
+ ```
51
+
52
+ ## CCT OAuth bypass — `solve_oauth_cct`
53
+
54
+ Chrome Custom Tabs and SafariViewController hide the OAuth flow behind a system-owned web view that no UI driver can introspect. We do **not** try to drive CCT directly. Instead:
55
+
56
+ 1. Launch Playwright Chromium with the provider's `authorize` URL.
57
+ 2. Submit credentials (optional `fillFlow`) — pause for human MFA when needed.
58
+ 3. Intercept the redirect to the app scheme (`com.example.myapp://callback?code=…`) before Chromium drops it as `ERR_UNKNOWN_URL_SCHEME`.
59
+ 4. Deliver the same URL into the app via `Device.shell()`:
60
+ - Android: `adb shell am start -W -a android.intent.action.VIEW -d <url>`
61
+ - iOS Sim: `xcrun simctl openurl <udid> <url>`
62
+ - iOS physical: requires `idb` (out of scope for this server today).
63
+
64
+ The Flutter app's deep-link handler cannot distinguish the dispatched intent from a real CCT close — the same `Intent.ACTION_VIEW` arrives with the same URL. PKCE state stays valid because the app initiated the flow (its `code_verifier` is in its own memory).
65
+
66
+ ## Configuration
67
+
68
+ | Env var | Default | Purpose |
69
+ | ----------------------------------------- | ------------ | --------------------------------------- |
70
+ | `FLUTTER_ULTRA_ADB` | `adb` | Path to the `adb` binary |
71
+ | `FLUTTER_ULTRA_XCRUN` | `xcrun` | Path to `xcrun` |
72
+ | `FLUTTER_ULTRA_GO_IOS_BIN` | `ios` | Path to the go-ios `ios` CLI |
73
+ | `FLUTTER_ULTRA_TOOL_TIMEOUT_<NAME_UPPER>` | tool default | Override per-tool watchdog ceiling (ms) |
74
+ | `FLUTTER_ULTRA_LOG_LEVEL` | `info` | `debug` / `info` / `warn` / `error` |
75
+
76
+ ## Platform support
77
+
78
+ | Tool | Linux | macOS | Windows |
79
+ | ------------------------------------ | ----------------- | ----- | ------------------------------ |
80
+ | Android tools | yes (adb on PATH) | yes | yes |
81
+ | iOS Simulator tools | no | yes | no |
82
+ | iOS physical (go-ios) | yes | yes | yes (depends on go-ios binary) |
83
+ | `solve_oauth_cct` (Android dispatch) | yes | yes | yes |
84
+ | `solve_oauth_cct` (iOS sim dispatch) | no | yes | no |
85
+
86
+ iOS tools called on non-darwin hosts return a structured "unsupported" payload (per AC-NM3) — they do not crash the server.
87
+
88
+ ## Acceptance criteria
89
+
90
+ - **AC-NM1** — `dismiss_permission_dialog(intent='allow')` clears a runtime permission within 3s on the Android emulator.
91
+ - **AC-NM2** — On iOS Simulator (Mac), `native_tap` on a SafariViewController OAuth login button completes the flow combined with `wait_for_url`.
92
+ - **AC-NM3** — iOS tools cleanly return `unsupported` on win32 / linux.
93
+ - **AC-NM4** — `solve_oauth_cct` end-to-end completes an OIDC OAuth flow against your identity provider on Android emulator within 60s.
94
+
95
+ ## Development
96
+
97
+ ```bash
98
+ npm install # at repo root
99
+ npm run -w @flutter-ultra/flutter-ultra-native-mobile build
100
+ npm run -w @flutter-ultra/flutter-ultra-native-mobile test
101
+ ```
102
+
103
+ The unit-test suite focuses on the host-side parsers (`adb devices -l`, UIAutomator XML, simctl JSON) and the `Device` abstraction — full device-loop tests require an attached Android emulator and live in the cross-platform e2e workflow.