@bdayadev/flutter-ultra-mcp 0.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (57) hide show
  1. package/.claude-plugin/plugin.json +14 -0
  2. package/.mcp.json +67 -0
  3. package/CODE_OF_CONDUCT.md +83 -0
  4. package/CONTRIBUTING.md +108 -0
  5. package/LICENSE +201 -0
  6. package/README.md +77 -0
  7. package/SECURITY.md +19 -0
  8. package/dart/ultra_flutter/CHANGELOG.md +34 -0
  9. package/dart/ultra_flutter/EXTENSIONS.md +61 -0
  10. package/dart/ultra_flutter/README.md +109 -0
  11. package/dart/ultra_flutter/pubspec.yaml +37 -0
  12. package/dart/ultra_flutter_devtools/README.md +7 -0
  13. package/dart/ultra_flutter_devtools/pubspec.yaml +27 -0
  14. package/docs/UPSTREAM-PATROL-PRS.md +5 -0
  15. package/docs/UPSTREAM-SENTRY-PR.md +62 -0
  16. package/docs/discovery-empirics.md +435 -0
  17. package/examples/counter-app/README.md +24 -0
  18. package/examples/counter-app/pubspec.yaml +23 -0
  19. package/examples/oidc-app/README.md +48 -0
  20. package/examples/oidc-app/pubspec.yaml +24 -0
  21. package/package.json +82 -0
  22. package/packages/flutter-ultra-browser/README.md +29 -0
  23. package/packages/flutter-ultra-browser/package.json +39 -0
  24. package/packages/flutter-ultra-build/README.md +60 -0
  25. package/packages/flutter-ultra-build/package.json +38 -0
  26. package/packages/flutter-ultra-devtools/README.md +7 -0
  27. package/packages/flutter-ultra-devtools/package.json +36 -0
  28. package/packages/flutter-ultra-gesture/README.md +51 -0
  29. package/packages/flutter-ultra-gesture/package.json +58 -0
  30. package/packages/flutter-ultra-native-desktop/README.md +131 -0
  31. package/packages/flutter-ultra-native-desktop/package.json +81 -0
  32. package/packages/flutter-ultra-native-mobile/README.md +103 -0
  33. package/packages/flutter-ultra-native-mobile/package.json +72 -0
  34. package/packages/flutter-ultra-patrol/README.md +40 -0
  35. package/packages/flutter-ultra-patrol/package.json +38 -0
  36. package/packages/flutter-ultra-runtime/README.md +63 -0
  37. package/packages/flutter-ultra-runtime/package.json +69 -0
  38. package/shared/contracts/package.json +31 -0
  39. package/shared/device-router/README.md +51 -0
  40. package/shared/device-router/package.json +62 -0
  41. package/shared/keyring/README.md +7 -0
  42. package/shared/keyring/package.json +24 -0
  43. package/shared/mcp-runtime/README.md +116 -0
  44. package/shared/mcp-runtime/package.json +58 -0
  45. package/shared/state-store/README.md +66 -0
  46. package/shared/state-store/package.json +60 -0
  47. package/shared/vm-service-client/README.md +135 -0
  48. package/shared/vm-service-client/package.json +62 -0
  49. package/skills/_internal-on-tool-failure/SKILL.md +13 -0
  50. package/skills/_internal-session-bootstrap/SKILL.md +18 -0
  51. package/skills/debug/SKILL.md +20 -0
  52. package/skills/devtools/SKILL.md +21 -0
  53. package/skills/drive/SKILL.md +20 -0
  54. package/skills/scaffold/SKILL.md +21 -0
  55. package/skills/setup/SKILL.md +26 -0
  56. package/skills/test/SKILL.md +19 -0
  57. package/skills/tour/SKILL.md +22 -0
@@ -0,0 +1,39 @@
1
+ {
2
+ "name": "@flutter-ultra/flutter-ultra-browser",
3
+ "version": "0.0.0",
4
+ "private": true,
5
+ "description": "MCP server for Playwright-driven browser automation: OAuth redirects, popups, console capture, network, sandboxed JS.",
6
+ "license": "Apache-2.0",
7
+ "type": "module",
8
+ "main": "dist/index.js",
9
+ "types": "dist/index.d.ts",
10
+ "bin": {
11
+ "flutter-ultra-browser": "dist/bin.js"
12
+ },
13
+ "files": [
14
+ "dist",
15
+ "README.md"
16
+ ],
17
+ "scripts": {
18
+ "build": "tsc -b",
19
+ "lint": "eslint src",
20
+ "test": "vitest run",
21
+ "typecheck": "tsc -b --noEmit",
22
+ "clean": "rimraf dist .turbo *.tsbuildinfo"
23
+ },
24
+ "dependencies": {
25
+ "@modelcontextprotocol/sdk": "^1.0.4",
26
+ "playwright-core": "^1.49.0",
27
+ "proper-lockfile": "^4.1.2",
28
+ "superjson": "^2.2.1",
29
+ "zod": "^3.23.8",
30
+ "zod-to-json-schema": "^3.23.5"
31
+ },
32
+ "devDependencies": {
33
+ "@types/proper-lockfile": "^4.1.4",
34
+ "playwright": "^1.49.0",
35
+ "rimraf": "^6.0.0",
36
+ "typescript": "^5.6.0",
37
+ "vitest": "^2.0.0"
38
+ }
39
+ }
@@ -0,0 +1,60 @@
1
+ # @flutter-ultra/flutter-ultra-build
2
+
3
+ MCP server for Flutter **build-time** tasks: pubspec, codegen, analyze, format, tests, builds, signing, l10n, assets, web validators.
4
+
5
+ Part of the [flutter-ultra-mcp](https://github.com/Bdaya-Dev/flutter-ultra-mcp) plugin (8 specialized MCP servers for Flutter automation). This package focuses on _build-time_ surface area — it never connects to a running app. For runtime control (hot reload, widget inspection) see `@flutter-ultra/flutter-ultra-runtime`.
6
+
7
+ ## Tool groups (~50 tools)
8
+
9
+ | Group | Representative tools |
10
+ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
11
+ | Project meta | `list_projects`, `project_info`, `list_flavors`, `list_dart_defines` |
12
+ | Analysis & format | `analyze`, `format`, `fix`, `fix_preview`, `flutter_doctor`, `flutter_clean`, `pub_cache_repair` |
13
+ | Pubspec & deps | `pub_get`, `pub_add`, `pub_remove`, `pub_upgrade`, `pub_outdated`, `pub_deps`, `pub_dev_search`, `pubspec_overrides_*`, `start_pub_upgrade_major` (+ poll / get / cancel) |
14
+ | Codegen | `start_build_runner_build` (+ poll / get / cancel), `start_build_runner_watch` (+ poll / stop), `flutter_gen_l10n` |
15
+ | Tests | `start_run_unit_tests`, `start_run_widget_tests`, `start_run_integration_tests`, `start_run_golden_tests`, `start_update_goldens` (each + poll / get / cancel), `test_filter` |
16
+ | Builds | `start_build_apk` / `start_build_appbundle` / `start_build_ipa` (Mac) / `start_build_web` / `start_build_windows` (Win) / `start_build_macos` (Mac) / `start_build_linux` (Linux), each with `poll_build_<plat>_job` / `get_*_result` / `cancel_*_job` |
17
+ | Mobile signing | `verify_android_signing`, `verify_ios_signing` (Mac-only), `set_bundle_id` (atomic Android+iOS update) |
18
+ | l10n / ARB | `list_missing_translations`, `arb_diff`, `arb_add_key`, `arb_remove_key` |
19
+ | Assets | `add_asset`, `validate_assets`, `list_orphan_assets` |
20
+ | Web pre-flight | `validate_web_redirect`, `validate_canvaskit_vs_html_consistency`, `flush_service_worker` |
21
+
22
+ All long-running tools (builds, codegen, tests, pub-upgrade-major) follow the **MARATHON split-tool pattern** documented in the plan §17.5: each exposes a `start_<x>` / `poll_<x>_job` / `get_<x>_result` / `cancel_<x>_job` quartet so no sync MCP call ever exceeds 55 s. Job state persists at `${CLAUDE_PLUGIN_DATA}/state/jobs/<jobId>.json` and survives MCP server restarts.
23
+
24
+ ## Cross-cutting behavior
25
+
26
+ - **Watchdog**: every tool wrapped in `withWatchdog` with a per-tool ceiling. Host-side cancellation propagates to child processes via `SIGTERM` → 2 s grace → `SIGKILL`.
27
+ - **Logging**: stderr only, JSON-lines, respects `FLUTTER_ULTRA_LOG_LEVEL`. Stdout reserved for JSON-RPC framing.
28
+ - **Keep-alive**: 30 s `notifications/message` (debug) defeats the Bun-idle-SIGKILL bug ([claude-code #58004](https://github.com/anthropics/claude-code/issues/58004)).
29
+ - **Per-tool timeout overrides**: `FLUTTER_ULTRA_TOOL_TIMEOUT_<TOOL_NAME>=<ms>` env var raises the ceiling for power users.
30
+
31
+ ## CLI dependencies
32
+
33
+ Tools shell out to:
34
+
35
+ - `dart` (required) — analysis, format, fix, pub, build_runner
36
+ - `flutter` (required for any Flutter-specific tool) — pub, gen-l10n, test, build
37
+ - `keytool` (optional, for `verify_android_signing`)
38
+ - `xcodebuild` (Mac-only, for `verify_ios_signing`)
39
+
40
+ Resolution order: env override (`FLUTTER_ULTRA_DART_BIN`, `FLUTTER_ULTRA_FLUTTER_BIN`) → PATH lookup. Missing CLIs raise `FlutterCliMissingError` with install hints.
41
+
42
+ ## Running standalone
43
+
44
+ ```jsonc
45
+ // .claude/mcp.json
46
+ {
47
+ "mcpServers": {
48
+ "flutter-ultra-build": {
49
+ "command": "node",
50
+ "args": ["${CLAUDE_PLUGIN_ROOT}/servers/flutter-ultra-build/dist/index.js"],
51
+ },
52
+ },
53
+ }
54
+ ```
55
+
56
+ ## References
57
+
58
+ - Plan §5.1 — tool catalogue (build server)
59
+ - Plan §16 — naming, schema, output conventions (binding for all servers)
60
+ - Plan §17 — timeout resilience model + split-tool pattern (binding)
@@ -0,0 +1,38 @@
1
+ {
2
+ "name": "@flutter-ultra/flutter-ultra-build",
3
+ "version": "0.0.0",
4
+ "private": true,
5
+ "description": "MCP server for Flutter build-time tasks: pubspec, codegen, analyze, format, tests, builds, l10n, assets.",
6
+ "license": "Apache-2.0",
7
+ "type": "module",
8
+ "main": "dist/index.js",
9
+ "types": "dist/index.d.ts",
10
+ "bin": {
11
+ "flutter-ultra-build": "dist/index.js"
12
+ },
13
+ "files": [
14
+ "dist",
15
+ "README.md"
16
+ ],
17
+ "scripts": {
18
+ "build": "tsc -b",
19
+ "lint": "eslint src",
20
+ "test": "vitest run",
21
+ "typecheck": "tsc -b --noEmit",
22
+ "clean": "rimraf dist .turbo *.tsbuildinfo"
23
+ },
24
+ "dependencies": {
25
+ "@modelcontextprotocol/sdk": "^1.18.0",
26
+ "execa": "^9.5.1",
27
+ "fs-extra": "^11.2.0",
28
+ "nanoid": "^5.0.9",
29
+ "proper-lockfile": "^4.1.2",
30
+ "yaml": "^2.6.1",
31
+ "zod": "^3.25.76"
32
+ },
33
+ "devDependencies": {
34
+ "@types/fs-extra": "^11.0.4",
35
+ "@types/proper-lockfile": "^4.1.4",
36
+ "vitest": "^2.1.9"
37
+ }
38
+ }
@@ -0,0 +1,7 @@
1
+ # @flutter-ultra/flutter-ultra-devtools
2
+
3
+ MCP server backing the **DevTools panel**: live MCP activity, attached sessions, recent tool calls, errors, screenshot grid. Tails `state/tool-events.jsonl` and pushes via WebSocket to the panel iframe.
4
+
5
+ **Status:** scaffold stub. Implementation owner: **wave-2 devtools worker** (see plan §12).
6
+
7
+ Pairs with the `ultra_flutter_devtools` Dart package (the DevTools extension UI) — see `packages/ultra_flutter_devtools/`.
@@ -0,0 +1,36 @@
1
+ {
2
+ "name": "@flutter-ultra/flutter-ultra-devtools",
3
+ "version": "0.0.0",
4
+ "private": true,
5
+ "description": "MCP server backing the Flutter DevTools panel: live MCP activity, sessions, tool-call timeline.",
6
+ "license": "Apache-2.0",
7
+ "type": "module",
8
+ "main": "dist/index.js",
9
+ "types": "dist/index.d.ts",
10
+ "bin": {
11
+ "flutter-ultra-devtools": "dist/index.js"
12
+ },
13
+ "files": [
14
+ "dist",
15
+ "README.md"
16
+ ],
17
+ "scripts": {
18
+ "build": "tsc -b",
19
+ "lint": "eslint src",
20
+ "test": "vitest run",
21
+ "typecheck": "tsc -b --noEmit",
22
+ "clean": "rimraf dist .turbo *.tsbuildinfo"
23
+ },
24
+ "dependencies": {
25
+ "@flutter-ultra/mcp-runtime": "*",
26
+ "@flutter-ultra/state-store": "*",
27
+ "ws": "^8.18.0",
28
+ "zod": "^3.23.8"
29
+ },
30
+ "devDependencies": {
31
+ "@types/ws": "^8.5.13",
32
+ "rimraf": "^6.0.0",
33
+ "typescript": "^5.6.0",
34
+ "vitest": "^3.2.4"
35
+ }
36
+ }
@@ -0,0 +1,51 @@
1
+ # @flutter-ultra/flutter-ultra-gesture
2
+
3
+ MCP server for **gesture dispatch** inside a running Flutter app via the in-app `ultra_flutter` mixin. Tools call `ext.flutter.ultra.*` service extensions registered by `UltraFlutterBinding` in the target app, so the app must include `package:ultra_flutter` and register the binding for any tool here to work.
4
+
5
+ ## Prerequisites
6
+
7
+ - A Flutter app running in `--debug` mode with `package:ultra_flutter` ≥ 0.0.1 registered as a binding mixin.
8
+ - The runtime server (`@flutter-ultra/flutter-ultra-runtime`) attached to the session — it owns session discovery and publishes `state/sessions.json` + `state/session-<id>.json`. This server is a read-only consumer of that state.
9
+
10
+ ## Tool catalogue (17)
11
+
12
+ | Tool | Maps to |
13
+ | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
14
+ | `interactive_elements` | `ext.flutter.ultra.interactiveElements` (rev-23 tightened: no truncation by default, optional pagination, sortBy, `withinSubtree` + `kinds` + `hasKey` filters) |
15
+ | `tap` / `double_tap` / `long_press` | `ext.flutter.ultra.tap` / `.doubleTap` / `.longPress` |
16
+ | `enter_text` / `clear_text` | `ext.flutter.ultra.enterText` / `.clearText` |
17
+ | `swipe` / `pinch_zoom` | `ext.flutter.ultra.swipe` / `.pinchZoom` |
18
+ | `scroll_to` / `scroll_until_visible` | `ext.flutter.ultra.scrollTo` (server-side polling for `scroll_until_visible`) |
19
+ | `take_screenshots` | `ext.flutter.ultra.takeScreenshots` |
20
+ | `take_responsive_screenshots` | Delegates: web → `flutter-ultra-browser` (CDP `setDeviceMetricsOverride`); native → single-viewport fallback w/ `nativeMultiViewportUnavailable` warning |
21
+ | `start_screencast` / `stop_screencast` | `ext.flutter.ultra.startScreencast` / `.stopScreencast` |
22
+ | `call_custom_extension` | invokes any user-registered `registerUltraExtension(name, ...)` |
23
+ | `list_custom_extensions` | `ext.flutter.ultra.listExtensions` |
24
+ | `wait_for` | server-side polling: `interactiveElements` every 200ms with finder match |
25
+
26
+ ## Finder spec
27
+
28
+ All matcher-accepting tools share `FinderSpec` (Zod discriminated union):
29
+
30
+ ```ts
31
+ { kind: 'key', value: string }
32
+ { kind: 'text', value: string, matchType?: 'exact' | 'contains' | 'regex' } // default 'exact'
33
+ { kind: 'type', value: string }
34
+ { kind: 'coords', x: number, y: number }
35
+ { kind: 'focused' }
36
+ { kind: 'tooltip', value: string } // server-side filter via interactiveElements
37
+ { kind: 'semantics', label: string, matchType?: 'exact' | 'contains' } // server-side filter
38
+ { kind: 'descendant', of: FinderSpec, matching: FinderSpec } // server-side filter
39
+ ```
40
+
41
+ Server-side filters (`tooltip`/`semantics`/`descendant`) materialise the element list via `interactive_elements`, resolve a coordinate, then issue `ext.flutter.ultra.tap` with `{x, y}`. Native finders (`key`/`text`/`type`/`coords`/`focused`) call the extension directly.
42
+
43
+ ## Configuration
44
+
45
+ Reads `state/sessions.json` and `state/session-<id>.json` written by `flutter-ultra-runtime`. Override the state directory with `FLUTTER_ULTRA_STATE_DIR` (defaults to `${FLUTTER_ULTRA_DATA:-~/.flutter-ultra-mcp}/state`).
46
+
47
+ DDS client name is `flutter-ultra/gesture/<pid>` so this server identifies as a distinct DDS client from the runtime server (per plan §7.2).
48
+
49
+ ## Coordination
50
+
51
+ This server **never writes** to `sessions.json`. Session lifecycle is owned by `flutter-ultra-runtime`. We open our own VM-service WebSocket per session and cache one `VmServiceClient` per active session id. The cache is dropped when the runtime server marks the session `terminated`.
@@ -0,0 +1,58 @@
1
+ {
2
+ "name": "@flutter-ultra/flutter-ultra-gesture",
3
+ "version": "0.0.1",
4
+ "private": false,
5
+ "description": "MCP server for in-app gesture dispatch via the ultra_flutter mixin: tap, text entry, scroll, screenshot, screencast.",
6
+ "license": "Apache-2.0",
7
+ "homepage": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/tree/main/packages/flutter-ultra-gesture",
8
+ "repository": {
9
+ "type": "git",
10
+ "url": "git+https://github.com/Bdaya-Dev/flutter-ultra-mcp.git",
11
+ "directory": "packages/flutter-ultra-gesture"
12
+ },
13
+ "bugs": {
14
+ "url": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/issues"
15
+ },
16
+ "keywords": [
17
+ "flutter",
18
+ "mcp",
19
+ "ultra_flutter",
20
+ "gesture",
21
+ "tap",
22
+ "screencast"
23
+ ],
24
+ "type": "module",
25
+ "main": "dist/index.js",
26
+ "types": "dist/index.d.ts",
27
+ "bin": {
28
+ "flutter-ultra-gesture": "dist/bin.js"
29
+ },
30
+ "files": [
31
+ "dist",
32
+ "README.md"
33
+ ],
34
+ "scripts": {
35
+ "build": "tsc -b",
36
+ "lint": "eslint src",
37
+ "test": "vitest run",
38
+ "typecheck": "tsc -b --noEmit",
39
+ "clean": "rimraf dist .turbo *.tsbuildinfo"
40
+ },
41
+ "dependencies": {
42
+ "@flutter-ultra/vm-service-client": "^0.0.1",
43
+ "@modelcontextprotocol/sdk": "^1.0.4",
44
+ "zod": "^3.23.8"
45
+ },
46
+ "devDependencies": {
47
+ "@types/node": "^22.0.0",
48
+ "rimraf": "^6.0.0",
49
+ "typescript": "^5.6.0",
50
+ "vitest": "^2.1.0"
51
+ },
52
+ "engines": {
53
+ "node": ">=20"
54
+ },
55
+ "publishConfig": {
56
+ "access": "public"
57
+ }
58
+ }
@@ -0,0 +1,131 @@
1
+ # @flutter-ultra/flutter-ultra-native-desktop
2
+
3
+ Cross-platform MCP server for **native desktop UI automation** — drives the
4
+ OS-level a11y tree on macOS (AXUIElement via a Swift sidecar), Windows (UIA
5
+ via a FlaUI C# sidecar), and Linux (AT-SPI 2 via a PyGObject sidecar). Used
6
+ by Claude Code skills to introspect dialogs, click buttons, type text, and
7
+ capture window screenshots in apps that Flutter's VM service can't reach.
8
+
9
+ ## Architecture
10
+
11
+ A single platform sidecar process is spawned per MCP-server lifetime. On
12
+ crash the cached entry is dropped and the next tool invocation respawns it.
13
+ Tool handlers stay platform-agnostic — the `DesktopBackend` interface
14
+ (`src/types.ts`) hides every OS-specific quirk. `src/index.ts` switches on
15
+ `process.platform` and selects `MacDesktopBackend`, `WinDesktopBackend`, or
16
+ `LinuxDesktopBackend` accordingly; the registry registers tools only when
17
+ the backend reports `helperPresent && permissionGranted`.
18
+
19
+ ## Tool surface (9 tools, plan §5.6)
20
+
21
+ | Tool | Purpose |
22
+ | ----------------------- | ------------------------------------------------------------------------ |
23
+ | `list_windows` | Visible top-level windows, optionally filtered by process / title. |
24
+ | `dump_window_tree` | Full a11y tree for a window, depth-bounded. |
25
+ | `desktop_query` | XPath subset: `//role`, `//role[@name="X"]`, `//*[@label~="X"]`. |
26
+ | `desktop_click` | Click by element id (preferred) or screen coordinates. |
27
+ | `desktop_type` | Type text; optionally clear field first; optional element focus. |
28
+ | `desktop_screenshot` | PNG (base64) — window or full screen. |
29
+ | `select_file_in_dialog` | Type a path into the frontmost file dialog + click Open/Save. |
30
+ | `confirm_dialog` | Click a dialog button by intent (allow/deny/ok/cancel/yes/no/open/save). |
31
+ | `wait_for_window` | Poll for a window matching title regex / process; configurable timeout. |
32
+
33
+ Zero tools register when the per-OS sidecar is missing or the a11y bus is
34
+ unreachable (AC-ND4). Startup logs explain why so users can self-remediate.
35
+
36
+ ## Linux path — AT-SPI via PyGObject
37
+
38
+ The Linux backend invokes `python3 -u -m atspi_bridge` from the package's
39
+ `sidecars/linux-atspi/` directory. The Python sidecar:
40
+
41
+ - Wraps `gi.repository.Atspi` for window enumeration, accessible-tree
42
+ introspection, and action invocation (`Action.do_action` for click,
43
+ `EditableText.insert_text` for type).
44
+ - Shells out to `grim` on Wayland or `scrot`/`import` on X11 for
45
+ screenshots. Both branches required because Wayland sandboxing makes
46
+ X11 screenshot APIs unusable.
47
+ - Shells out to `xdotool` (X11) or `ydotool` (Wayland) for
48
+ cursor-coordinate input synthesis when AT-SPI alone can't reach a
49
+ widget (Flutter Linux desktop is the common case — its custom-painted
50
+ widgets don't expose the EditableText interface).
51
+
52
+ ### Distro support
53
+
54
+ | Distro family | Install command |
55
+ | ---------------------------- | ------------------------------------------------------------------ |
56
+ | Debian / Ubuntu / Mint / Pop | `sudo apt-get install -y python3-gi gir1.2-atspi-2.0 at-spi2-core` |
57
+ | Fedora / RHEL / Rocky / Alma | `sudo dnf install -y python3-gobject atspi at-spi2-core` |
58
+ | Arch / Manjaro / EndeavourOS | `sudo pacman -S --needed python-gobject at-spi2-core` |
59
+ | openSUSE (Leap & Tumbleweed) | `sudo zypper install -y python3-gobject typelib-1_0-Atspi-2_0` |
60
+ | Alpine | `sudo apk add py3-gobject3 at-spi2-core` |
61
+
62
+ Use the exported `detectLocalDistro()` helper to print the exact command
63
+ for the running host.
64
+
65
+ ### Wayland caveat
66
+
67
+ AT-SPI works fully on **X11**. On **Wayland** the coverage varies by
68
+ toolkit:
69
+
70
+ | Toolkit | Coverage |
71
+ | ---------------------------------------------- | --------------------------------------------------------------- |
72
+ | GTK 3 / GTK 4 | Fully exposed |
73
+ | Qt 5 / Qt 6 with `QT_ACCESSIBILITY=1` | Fully exposed |
74
+ | Electron with `--force-renderer-accessibility` | Fully exposed |
75
+ | Flutter Linux desktop | Active window only (flutter/flutter#107016) — use ultra_flutter |
76
+
77
+ When `XDG_SESSION_TYPE=wayland` the backend sets `capabilities.waylandLimited
78
+ = true` so the registry can present a one-shot warning. For Flutter Linux
79
+ apps prefer the in-app `ultra_flutter` binding via
80
+ `@flutter-ultra/flutter-ultra-gesture` / `@flutter-ultra/flutter-ultra-runtime`.
81
+
82
+ ### Headless / minimal compositors
83
+
84
+ Sway, river, and hyprland do NOT auto-spawn `at-spi-dbus-bus`. Enable it
85
+ explicitly:
86
+
87
+ ```bash
88
+ systemctl --user enable --now at-spi-dbus-bus
89
+ ```
90
+
91
+ GNOME, KDE, XFCE, MATE, and Cinnamon auto-spawn it on session start.
92
+
93
+ ## Device router placeholder
94
+
95
+ The TS server consumes the `Device` interface from `src/device/types.ts`.
96
+ Today only `LocalDevice` ships; SSH and WSL `Device` implementations land
97
+ post-wave-3 in `@flutter-ultra/device-router`. The Python sidecar runs
98
+ inside the target Linux environment (local host, WSL distro, or remote
99
+ SSH-Linux); the TS server stays on the MCP host and pipes JSON-RPC across
100
+ stdio. No backend code changes when the router lands — the platform
101
+ switch in `src/index.ts` keeps using `new LocalDevice()` until the router
102
+ swaps it for `routerDevice.select(deviceId)`.
103
+
104
+ ## Development
105
+
106
+ ```bash
107
+ # Repo root
108
+ npm ci
109
+ npm run -w @flutter-ultra/flutter-ultra-native-desktop build
110
+ npm run -w @flutter-ultra/flutter-ultra-native-desktop test
111
+ npm run -w @flutter-ultra/flutter-ultra-native-desktop typecheck
112
+
113
+ # Python sidecar tests (Linux only)
114
+ cd packages/flutter-ultra-native-desktop/sidecars/linux-atspi
115
+ sudo apt-get install -y python3-gi gir1.2-atspi-2.0 at-spi2-core
116
+ pip install pytest
117
+ PYTHONPATH=. pytest tests/
118
+ ```
119
+
120
+ ## CI
121
+
122
+ | Workflow | Purpose |
123
+ | ------------------------------------- | ------------------------------------------------------------------- |
124
+ | `.github/workflows/ci.yml` | Cross-platform TS unit tests (ubuntu/macos/windows × Node 20/22) |
125
+ | `.github/workflows/sidecar-macos.yml` | Build the Swift helper on macos-latest + TCC probe smoke |
126
+ | `.github/workflows/sidecar-linux.yml` | Python sidecar pytest matrix + Xvfb-driven AT-SPI integration smoke |
127
+
128
+ ## Status
129
+
130
+ Wave 3 complete. macOS path merged via PR #17 (worker-J). Linux path
131
+ shipped in this PR (worker-K). Windows path in flight (worker-I).
@@ -0,0 +1,81 @@
1
+ {
2
+ "name": "@flutter-ultra/flutter-ultra-native-desktop",
3
+ "version": "0.0.1",
4
+ "private": false,
5
+ "description": "MCP server for native desktop UIs: Windows UIA (FlaUI sidecar), macOS AXUIElement (Swift sidecar + TCC UX), Linux AT-SPI (PyGObject sidecar). Cross-OS feature detection registers ZERO tools per OS path when its sidecar is unavailable.",
6
+ "license": "Apache-2.0",
7
+ "homepage": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/tree/main/packages/flutter-ultra-native-desktop",
8
+ "repository": {
9
+ "type": "git",
10
+ "url": "git+https://github.com/Bdaya-Dev/flutter-ultra-mcp.git",
11
+ "directory": "packages/flutter-ultra-native-desktop"
12
+ },
13
+ "bugs": {
14
+ "url": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/issues"
15
+ },
16
+ "keywords": [
17
+ "mcp",
18
+ "desktop",
19
+ "axuielement",
20
+ "accessibility",
21
+ "macos",
22
+ "tcc",
23
+ "ui-automation",
24
+ "at-spi",
25
+ "atspi",
26
+ "linux",
27
+ "pygobject"
28
+ ],
29
+ "type": "module",
30
+ "main": "dist/index.js",
31
+ "module": "dist/index.js",
32
+ "types": "dist/index.d.ts",
33
+ "bin": {
34
+ "flutter-ultra-native-desktop": "dist/bin.js"
35
+ },
36
+ "exports": {
37
+ ".": {
38
+ "types": "./dist/index.d.ts",
39
+ "import": "./dist/index.js"
40
+ }
41
+ },
42
+ "sideEffects": false,
43
+ "files": [
44
+ "dist",
45
+ "README.md",
46
+ "sidecars/macos-swift/Package.swift",
47
+ "sidecars/macos-swift/Sources",
48
+ "sidecars/macos-swift/bin",
49
+ "sidecars/macos-swift/README.md",
50
+ "sidecars/windows-flaui/FlutterUltraWinHelper",
51
+ "sidecars/windows-flaui/README.md",
52
+ "sidecars/linux-atspi/atspi_bridge",
53
+ "sidecars/linux-atspi/pyproject.toml",
54
+ "sidecars/linux-atspi/requirements.txt",
55
+ "sidecars/linux-atspi/README.md"
56
+ ],
57
+ "scripts": {
58
+ "build": "tsc -b",
59
+ "build:sidecar:windows": "dotnet publish sidecars/windows-flaui/FlutterUltraWinHelper/FlutterUltraWinHelper.csproj -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true -p:PublishTrimmed=false -o sidecars/windows-flaui/bin",
60
+ "lint": "eslint src",
61
+ "test": "vitest run",
62
+ "typecheck": "tsc -b --noEmit",
63
+ "clean": "rimraf dist .turbo *.tsbuildinfo sidecars/windows-flaui/bin sidecars/windows-flaui/FlutterUltraWinHelper/bin sidecars/windows-flaui/FlutterUltraWinHelper/obj"
64
+ },
65
+ "dependencies": {
66
+ "@flutter-ultra/mcp-runtime": "^0.0.1",
67
+ "@modelcontextprotocol/sdk": "^1.29.0",
68
+ "zod": "^3.23.8"
69
+ },
70
+ "devDependencies": {
71
+ "rimraf": "^6.0.0",
72
+ "typescript": "^5.6.0",
73
+ "vitest": "^2.1.0"
74
+ },
75
+ "engines": {
76
+ "node": ">=20"
77
+ },
78
+ "publishConfig": {
79
+ "access": "public"
80
+ }
81
+ }
@@ -0,0 +1,103 @@
1
+ # `@flutter-ultra/flutter-ultra-native-mobile`
2
+
3
+ MCP server for **native mobile overlay automation**: Android UIAutomator via `adb`, iOS XCUITest via `xcrun simctl` (simulators, Mac only) and `go-ios` (physical devices). Owns the **Chrome Custom Tabs / SafariViewController OAuth** problem with the `solve_oauth_cct` composite tool — see [plan §5.5.1](../../docs/) for the full design.
4
+
5
+ Part of the **Flutter Ultra MCP** plugin for Claude Code. The server runs as one of the eight MCP processes registered in `.mcp.json`.
6
+
7
+ ## Tool catalogue
8
+
9
+ | Tool | Class | Purpose |
10
+ | ------------------------------------------------------------- | ----- | ------------------------------------------------------- |
11
+ | `list_devices` | quick | Enumerate Android + iOS (sim + physical) devices |
12
+ | `dump_a11y_tree` | long | Dump UIAutomator XML / XCUITest a11y as structured tree |
13
+ | `wait_for_native_element` | long | Poll the a11y tree until a finder matches |
14
+ | `native_tap` | quick | Tap by coordinates or finder (bounds-center) |
15
+ | `native_type` | quick | Type into focused input |
16
+ | `native_swipe` | quick | Swipe between two coordinates |
17
+ | `native_back` / `_home` / `_app_switch` / `_open_settings` | quick | System keys / intents |
18
+ | `native_pin_lock` | quick | Toggle Android lock-task (kiosk mode) |
19
+ | `dismiss_permission_dialog` | quick | Smart allow/deny on runtime permission prompt |
20
+ | `native_permission_grant` / `_deny` | quick | `pm grant` / `pm revoke` |
21
+ | `take_device_screenshot` | quick | Base64 PNG via image content |
22
+ | `set_device_orientation` | quick | Portrait / landscape |
23
+ | `native_clipboard_set` / `_get` | quick | Device clipboard |
24
+ | `start_device_logs` / `poll_device_logs` / `stop_device_logs` | quick | Split-tool logcat / oslog tail |
25
+ | `solve_oauth_cct` | long | CCT/SVC OAuth via Playwright + deep-link dispatch |
26
+
27
+ All long-running tools are wrapped by the shared `runWithWatchdog` so the 60-second MCP client ceiling never kills them.
28
+
29
+ ## Architecture
30
+
31
+ Every tool routes shell commands through a `Device` abstraction:
32
+
33
+ ```text
34
+ Tool handler ─► registry.get(deviceId) ─► AndroidDevice ─► adb -s <udid> ...
35
+ ─► IosSimDevice ─► xcrun simctl ...
36
+ ─► IosPhysicalDev ─► go-ios ...
37
+ ```
38
+
39
+ `AndroidDevice` / `IosSimDevice` / `IosPhysicalDevice` all implement the same `DeviceTransport` interface (`shell()`, `upload()`, `download()`, `isAlive()`, `dispose()`). A future remote-device adapter (WSL, SSH) drops in by implementing the same interface — no tool code changes.
40
+
41
+ ```mermaid
42
+ flowchart LR
43
+ A[Tool handler] --> R[DeviceRegistry]
44
+ R -->|cached| D[DeviceTransport]
45
+ D --> Adb[AndroidDevice<br/>adb -s udid]
46
+ D --> Sim[IosSimDevice<br/>xcrun simctl]
47
+ D --> Phy[IosPhysicalDevice<br/>go-ios]
48
+ D -.future.-> Ssh[SshDevice<br/>ssh user@host adb ...]
49
+ D -.future.-> Wsl[WslDevice<br/>wsl -- adb ...]
50
+ ```
51
+
52
+ ## CCT OAuth bypass — `solve_oauth_cct`
53
+
54
+ Chrome Custom Tabs and SafariViewController hide the OAuth flow behind a system-owned web view that no UI driver can introspect. We do **not** try to drive CCT directly. Instead:
55
+
56
+ 1. Launch Playwright Chromium with the provider's `authorize` URL.
57
+ 2. Submit credentials (optional `fillFlow`) — pause for human MFA when needed.
58
+ 3. Intercept the redirect to the app scheme (`com.example.myapp://callback?code=…`) before Chromium drops it as `ERR_UNKNOWN_URL_SCHEME`.
59
+ 4. Deliver the same URL into the app via `Device.shell()`:
60
+ - Android: `adb shell am start -W -a android.intent.action.VIEW -d <url>`
61
+ - iOS Sim: `xcrun simctl openurl <udid> <url>`
62
+ - iOS physical: requires `idb` (out of scope for this server today).
63
+
64
+ The Flutter app's deep-link handler cannot distinguish the dispatched intent from a real CCT close — the same `Intent.ACTION_VIEW` arrives with the same URL. PKCE state stays valid because the app initiated the flow (its `code_verifier` is in its own memory).
65
+
66
+ ## Configuration
67
+
68
+ | Env var | Default | Purpose |
69
+ | ----------------------------------------- | ------------ | --------------------------------------- |
70
+ | `FLUTTER_ULTRA_ADB` | `adb` | Path to the `adb` binary |
71
+ | `FLUTTER_ULTRA_XCRUN` | `xcrun` | Path to `xcrun` |
72
+ | `FLUTTER_ULTRA_GO_IOS_BIN` | `ios` | Path to the go-ios `ios` CLI |
73
+ | `FLUTTER_ULTRA_TOOL_TIMEOUT_<NAME_UPPER>` | tool default | Override per-tool watchdog ceiling (ms) |
74
+ | `FLUTTER_ULTRA_LOG_LEVEL` | `info` | `debug` / `info` / `warn` / `error` |
75
+
76
+ ## Platform support
77
+
78
+ | Tool | Linux | macOS | Windows |
79
+ | ------------------------------------ | ----------------- | ----- | ------------------------------ |
80
+ | Android tools | yes (adb on PATH) | yes | yes |
81
+ | iOS Simulator tools | no | yes | no |
82
+ | iOS physical (go-ios) | yes | yes | yes (depends on go-ios binary) |
83
+ | `solve_oauth_cct` (Android dispatch) | yes | yes | yes |
84
+ | `solve_oauth_cct` (iOS sim dispatch) | no | yes | no |
85
+
86
+ iOS tools called on non-darwin hosts return a structured "unsupported" payload (per AC-NM3) — they do not crash the server.
87
+
88
+ ## Acceptance criteria
89
+
90
+ - **AC-NM1** — `dismiss_permission_dialog(intent='allow')` clears a runtime permission within 3s on the Android emulator.
91
+ - **AC-NM2** — On iOS Simulator (Mac), `native_tap` on a SafariViewController OAuth login button completes the flow combined with `wait_for_url`.
92
+ - **AC-NM3** — iOS tools cleanly return `unsupported` on win32 / linux.
93
+ - **AC-NM4** — `solve_oauth_cct` end-to-end completes an OIDC OAuth flow against your identity provider on Android emulator within 60s.
94
+
95
+ ## Development
96
+
97
+ ```bash
98
+ npm install # at repo root
99
+ npm run -w @flutter-ultra/flutter-ultra-native-mobile build
100
+ npm run -w @flutter-ultra/flutter-ultra-native-mobile test
101
+ ```
102
+
103
+ The unit-test suite focuses on the host-side parsers (`adb devices -l`, UIAutomator XML, simctl JSON) and the `Device` abstraction — full device-loop tests require an attached Android emulator and live in the cross-platform e2e workflow.