@bdayadev/flutter-ultra-mcp 0.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +14 -0
- package/.mcp.json +67 -0
- package/CODE_OF_CONDUCT.md +83 -0
- package/CONTRIBUTING.md +108 -0
- package/LICENSE +201 -0
- package/README.md +77 -0
- package/SECURITY.md +19 -0
- package/dart/ultra_flutter/CHANGELOG.md +34 -0
- package/dart/ultra_flutter/EXTENSIONS.md +61 -0
- package/dart/ultra_flutter/README.md +109 -0
- package/dart/ultra_flutter/pubspec.yaml +37 -0
- package/dart/ultra_flutter_devtools/README.md +7 -0
- package/dart/ultra_flutter_devtools/pubspec.yaml +27 -0
- package/docs/UPSTREAM-PATROL-PRS.md +5 -0
- package/docs/UPSTREAM-SENTRY-PR.md +62 -0
- package/docs/discovery-empirics.md +435 -0
- package/examples/counter-app/README.md +24 -0
- package/examples/counter-app/pubspec.yaml +23 -0
- package/examples/oidc-app/README.md +48 -0
- package/examples/oidc-app/pubspec.yaml +24 -0
- package/package.json +82 -0
- package/packages/flutter-ultra-browser/README.md +29 -0
- package/packages/flutter-ultra-browser/package.json +39 -0
- package/packages/flutter-ultra-build/README.md +60 -0
- package/packages/flutter-ultra-build/package.json +38 -0
- package/packages/flutter-ultra-devtools/README.md +7 -0
- package/packages/flutter-ultra-devtools/package.json +36 -0
- package/packages/flutter-ultra-gesture/README.md +51 -0
- package/packages/flutter-ultra-gesture/package.json +58 -0
- package/packages/flutter-ultra-native-desktop/README.md +131 -0
- package/packages/flutter-ultra-native-desktop/package.json +81 -0
- package/packages/flutter-ultra-native-mobile/README.md +103 -0
- package/packages/flutter-ultra-native-mobile/package.json +72 -0
- package/packages/flutter-ultra-patrol/README.md +40 -0
- package/packages/flutter-ultra-patrol/package.json +38 -0
- package/packages/flutter-ultra-runtime/README.md +63 -0
- package/packages/flutter-ultra-runtime/package.json +69 -0
- package/shared/contracts/package.json +31 -0
- package/shared/device-router/README.md +51 -0
- package/shared/device-router/package.json +62 -0
- package/shared/keyring/README.md +7 -0
- package/shared/keyring/package.json +24 -0
- package/shared/mcp-runtime/README.md +116 -0
- package/shared/mcp-runtime/package.json +58 -0
- package/shared/state-store/README.md +66 -0
- package/shared/state-store/package.json +60 -0
- package/shared/vm-service-client/README.md +135 -0
- package/shared/vm-service-client/package.json +62 -0
- package/skills/_internal-on-tool-failure/SKILL.md +13 -0
- package/skills/_internal-session-bootstrap/SKILL.md +18 -0
- package/skills/debug/SKILL.md +20 -0
- package/skills/devtools/SKILL.md +21 -0
- package/skills/drive/SKILL.md +20 -0
- package/skills/scaffold/SKILL.md +21 -0
- package/skills/setup/SKILL.md +26 -0
- package/skills/test/SKILL.md +19 -0
- package/skills/tour/SKILL.md +22 -0
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@flutter-ultra/flutter-ultra-browser",
|
|
3
|
+
"version": "0.0.0",
|
|
4
|
+
"private": true,
|
|
5
|
+
"description": "MCP server for Playwright-driven browser automation: OAuth redirects, popups, console capture, network, sandboxed JS.",
|
|
6
|
+
"license": "Apache-2.0",
|
|
7
|
+
"type": "module",
|
|
8
|
+
"main": "dist/index.js",
|
|
9
|
+
"types": "dist/index.d.ts",
|
|
10
|
+
"bin": {
|
|
11
|
+
"flutter-ultra-browser": "dist/bin.js"
|
|
12
|
+
},
|
|
13
|
+
"files": [
|
|
14
|
+
"dist",
|
|
15
|
+
"README.md"
|
|
16
|
+
],
|
|
17
|
+
"scripts": {
|
|
18
|
+
"build": "tsc -b",
|
|
19
|
+
"lint": "eslint src",
|
|
20
|
+
"test": "vitest run",
|
|
21
|
+
"typecheck": "tsc -b --noEmit",
|
|
22
|
+
"clean": "rimraf dist .turbo *.tsbuildinfo"
|
|
23
|
+
},
|
|
24
|
+
"dependencies": {
|
|
25
|
+
"@modelcontextprotocol/sdk": "^1.0.4",
|
|
26
|
+
"playwright-core": "^1.49.0",
|
|
27
|
+
"proper-lockfile": "^4.1.2",
|
|
28
|
+
"superjson": "^2.2.1",
|
|
29
|
+
"zod": "^3.23.8",
|
|
30
|
+
"zod-to-json-schema": "^3.23.5"
|
|
31
|
+
},
|
|
32
|
+
"devDependencies": {
|
|
33
|
+
"@types/proper-lockfile": "^4.1.4",
|
|
34
|
+
"playwright": "^1.49.0",
|
|
35
|
+
"rimraf": "^6.0.0",
|
|
36
|
+
"typescript": "^5.6.0",
|
|
37
|
+
"vitest": "^2.0.0"
|
|
38
|
+
}
|
|
39
|
+
}
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# @flutter-ultra/flutter-ultra-build
|
|
2
|
+
|
|
3
|
+
MCP server for Flutter **build-time** tasks: pubspec, codegen, analyze, format, tests, builds, signing, l10n, assets, web validators.
|
|
4
|
+
|
|
5
|
+
Part of the [flutter-ultra-mcp](https://github.com/Bdaya-Dev/flutter-ultra-mcp) plugin (8 specialized MCP servers for Flutter automation). This package focuses on _build-time_ surface area — it never connects to a running app. For runtime control (hot reload, widget inspection) see `@flutter-ultra/flutter-ultra-runtime`.
|
|
6
|
+
|
|
7
|
+
## Tool groups (~50 tools)
|
|
8
|
+
|
|
9
|
+
| Group | Representative tools |
|
|
10
|
+
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
11
|
+
| Project meta | `list_projects`, `project_info`, `list_flavors`, `list_dart_defines` |
|
|
12
|
+
| Analysis & format | `analyze`, `format`, `fix`, `fix_preview`, `flutter_doctor`, `flutter_clean`, `pub_cache_repair` |
|
|
13
|
+
| Pubspec & deps | `pub_get`, `pub_add`, `pub_remove`, `pub_upgrade`, `pub_outdated`, `pub_deps`, `pub_dev_search`, `pubspec_overrides_*`, `start_pub_upgrade_major` (+ poll / get / cancel) |
|
|
14
|
+
| Codegen | `start_build_runner_build` (+ poll / get / cancel), `start_build_runner_watch` (+ poll / stop), `flutter_gen_l10n` |
|
|
15
|
+
| Tests | `start_run_unit_tests`, `start_run_widget_tests`, `start_run_integration_tests`, `start_run_golden_tests`, `start_update_goldens` (each + poll / get / cancel), `test_filter` |
|
|
16
|
+
| Builds | `start_build_apk` / `start_build_appbundle` / `start_build_ipa` (Mac) / `start_build_web` / `start_build_windows` (Win) / `start_build_macos` (Mac) / `start_build_linux` (Linux), each with `poll_build_<plat>_job` / `get_*_result` / `cancel_*_job` |
|
|
17
|
+
| Mobile signing | `verify_android_signing`, `verify_ios_signing` (Mac-only), `set_bundle_id` (atomic Android+iOS update) |
|
|
18
|
+
| l10n / ARB | `list_missing_translations`, `arb_diff`, `arb_add_key`, `arb_remove_key` |
|
|
19
|
+
| Assets | `add_asset`, `validate_assets`, `list_orphan_assets` |
|
|
20
|
+
| Web pre-flight | `validate_web_redirect`, `validate_canvaskit_vs_html_consistency`, `flush_service_worker` |
|
|
21
|
+
|
|
22
|
+
All long-running tools (builds, codegen, tests, pub-upgrade-major) follow the **MARATHON split-tool pattern** documented in the plan §17.5: each exposes a `start_<x>` / `poll_<x>_job` / `get_<x>_result` / `cancel_<x>_job` quartet so no sync MCP call ever exceeds 55 s. Job state persists at `${CLAUDE_PLUGIN_DATA}/state/jobs/<jobId>.json` and survives MCP server restarts.
|
|
23
|
+
|
|
24
|
+
## Cross-cutting behavior
|
|
25
|
+
|
|
26
|
+
- **Watchdog**: every tool wrapped in `withWatchdog` with a per-tool ceiling. Host-side cancellation propagates to child processes via `SIGTERM` → 2 s grace → `SIGKILL`.
|
|
27
|
+
- **Logging**: stderr only, JSON-lines, respects `FLUTTER_ULTRA_LOG_LEVEL`. Stdout reserved for JSON-RPC framing.
|
|
28
|
+
- **Keep-alive**: 30 s `notifications/message` (debug) defeats the Bun-idle-SIGKILL bug ([claude-code #58004](https://github.com/anthropics/claude-code/issues/58004)).
|
|
29
|
+
- **Per-tool timeout overrides**: `FLUTTER_ULTRA_TOOL_TIMEOUT_<TOOL_NAME>=<ms>` env var raises the ceiling for power users.
|
|
30
|
+
|
|
31
|
+
## CLI dependencies
|
|
32
|
+
|
|
33
|
+
Tools shell out to:
|
|
34
|
+
|
|
35
|
+
- `dart` (required) — analysis, format, fix, pub, build_runner
|
|
36
|
+
- `flutter` (required for any Flutter-specific tool) — pub, gen-l10n, test, build
|
|
37
|
+
- `keytool` (optional, for `verify_android_signing`)
|
|
38
|
+
- `xcodebuild` (Mac-only, for `verify_ios_signing`)
|
|
39
|
+
|
|
40
|
+
Resolution order: env override (`FLUTTER_ULTRA_DART_BIN`, `FLUTTER_ULTRA_FLUTTER_BIN`) → PATH lookup. Missing CLIs raise `FlutterCliMissingError` with install hints.
|
|
41
|
+
|
|
42
|
+
## Running standalone
|
|
43
|
+
|
|
44
|
+
```jsonc
|
|
45
|
+
// .claude/mcp.json
|
|
46
|
+
{
|
|
47
|
+
"mcpServers": {
|
|
48
|
+
"flutter-ultra-build": {
|
|
49
|
+
"command": "node",
|
|
50
|
+
"args": ["${CLAUDE_PLUGIN_ROOT}/servers/flutter-ultra-build/dist/index.js"],
|
|
51
|
+
},
|
|
52
|
+
},
|
|
53
|
+
}
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## References
|
|
57
|
+
|
|
58
|
+
- Plan §5.1 — tool catalogue (build server)
|
|
59
|
+
- Plan §16 — naming, schema, output conventions (binding for all servers)
|
|
60
|
+
- Plan §17 — timeout resilience model + split-tool pattern (binding)
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@flutter-ultra/flutter-ultra-build",
|
|
3
|
+
"version": "0.0.0",
|
|
4
|
+
"private": true,
|
|
5
|
+
"description": "MCP server for Flutter build-time tasks: pubspec, codegen, analyze, format, tests, builds, l10n, assets.",
|
|
6
|
+
"license": "Apache-2.0",
|
|
7
|
+
"type": "module",
|
|
8
|
+
"main": "dist/index.js",
|
|
9
|
+
"types": "dist/index.d.ts",
|
|
10
|
+
"bin": {
|
|
11
|
+
"flutter-ultra-build": "dist/index.js"
|
|
12
|
+
},
|
|
13
|
+
"files": [
|
|
14
|
+
"dist",
|
|
15
|
+
"README.md"
|
|
16
|
+
],
|
|
17
|
+
"scripts": {
|
|
18
|
+
"build": "tsc -b",
|
|
19
|
+
"lint": "eslint src",
|
|
20
|
+
"test": "vitest run",
|
|
21
|
+
"typecheck": "tsc -b --noEmit",
|
|
22
|
+
"clean": "rimraf dist .turbo *.tsbuildinfo"
|
|
23
|
+
},
|
|
24
|
+
"dependencies": {
|
|
25
|
+
"@modelcontextprotocol/sdk": "^1.18.0",
|
|
26
|
+
"execa": "^9.5.1",
|
|
27
|
+
"fs-extra": "^11.2.0",
|
|
28
|
+
"nanoid": "^5.0.9",
|
|
29
|
+
"proper-lockfile": "^4.1.2",
|
|
30
|
+
"yaml": "^2.6.1",
|
|
31
|
+
"zod": "^3.25.76"
|
|
32
|
+
},
|
|
33
|
+
"devDependencies": {
|
|
34
|
+
"@types/fs-extra": "^11.0.4",
|
|
35
|
+
"@types/proper-lockfile": "^4.1.4",
|
|
36
|
+
"vitest": "^2.1.9"
|
|
37
|
+
}
|
|
38
|
+
}
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
# @flutter-ultra/flutter-ultra-devtools
|
|
2
|
+
|
|
3
|
+
MCP server backing the **DevTools panel**: live MCP activity, attached sessions, recent tool calls, errors, screenshot grid. Tails `state/tool-events.jsonl` and pushes via WebSocket to the panel iframe.
|
|
4
|
+
|
|
5
|
+
**Status:** scaffold stub. Implementation owner: **wave-2 devtools worker** (see plan §12).
|
|
6
|
+
|
|
7
|
+
Pairs with the `ultra_flutter_devtools` Dart package (the DevTools extension UI) — see `packages/ultra_flutter_devtools/`.
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@flutter-ultra/flutter-ultra-devtools",
|
|
3
|
+
"version": "0.0.0",
|
|
4
|
+
"private": true,
|
|
5
|
+
"description": "MCP server backing the Flutter DevTools panel: live MCP activity, sessions, tool-call timeline.",
|
|
6
|
+
"license": "Apache-2.0",
|
|
7
|
+
"type": "module",
|
|
8
|
+
"main": "dist/index.js",
|
|
9
|
+
"types": "dist/index.d.ts",
|
|
10
|
+
"bin": {
|
|
11
|
+
"flutter-ultra-devtools": "dist/index.js"
|
|
12
|
+
},
|
|
13
|
+
"files": [
|
|
14
|
+
"dist",
|
|
15
|
+
"README.md"
|
|
16
|
+
],
|
|
17
|
+
"scripts": {
|
|
18
|
+
"build": "tsc -b",
|
|
19
|
+
"lint": "eslint src",
|
|
20
|
+
"test": "vitest run",
|
|
21
|
+
"typecheck": "tsc -b --noEmit",
|
|
22
|
+
"clean": "rimraf dist .turbo *.tsbuildinfo"
|
|
23
|
+
},
|
|
24
|
+
"dependencies": {
|
|
25
|
+
"@flutter-ultra/mcp-runtime": "*",
|
|
26
|
+
"@flutter-ultra/state-store": "*",
|
|
27
|
+
"ws": "^8.18.0",
|
|
28
|
+
"zod": "^3.23.8"
|
|
29
|
+
},
|
|
30
|
+
"devDependencies": {
|
|
31
|
+
"@types/ws": "^8.5.13",
|
|
32
|
+
"rimraf": "^6.0.0",
|
|
33
|
+
"typescript": "^5.6.0",
|
|
34
|
+
"vitest": "^3.2.4"
|
|
35
|
+
}
|
|
36
|
+
}
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# @flutter-ultra/flutter-ultra-gesture
|
|
2
|
+
|
|
3
|
+
MCP server for **gesture dispatch** inside a running Flutter app via the in-app `ultra_flutter` mixin. Tools call `ext.flutter.ultra.*` service extensions registered by `UltraFlutterBinding` in the target app, so the app must include `package:ultra_flutter` and register the binding for any tool here to work.
|
|
4
|
+
|
|
5
|
+
## Prerequisites
|
|
6
|
+
|
|
7
|
+
- A Flutter app running in `--debug` mode with `package:ultra_flutter` ≥ 0.0.1 registered as a binding mixin.
|
|
8
|
+
- The runtime server (`@flutter-ultra/flutter-ultra-runtime`) attached to the session — it owns session discovery and publishes `state/sessions.json` + `state/session-<id>.json`. This server is a read-only consumer of that state.
|
|
9
|
+
|
|
10
|
+
## Tool catalogue (17)
|
|
11
|
+
|
|
12
|
+
| Tool | Maps to |
|
|
13
|
+
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
14
|
+
| `interactive_elements` | `ext.flutter.ultra.interactiveElements` (rev-23 tightened: no truncation by default, optional pagination, sortBy, `withinSubtree` + `kinds` + `hasKey` filters) |
|
|
15
|
+
| `tap` / `double_tap` / `long_press` | `ext.flutter.ultra.tap` / `.doubleTap` / `.longPress` |
|
|
16
|
+
| `enter_text` / `clear_text` | `ext.flutter.ultra.enterText` / `.clearText` |
|
|
17
|
+
| `swipe` / `pinch_zoom` | `ext.flutter.ultra.swipe` / `.pinchZoom` |
|
|
18
|
+
| `scroll_to` / `scroll_until_visible` | `ext.flutter.ultra.scrollTo` (server-side polling for `scroll_until_visible`) |
|
|
19
|
+
| `take_screenshots` | `ext.flutter.ultra.takeScreenshots` |
|
|
20
|
+
| `take_responsive_screenshots` | Delegates: web → `flutter-ultra-browser` (CDP `setDeviceMetricsOverride`); native → single-viewport fallback w/ `nativeMultiViewportUnavailable` warning |
|
|
21
|
+
| `start_screencast` / `stop_screencast` | `ext.flutter.ultra.startScreencast` / `.stopScreencast` |
|
|
22
|
+
| `call_custom_extension` | invokes any user-registered `registerUltraExtension(name, ...)` |
|
|
23
|
+
| `list_custom_extensions` | `ext.flutter.ultra.listExtensions` |
|
|
24
|
+
| `wait_for` | server-side polling: `interactiveElements` every 200ms with finder match |
|
|
25
|
+
|
|
26
|
+
## Finder spec
|
|
27
|
+
|
|
28
|
+
All matcher-accepting tools share `FinderSpec` (Zod discriminated union):
|
|
29
|
+
|
|
30
|
+
```ts
|
|
31
|
+
{ kind: 'key', value: string }
|
|
32
|
+
{ kind: 'text', value: string, matchType?: 'exact' | 'contains' | 'regex' } // default 'exact'
|
|
33
|
+
{ kind: 'type', value: string }
|
|
34
|
+
{ kind: 'coords', x: number, y: number }
|
|
35
|
+
{ kind: 'focused' }
|
|
36
|
+
{ kind: 'tooltip', value: string } // server-side filter via interactiveElements
|
|
37
|
+
{ kind: 'semantics', label: string, matchType?: 'exact' | 'contains' } // server-side filter
|
|
38
|
+
{ kind: 'descendant', of: FinderSpec, matching: FinderSpec } // server-side filter
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Server-side filters (`tooltip`/`semantics`/`descendant`) materialise the element list via `interactive_elements`, resolve a coordinate, then issue `ext.flutter.ultra.tap` with `{x, y}`. Native finders (`key`/`text`/`type`/`coords`/`focused`) call the extension directly.
|
|
42
|
+
|
|
43
|
+
## Configuration
|
|
44
|
+
|
|
45
|
+
Reads `state/sessions.json` and `state/session-<id>.json` written by `flutter-ultra-runtime`. Override the state directory with `FLUTTER_ULTRA_STATE_DIR` (defaults to `${FLUTTER_ULTRA_DATA:-~/.flutter-ultra-mcp}/state`).
|
|
46
|
+
|
|
47
|
+
DDS client name is `flutter-ultra/gesture/<pid>` so this server identifies as a distinct DDS client from the runtime server (per plan §7.2).
|
|
48
|
+
|
|
49
|
+
## Coordination
|
|
50
|
+
|
|
51
|
+
This server **never writes** to `sessions.json`. Session lifecycle is owned by `flutter-ultra-runtime`. We open our own VM-service WebSocket per session and cache one `VmServiceClient` per active session id. The cache is dropped when the runtime server marks the session `terminated`.
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@flutter-ultra/flutter-ultra-gesture",
|
|
3
|
+
"version": "0.0.1",
|
|
4
|
+
"private": false,
|
|
5
|
+
"description": "MCP server for in-app gesture dispatch via the ultra_flutter mixin: tap, text entry, scroll, screenshot, screencast.",
|
|
6
|
+
"license": "Apache-2.0",
|
|
7
|
+
"homepage": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/tree/main/packages/flutter-ultra-gesture",
|
|
8
|
+
"repository": {
|
|
9
|
+
"type": "git",
|
|
10
|
+
"url": "git+https://github.com/Bdaya-Dev/flutter-ultra-mcp.git",
|
|
11
|
+
"directory": "packages/flutter-ultra-gesture"
|
|
12
|
+
},
|
|
13
|
+
"bugs": {
|
|
14
|
+
"url": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/issues"
|
|
15
|
+
},
|
|
16
|
+
"keywords": [
|
|
17
|
+
"flutter",
|
|
18
|
+
"mcp",
|
|
19
|
+
"ultra_flutter",
|
|
20
|
+
"gesture",
|
|
21
|
+
"tap",
|
|
22
|
+
"screencast"
|
|
23
|
+
],
|
|
24
|
+
"type": "module",
|
|
25
|
+
"main": "dist/index.js",
|
|
26
|
+
"types": "dist/index.d.ts",
|
|
27
|
+
"bin": {
|
|
28
|
+
"flutter-ultra-gesture": "dist/bin.js"
|
|
29
|
+
},
|
|
30
|
+
"files": [
|
|
31
|
+
"dist",
|
|
32
|
+
"README.md"
|
|
33
|
+
],
|
|
34
|
+
"scripts": {
|
|
35
|
+
"build": "tsc -b",
|
|
36
|
+
"lint": "eslint src",
|
|
37
|
+
"test": "vitest run",
|
|
38
|
+
"typecheck": "tsc -b --noEmit",
|
|
39
|
+
"clean": "rimraf dist .turbo *.tsbuildinfo"
|
|
40
|
+
},
|
|
41
|
+
"dependencies": {
|
|
42
|
+
"@flutter-ultra/vm-service-client": "^0.0.1",
|
|
43
|
+
"@modelcontextprotocol/sdk": "^1.0.4",
|
|
44
|
+
"zod": "^3.23.8"
|
|
45
|
+
},
|
|
46
|
+
"devDependencies": {
|
|
47
|
+
"@types/node": "^22.0.0",
|
|
48
|
+
"rimraf": "^6.0.0",
|
|
49
|
+
"typescript": "^5.6.0",
|
|
50
|
+
"vitest": "^2.1.0"
|
|
51
|
+
},
|
|
52
|
+
"engines": {
|
|
53
|
+
"node": ">=20"
|
|
54
|
+
},
|
|
55
|
+
"publishConfig": {
|
|
56
|
+
"access": "public"
|
|
57
|
+
}
|
|
58
|
+
}
|
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
# @flutter-ultra/flutter-ultra-native-desktop
|
|
2
|
+
|
|
3
|
+
Cross-platform MCP server for **native desktop UI automation** — drives the
|
|
4
|
+
OS-level a11y tree on macOS (AXUIElement via a Swift sidecar), Windows (UIA
|
|
5
|
+
via a FlaUI C# sidecar), and Linux (AT-SPI 2 via a PyGObject sidecar). Used
|
|
6
|
+
by Claude Code skills to introspect dialogs, click buttons, type text, and
|
|
7
|
+
capture window screenshots in apps that Flutter's VM service can't reach.
|
|
8
|
+
|
|
9
|
+
## Architecture
|
|
10
|
+
|
|
11
|
+
A single platform sidecar process is spawned per MCP-server lifetime. On
|
|
12
|
+
crash the cached entry is dropped and the next tool invocation respawns it.
|
|
13
|
+
Tool handlers stay platform-agnostic — the `DesktopBackend` interface
|
|
14
|
+
(`src/types.ts`) hides every OS-specific quirk. `src/index.ts` switches on
|
|
15
|
+
`process.platform` and selects `MacDesktopBackend`, `WinDesktopBackend`, or
|
|
16
|
+
`LinuxDesktopBackend` accordingly; the registry registers tools only when
|
|
17
|
+
the backend reports `helperPresent && permissionGranted`.
|
|
18
|
+
|
|
19
|
+
## Tool surface (9 tools, plan §5.6)
|
|
20
|
+
|
|
21
|
+
| Tool | Purpose |
|
|
22
|
+
| ----------------------- | ------------------------------------------------------------------------ |
|
|
23
|
+
| `list_windows` | Visible top-level windows, optionally filtered by process / title. |
|
|
24
|
+
| `dump_window_tree` | Full a11y tree for a window, depth-bounded. |
|
|
25
|
+
| `desktop_query` | XPath subset: `//role`, `//role[@name="X"]`, `//*[@label~="X"]`. |
|
|
26
|
+
| `desktop_click` | Click by element id (preferred) or screen coordinates. |
|
|
27
|
+
| `desktop_type` | Type text; optionally clear field first; optional element focus. |
|
|
28
|
+
| `desktop_screenshot` | PNG (base64) — window or full screen. |
|
|
29
|
+
| `select_file_in_dialog` | Type a path into the frontmost file dialog + click Open/Save. |
|
|
30
|
+
| `confirm_dialog` | Click a dialog button by intent (allow/deny/ok/cancel/yes/no/open/save). |
|
|
31
|
+
| `wait_for_window` | Poll for a window matching title regex / process; configurable timeout. |
|
|
32
|
+
|
|
33
|
+
Zero tools register when the per-OS sidecar is missing or the a11y bus is
|
|
34
|
+
unreachable (AC-ND4). Startup logs explain why so users can self-remediate.
|
|
35
|
+
|
|
36
|
+
## Linux path — AT-SPI via PyGObject
|
|
37
|
+
|
|
38
|
+
The Linux backend invokes `python3 -u -m atspi_bridge` from the package's
|
|
39
|
+
`sidecars/linux-atspi/` directory. The Python sidecar:
|
|
40
|
+
|
|
41
|
+
- Wraps `gi.repository.Atspi` for window enumeration, accessible-tree
|
|
42
|
+
introspection, and action invocation (`Action.do_action` for click,
|
|
43
|
+
`EditableText.insert_text` for type).
|
|
44
|
+
- Shells out to `grim` on Wayland or `scrot`/`import` on X11 for
|
|
45
|
+
screenshots. Both branches required because Wayland sandboxing makes
|
|
46
|
+
X11 screenshot APIs unusable.
|
|
47
|
+
- Shells out to `xdotool` (X11) or `ydotool` (Wayland) for
|
|
48
|
+
cursor-coordinate input synthesis when AT-SPI alone can't reach a
|
|
49
|
+
widget (Flutter Linux desktop is the common case — its custom-painted
|
|
50
|
+
widgets don't expose the EditableText interface).
|
|
51
|
+
|
|
52
|
+
### Distro support
|
|
53
|
+
|
|
54
|
+
| Distro family | Install command |
|
|
55
|
+
| ---------------------------- | ------------------------------------------------------------------ |
|
|
56
|
+
| Debian / Ubuntu / Mint / Pop | `sudo apt-get install -y python3-gi gir1.2-atspi-2.0 at-spi2-core` |
|
|
57
|
+
| Fedora / RHEL / Rocky / Alma | `sudo dnf install -y python3-gobject atspi at-spi2-core` |
|
|
58
|
+
| Arch / Manjaro / EndeavourOS | `sudo pacman -S --needed python-gobject at-spi2-core` |
|
|
59
|
+
| openSUSE (Leap & Tumbleweed) | `sudo zypper install -y python3-gobject typelib-1_0-Atspi-2_0` |
|
|
60
|
+
| Alpine | `sudo apk add py3-gobject3 at-spi2-core` |
|
|
61
|
+
|
|
62
|
+
Use the exported `detectLocalDistro()` helper to print the exact command
|
|
63
|
+
for the running host.
|
|
64
|
+
|
|
65
|
+
### Wayland caveat
|
|
66
|
+
|
|
67
|
+
AT-SPI works fully on **X11**. On **Wayland** the coverage varies by
|
|
68
|
+
toolkit:
|
|
69
|
+
|
|
70
|
+
| Toolkit | Coverage |
|
|
71
|
+
| ---------------------------------------------- | --------------------------------------------------------------- |
|
|
72
|
+
| GTK 3 / GTK 4 | Fully exposed |
|
|
73
|
+
| Qt 5 / Qt 6 with `QT_ACCESSIBILITY=1` | Fully exposed |
|
|
74
|
+
| Electron with `--force-renderer-accessibility` | Fully exposed |
|
|
75
|
+
| Flutter Linux desktop | Active window only (flutter/flutter#107016) — use ultra_flutter |
|
|
76
|
+
|
|
77
|
+
When `XDG_SESSION_TYPE=wayland` the backend sets `capabilities.waylandLimited
|
|
78
|
+
= true` so the registry can present a one-shot warning. For Flutter Linux
|
|
79
|
+
apps prefer the in-app `ultra_flutter` binding via
|
|
80
|
+
`@flutter-ultra/flutter-ultra-gesture` / `@flutter-ultra/flutter-ultra-runtime`.
|
|
81
|
+
|
|
82
|
+
### Headless / minimal compositors
|
|
83
|
+
|
|
84
|
+
Sway, river, and hyprland do NOT auto-spawn `at-spi-dbus-bus`. Enable it
|
|
85
|
+
explicitly:
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
systemctl --user enable --now at-spi-dbus-bus
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
GNOME, KDE, XFCE, MATE, and Cinnamon auto-spawn it on session start.
|
|
92
|
+
|
|
93
|
+
## Device router placeholder
|
|
94
|
+
|
|
95
|
+
The TS server consumes the `Device` interface from `src/device/types.ts`.
|
|
96
|
+
Today only `LocalDevice` ships; SSH and WSL `Device` implementations land
|
|
97
|
+
post-wave-3 in `@flutter-ultra/device-router`. The Python sidecar runs
|
|
98
|
+
inside the target Linux environment (local host, WSL distro, or remote
|
|
99
|
+
SSH-Linux); the TS server stays on the MCP host and pipes JSON-RPC across
|
|
100
|
+
stdio. No backend code changes when the router lands — the platform
|
|
101
|
+
switch in `src/index.ts` keeps using `new LocalDevice()` until the router
|
|
102
|
+
swaps it for `routerDevice.select(deviceId)`.
|
|
103
|
+
|
|
104
|
+
## Development
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
# Repo root
|
|
108
|
+
npm ci
|
|
109
|
+
npm run -w @flutter-ultra/flutter-ultra-native-desktop build
|
|
110
|
+
npm run -w @flutter-ultra/flutter-ultra-native-desktop test
|
|
111
|
+
npm run -w @flutter-ultra/flutter-ultra-native-desktop typecheck
|
|
112
|
+
|
|
113
|
+
# Python sidecar tests (Linux only)
|
|
114
|
+
cd packages/flutter-ultra-native-desktop/sidecars/linux-atspi
|
|
115
|
+
sudo apt-get install -y python3-gi gir1.2-atspi-2.0 at-spi2-core
|
|
116
|
+
pip install pytest
|
|
117
|
+
PYTHONPATH=. pytest tests/
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
## CI
|
|
121
|
+
|
|
122
|
+
| Workflow | Purpose |
|
|
123
|
+
| ------------------------------------- | ------------------------------------------------------------------- |
|
|
124
|
+
| `.github/workflows/ci.yml` | Cross-platform TS unit tests (ubuntu/macos/windows × Node 20/22) |
|
|
125
|
+
| `.github/workflows/sidecar-macos.yml` | Build the Swift helper on macos-latest + TCC probe smoke |
|
|
126
|
+
| `.github/workflows/sidecar-linux.yml` | Python sidecar pytest matrix + Xvfb-driven AT-SPI integration smoke |
|
|
127
|
+
|
|
128
|
+
## Status
|
|
129
|
+
|
|
130
|
+
Wave 3 complete. macOS path merged via PR #17 (worker-J). Linux path
|
|
131
|
+
shipped in this PR (worker-K). Windows path in flight (worker-I).
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@flutter-ultra/flutter-ultra-native-desktop",
|
|
3
|
+
"version": "0.0.1",
|
|
4
|
+
"private": false,
|
|
5
|
+
"description": "MCP server for native desktop UIs: Windows UIA (FlaUI sidecar), macOS AXUIElement (Swift sidecar + TCC UX), Linux AT-SPI (PyGObject sidecar). Cross-OS feature detection registers ZERO tools per OS path when its sidecar is unavailable.",
|
|
6
|
+
"license": "Apache-2.0",
|
|
7
|
+
"homepage": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/tree/main/packages/flutter-ultra-native-desktop",
|
|
8
|
+
"repository": {
|
|
9
|
+
"type": "git",
|
|
10
|
+
"url": "git+https://github.com/Bdaya-Dev/flutter-ultra-mcp.git",
|
|
11
|
+
"directory": "packages/flutter-ultra-native-desktop"
|
|
12
|
+
},
|
|
13
|
+
"bugs": {
|
|
14
|
+
"url": "https://github.com/Bdaya-Dev/flutter-ultra-mcp/issues"
|
|
15
|
+
},
|
|
16
|
+
"keywords": [
|
|
17
|
+
"mcp",
|
|
18
|
+
"desktop",
|
|
19
|
+
"axuielement",
|
|
20
|
+
"accessibility",
|
|
21
|
+
"macos",
|
|
22
|
+
"tcc",
|
|
23
|
+
"ui-automation",
|
|
24
|
+
"at-spi",
|
|
25
|
+
"atspi",
|
|
26
|
+
"linux",
|
|
27
|
+
"pygobject"
|
|
28
|
+
],
|
|
29
|
+
"type": "module",
|
|
30
|
+
"main": "dist/index.js",
|
|
31
|
+
"module": "dist/index.js",
|
|
32
|
+
"types": "dist/index.d.ts",
|
|
33
|
+
"bin": {
|
|
34
|
+
"flutter-ultra-native-desktop": "dist/bin.js"
|
|
35
|
+
},
|
|
36
|
+
"exports": {
|
|
37
|
+
".": {
|
|
38
|
+
"types": "./dist/index.d.ts",
|
|
39
|
+
"import": "./dist/index.js"
|
|
40
|
+
}
|
|
41
|
+
},
|
|
42
|
+
"sideEffects": false,
|
|
43
|
+
"files": [
|
|
44
|
+
"dist",
|
|
45
|
+
"README.md",
|
|
46
|
+
"sidecars/macos-swift/Package.swift",
|
|
47
|
+
"sidecars/macos-swift/Sources",
|
|
48
|
+
"sidecars/macos-swift/bin",
|
|
49
|
+
"sidecars/macos-swift/README.md",
|
|
50
|
+
"sidecars/windows-flaui/FlutterUltraWinHelper",
|
|
51
|
+
"sidecars/windows-flaui/README.md",
|
|
52
|
+
"sidecars/linux-atspi/atspi_bridge",
|
|
53
|
+
"sidecars/linux-atspi/pyproject.toml",
|
|
54
|
+
"sidecars/linux-atspi/requirements.txt",
|
|
55
|
+
"sidecars/linux-atspi/README.md"
|
|
56
|
+
],
|
|
57
|
+
"scripts": {
|
|
58
|
+
"build": "tsc -b",
|
|
59
|
+
"build:sidecar:windows": "dotnet publish sidecars/windows-flaui/FlutterUltraWinHelper/FlutterUltraWinHelper.csproj -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true -p:PublishTrimmed=false -o sidecars/windows-flaui/bin",
|
|
60
|
+
"lint": "eslint src",
|
|
61
|
+
"test": "vitest run",
|
|
62
|
+
"typecheck": "tsc -b --noEmit",
|
|
63
|
+
"clean": "rimraf dist .turbo *.tsbuildinfo sidecars/windows-flaui/bin sidecars/windows-flaui/FlutterUltraWinHelper/bin sidecars/windows-flaui/FlutterUltraWinHelper/obj"
|
|
64
|
+
},
|
|
65
|
+
"dependencies": {
|
|
66
|
+
"@flutter-ultra/mcp-runtime": "^0.0.1",
|
|
67
|
+
"@modelcontextprotocol/sdk": "^1.29.0",
|
|
68
|
+
"zod": "^3.23.8"
|
|
69
|
+
},
|
|
70
|
+
"devDependencies": {
|
|
71
|
+
"rimraf": "^6.0.0",
|
|
72
|
+
"typescript": "^5.6.0",
|
|
73
|
+
"vitest": "^2.1.0"
|
|
74
|
+
},
|
|
75
|
+
"engines": {
|
|
76
|
+
"node": ">=20"
|
|
77
|
+
},
|
|
78
|
+
"publishConfig": {
|
|
79
|
+
"access": "public"
|
|
80
|
+
}
|
|
81
|
+
}
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
# `@flutter-ultra/flutter-ultra-native-mobile`
|
|
2
|
+
|
|
3
|
+
MCP server for **native mobile overlay automation**: Android UIAutomator via `adb`, iOS XCUITest via `xcrun simctl` (simulators, Mac only) and `go-ios` (physical devices). Owns the **Chrome Custom Tabs / SafariViewController OAuth** problem with the `solve_oauth_cct` composite tool — see [plan §5.5.1](../../docs/) for the full design.
|
|
4
|
+
|
|
5
|
+
Part of the **Flutter Ultra MCP** plugin for Claude Code. The server runs as one of the eight MCP processes registered in `.mcp.json`.
|
|
6
|
+
|
|
7
|
+
## Tool catalogue
|
|
8
|
+
|
|
9
|
+
| Tool | Class | Purpose |
|
|
10
|
+
| ------------------------------------------------------------- | ----- | ------------------------------------------------------- |
|
|
11
|
+
| `list_devices` | quick | Enumerate Android + iOS (sim + physical) devices |
|
|
12
|
+
| `dump_a11y_tree` | long | Dump UIAutomator XML / XCUITest a11y as structured tree |
|
|
13
|
+
| `wait_for_native_element` | long | Poll the a11y tree until a finder matches |
|
|
14
|
+
| `native_tap` | quick | Tap by coordinates or finder (bounds-center) |
|
|
15
|
+
| `native_type` | quick | Type into focused input |
|
|
16
|
+
| `native_swipe` | quick | Swipe between two coordinates |
|
|
17
|
+
| `native_back` / `_home` / `_app_switch` / `_open_settings` | quick | System keys / intents |
|
|
18
|
+
| `native_pin_lock` | quick | Toggle Android lock-task (kiosk mode) |
|
|
19
|
+
| `dismiss_permission_dialog` | quick | Smart allow/deny on runtime permission prompt |
|
|
20
|
+
| `native_permission_grant` / `_deny` | quick | `pm grant` / `pm revoke` |
|
|
21
|
+
| `take_device_screenshot` | quick | Base64 PNG via image content |
|
|
22
|
+
| `set_device_orientation` | quick | Portrait / landscape |
|
|
23
|
+
| `native_clipboard_set` / `_get` | quick | Device clipboard |
|
|
24
|
+
| `start_device_logs` / `poll_device_logs` / `stop_device_logs` | quick | Split-tool logcat / oslog tail |
|
|
25
|
+
| `solve_oauth_cct` | long | CCT/SVC OAuth via Playwright + deep-link dispatch |
|
|
26
|
+
|
|
27
|
+
All long-running tools are wrapped by the shared `runWithWatchdog` so the 60-second MCP client ceiling never kills them.
|
|
28
|
+
|
|
29
|
+
## Architecture
|
|
30
|
+
|
|
31
|
+
Every tool routes shell commands through a `Device` abstraction:
|
|
32
|
+
|
|
33
|
+
```text
|
|
34
|
+
Tool handler ─► registry.get(deviceId) ─► AndroidDevice ─► adb -s <udid> ...
|
|
35
|
+
─► IosSimDevice ─► xcrun simctl ...
|
|
36
|
+
─► IosPhysicalDev ─► go-ios ...
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
`AndroidDevice` / `IosSimDevice` / `IosPhysicalDevice` all implement the same `DeviceTransport` interface (`shell()`, `upload()`, `download()`, `isAlive()`, `dispose()`). A future remote-device adapter (WSL, SSH) drops in by implementing the same interface — no tool code changes.
|
|
40
|
+
|
|
41
|
+
```mermaid
|
|
42
|
+
flowchart LR
|
|
43
|
+
A[Tool handler] --> R[DeviceRegistry]
|
|
44
|
+
R -->|cached| D[DeviceTransport]
|
|
45
|
+
D --> Adb[AndroidDevice<br/>adb -s udid]
|
|
46
|
+
D --> Sim[IosSimDevice<br/>xcrun simctl]
|
|
47
|
+
D --> Phy[IosPhysicalDevice<br/>go-ios]
|
|
48
|
+
D -.future.-> Ssh[SshDevice<br/>ssh user@host adb ...]
|
|
49
|
+
D -.future.-> Wsl[WslDevice<br/>wsl -- adb ...]
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
## CCT OAuth bypass — `solve_oauth_cct`
|
|
53
|
+
|
|
54
|
+
Chrome Custom Tabs and SafariViewController hide the OAuth flow behind a system-owned web view that no UI driver can introspect. We do **not** try to drive CCT directly. Instead:
|
|
55
|
+
|
|
56
|
+
1. Launch Playwright Chromium with the provider's `authorize` URL.
|
|
57
|
+
2. Submit credentials (optional `fillFlow`) — pause for human MFA when needed.
|
|
58
|
+
3. Intercept the redirect to the app scheme (`com.example.myapp://callback?code=…`) before Chromium drops it as `ERR_UNKNOWN_URL_SCHEME`.
|
|
59
|
+
4. Deliver the same URL into the app via `Device.shell()`:
|
|
60
|
+
- Android: `adb shell am start -W -a android.intent.action.VIEW -d <url>`
|
|
61
|
+
- iOS Sim: `xcrun simctl openurl <udid> <url>`
|
|
62
|
+
- iOS physical: requires `idb` (out of scope for this server today).
|
|
63
|
+
|
|
64
|
+
The Flutter app's deep-link handler cannot distinguish the dispatched intent from a real CCT close — the same `Intent.ACTION_VIEW` arrives with the same URL. PKCE state stays valid because the app initiated the flow (its `code_verifier` is in its own memory).
|
|
65
|
+
|
|
66
|
+
## Configuration
|
|
67
|
+
|
|
68
|
+
| Env var | Default | Purpose |
|
|
69
|
+
| ----------------------------------------- | ------------ | --------------------------------------- |
|
|
70
|
+
| `FLUTTER_ULTRA_ADB` | `adb` | Path to the `adb` binary |
|
|
71
|
+
| `FLUTTER_ULTRA_XCRUN` | `xcrun` | Path to `xcrun` |
|
|
72
|
+
| `FLUTTER_ULTRA_GO_IOS_BIN` | `ios` | Path to the go-ios `ios` CLI |
|
|
73
|
+
| `FLUTTER_ULTRA_TOOL_TIMEOUT_<NAME_UPPER>` | tool default | Override per-tool watchdog ceiling (ms) |
|
|
74
|
+
| `FLUTTER_ULTRA_LOG_LEVEL` | `info` | `debug` / `info` / `warn` / `error` |
|
|
75
|
+
|
|
76
|
+
## Platform support
|
|
77
|
+
|
|
78
|
+
| Tool | Linux | macOS | Windows |
|
|
79
|
+
| ------------------------------------ | ----------------- | ----- | ------------------------------ |
|
|
80
|
+
| Android tools | yes (adb on PATH) | yes | yes |
|
|
81
|
+
| iOS Simulator tools | no | yes | no |
|
|
82
|
+
| iOS physical (go-ios) | yes | yes | yes (depends on go-ios binary) |
|
|
83
|
+
| `solve_oauth_cct` (Android dispatch) | yes | yes | yes |
|
|
84
|
+
| `solve_oauth_cct` (iOS sim dispatch) | no | yes | no |
|
|
85
|
+
|
|
86
|
+
iOS tools called on non-darwin hosts return a structured "unsupported" payload (per AC-NM3) — they do not crash the server.
|
|
87
|
+
|
|
88
|
+
## Acceptance criteria
|
|
89
|
+
|
|
90
|
+
- **AC-NM1** — `dismiss_permission_dialog(intent='allow')` clears a runtime permission within 3s on the Android emulator.
|
|
91
|
+
- **AC-NM2** — On iOS Simulator (Mac), `native_tap` on a SafariViewController OAuth login button completes the flow combined with `wait_for_url`.
|
|
92
|
+
- **AC-NM3** — iOS tools cleanly return `unsupported` on win32 / linux.
|
|
93
|
+
- **AC-NM4** — `solve_oauth_cct` end-to-end completes an OIDC OAuth flow against your identity provider on Android emulator within 60s.
|
|
94
|
+
|
|
95
|
+
## Development
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
npm install # at repo root
|
|
99
|
+
npm run -w @flutter-ultra/flutter-ultra-native-mobile build
|
|
100
|
+
npm run -w @flutter-ultra/flutter-ultra-native-mobile test
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
The unit-test suite focuses on the host-side parsers (`adb devices -l`, UIAutomator XML, simctl JSON) and the `Device` abstraction — full device-loop tests require an attached Android emulator and live in the cross-platform e2e workflow.
|