@mmerterden/multi-agent-toolkit-mcp 3.3.0 → 3.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +77 -0
- package/index.js +50 -64
- package/package.json +2 -2
- package/tools/a11y/index.js +335 -0
- package/ui-tree-dumper.swift +23 -1
package/CHANGELOG.md
CHANGED
|
@@ -15,6 +15,83 @@ Releases before this file exists are recorded in the git tags and commit history
|
|
|
15
15
|
|
|
16
16
|
---
|
|
17
17
|
|
|
18
|
+
## 3.5.0
|
|
19
|
+
|
|
20
|
+
### Added
|
|
21
|
+
|
|
22
|
+
- **`ios_accessibility_audit` checks the reading order.** A screen with perfect
|
|
23
|
+
labels is still unusable if VoiceOver walks it in the wrong order, and nothing
|
|
24
|
+
was looking. The audit compares the reading sequence against the visual layout
|
|
25
|
+
and reports the first element that arrives out of place.
|
|
26
|
+
|
|
27
|
+
The comparison is rows-then-columns, not a naive top-to-bottom sort: a home
|
|
28
|
+
screen is a grid, a toolbar is a row, and sorting by `y` alone calls every
|
|
29
|
+
correct grid broken. Elements share a row when their vertical extents overlap
|
|
30
|
+
by more than half the shorter one, which absorbs the few points of jitter real
|
|
31
|
+
layouts carry. `rtl: true` expects right-to-left within a row.
|
|
32
|
+
|
|
33
|
+
It is an `important` finding and never a `critical` one: a deliberate reading
|
|
34
|
+
order that differs from the visual one is legitimate, so this is a question
|
|
35
|
+
worth asking rather than a proven defect. Verified on a live home screen grid:
|
|
36
|
+
13 elements, reading order correct, no false positive.
|
|
37
|
+
|
|
38
|
+
- **Traits and hints.** The dumper now reads `AXRoleDescription` (what a screen
|
|
39
|
+
reader announces the element AS) and `AXHelp` (where the iOS
|
|
40
|
+
`accessibilityHint` bridges to). A control with no role announcement is read
|
|
41
|
+
out as bare text and the user cannot tell it is actionable, so that is an
|
|
42
|
+
`important` finding. A missing hint is not: hints are not required on every
|
|
43
|
+
element, so the result reports `hints_present` as a count and leaves the
|
|
44
|
+
judgement to the reader.
|
|
45
|
+
|
|
46
|
+
`AXSubrole` was probed first and carries nothing at all through this bridge,
|
|
47
|
+
so it is not read. Guessing at it would have produced a check that never fires.
|
|
48
|
+
|
|
49
|
+
### Note
|
|
50
|
+
|
|
51
|
+
Stage 2 of 4. Contrast, Dynamic Type and clipped text still cannot be reached
|
|
52
|
+
from a tree dump; they need Apple's own `XCUIAccessibilityAudit`, which is an
|
|
53
|
+
XCUITest run rather than a simctl call. The Android side is stage 4 and is
|
|
54
|
+
unverifiable on a machine with no adb.
|
|
55
|
+
|
|
56
|
+
## 3.4.0
|
|
57
|
+
|
|
58
|
+
### Fixed
|
|
59
|
+
|
|
60
|
+
- **The accessibility audits reported a clean screen they had never read.** On a
|
|
61
|
+
tree that came back empty they returned `elements_scanned: 0, total_issues: 0,
|
|
62
|
+
critical: 0, important: 0, warning: 0` - a full clean bill of health,
|
|
63
|
+
indistinguishable from an accessible screen, and the first reading is the one
|
|
64
|
+
anyone believes. Both audits now carry `measurable` and a `reason`, and an
|
|
65
|
+
unmeasurable run reports `total_issues: null`, never 0. A count of zero is a
|
|
66
|
+
measurement; none was taken.
|
|
67
|
+
|
|
68
|
+
- **The iOS tree was empty until the Simulator window had been activated.**
|
|
69
|
+
Measured on a live simulator: 1 node before activation, 28 after, and it stays
|
|
70
|
+
populated once the window is backgrounded again. The dumper now activates the
|
|
71
|
+
window itself. It also says what to do when Simulator.app is not running at
|
|
72
|
+
all, because booting a device with `simctl` is not enough - the accessibility
|
|
73
|
+
bridge lives in the UI app.
|
|
74
|
+
|
|
75
|
+
- **The iOS audit scored Simulator.app's own chrome as app findings.** The
|
|
76
|
+
dumped window carries the hardware buttons, the toolbar and the title text
|
|
77
|
+
beside the device screen, so Apple's 17x65pt Volume Up button was reported as
|
|
78
|
+
an app tap-target violation. The audit now scopes to the device screen, the
|
|
79
|
+
largest AXGroup child of the window. On a live Home screen this took the
|
|
80
|
+
result from 23 findings over 23 elements to 1 finding over 13: 22 of the 23
|
|
81
|
+
were the simulator's own interface.
|
|
82
|
+
|
|
83
|
+
- The scoring moved to `tools/a11y/` with a suite covering the degenerate tree,
|
|
84
|
+
the chrome exclusion, a scope that matches nothing, and the rule that an
|
|
85
|
+
element with no identifier is audited whatever the scope - you cannot scope
|
|
86
|
+
what you cannot identify, and skipping it would hide the finding that says so.
|
|
87
|
+
|
|
88
|
+
### Note
|
|
89
|
+
|
|
90
|
+
This is the first of four stages. It buys nothing new for a blind user yet; it
|
|
91
|
+
makes the existing checks honest so the next stages can be trusted. Reading
|
|
92
|
+
order, traits and hints come next, then Apple's own `XCUIAccessibilityAudit`
|
|
93
|
+
(contrast, Dynamic Type, clipped text), then the Android side.
|
|
94
|
+
|
|
18
95
|
## 3.3.0
|
|
19
96
|
|
|
20
97
|
### Added
|
package/index.js
CHANGED
|
@@ -30,6 +30,7 @@ import {
|
|
|
30
30
|
import { DESIGN_TOOLS, handleDesign } from "./tools/design-check/index.js";
|
|
31
31
|
import { parseLaunchOutput } from "./tools/launch-time/index.js";
|
|
32
32
|
import { parseLeaksOutput, parseMeminfoOutput, diffMeminfo } from "./tools/memory/index.js";
|
|
33
|
+
import { auditIosTree, auditAndroidDump } from "./tools/a11y/index.js";
|
|
33
34
|
import { interactiveElements } from "./tools/ui-inspect/index.js";
|
|
34
35
|
import { selectCrashReports } from "./tools/crash-logs/index.js";
|
|
35
36
|
import {
|
|
@@ -350,7 +351,7 @@ const IOS_TOOLS = [
|
|
|
350
351
|
{ name: "ios_get_app_container", description: "Get iOS app container path", inputSchema: { type: "object", properties: { bundle_id: { type: "string" }, container: { type: "string", enum: ["app", "data", "groups"] }, device_id: { type: "string" } }, required: ["bundle_id"] } },
|
|
351
352
|
{ name: "ios_erase_device", description: "Factory reset iOS simulator", inputSchema: { type: "object", properties: { device_id: { type: "string" } } } },
|
|
352
353
|
{ name: "ios_get_ui_tree", description: "Get iOS accessibility UI tree via macOS AX APIs. Pass `path` to write the raw dump to a file and return only its location - the full tree can be tens of KB.", inputSchema: { type: "object", properties: { max_depth: { type: "number" }, path: { type: "string", description: "Absolute file path to write the raw tree JSON to. The parent directory must already exist." } } } },
|
|
353
|
-
{ name: "ios_accessibility_audit", description: "Audit iOS app accessibility: missing labels,
|
|
354
|
+
{ name: "ios_accessibility_audit", description: "Audit iOS app accessibility on the booted simulator: missing labels, controls a screen reader cannot name, tap targets under 44pt, missing identifiers, and whether the reading order follows the visual layout. Reports measurable:false with a reason rather than a clean result when the tree could not be read - booting a device is not enough, Simulator.app must be running. Scopes to the device screen, never the simulator own chrome.", inputSchema: { type: "object", properties: { max_depth: { type: "number" }, scope: { type: "string", description: "Filter: only audit elements whose identifier starts with this prefix (e.g. 'login_', 'settings_'). Omit to audit all." }, rtl: { type: "boolean", description: "Expect right-to-left reading within a row. Default false." } } } },
|
|
354
355
|
{ name: "ios_biometric", description: "Simulate Face ID / Touch ID on iOS simulator (match or nomatch)", inputSchema: { type: "object", properties: { match: { type: "boolean", description: "true=success, false=failure" }, device_id: { type: "string" } }, required: ["match"] } },
|
|
355
356
|
{ name: "ios_archive_audit", description: "DEPRECATED - use ios_app_store_audit (18-rule deep scan). Lighter 6-check audit kept for backward compatibility; will be removed in the next major.", inputSchema: { type: "object", properties: { archive_path: { type: "string", description: "Path to .xcarchive" } }, required: ["archive_path"] } },
|
|
356
357
|
{ name: "ios_export_ipa", description: "Export a .xcarchive to a signed .ipa via xcodebuild -exportArchive. Generates the exportOptions.plist from the arguments (method defaults to app-store-connect), so callers do not have to hand-maintain one. Returns the .ipa path plus parsed errors; a run that exits 0 without producing an .ipa is reported as a failure. Pair with ios_testflight_validate for the pre-submission gate.", inputSchema: { type: "object", properties: { archive_path: { type: "string", description: "Absolute path to the .xcarchive" }, output_dir: { type: "string", description: "Directory to write the .ipa into" }, method: { type: "string", description: "Export method: app-store-connect (default) | release-testing | enterprise | development" }, team_id: { type: "string", description: "Apple Developer team ID" }, provisioning_profiles: { type: "object", description: "Map of bundleId -> provisioning profile name (manual signing)" }, signing_style: { type: "string", description: "automatic | manual" }, upload_symbols: { type: "boolean", description: "Include symbols (default true)" }, allow_provisioning_updates: { type: "boolean", description: "Off by default. Lets xcodebuild register devices and create/modify provisioning profiles in the developer account - a change on Apple's side, so it is opt-in" }, timeout_sec: { type: "number", description: "Default 900" } }, required: ["archive_path", "output_dir"] } },
|
|
@@ -500,38 +501,29 @@ async function handleIOS(name, args, ctx = {}) {
|
|
|
500
501
|
const script = join(__dirname, "ui-tree-dumper.swift");
|
|
501
502
|
if (!existsSync(script)) return "ui-tree-dumper.swift not found";
|
|
502
503
|
const depth = num(args.max_depth ?? 10, "max_depth");
|
|
503
|
-
const
|
|
504
|
-
|
|
504
|
+
const treeJson = run(`swift ${shq(script)} ${depth}`, { timeout: 30000 });
|
|
505
|
+
let tree = null;
|
|
505
506
|
try {
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
return JSON.stringify({ scope: scope || "all", elements_scanned: totalScanned, elements_skipped: totalSkipped, total_issues: issues.length, critical: issues.filter(i => i.severity === "critical").length, important: issues.filter(i => i.severity === "important").length, warning: issues.filter(i => i.severity === "warning").length, issues }, null, 2);
|
|
526
|
-
} catch (e) { return `ERROR parsing UI tree: ${e.message}\n\nRaw output:\n${treeJson?.substring(0, 500)}`; }
|
|
507
|
+
tree = JSON.parse(treeJson);
|
|
508
|
+
} catch {
|
|
509
|
+
tree = null;
|
|
510
|
+
}
|
|
511
|
+
const r = auditIosTree({ tree, scope: args.scope || null, rtl: args.rtl === true });
|
|
512
|
+
return JSON.stringify({
|
|
513
|
+
scope: r.scope,
|
|
514
|
+
measurable: r.measurable,
|
|
515
|
+
reason: r.reason,
|
|
516
|
+
elements_scanned: r.elementsScanned,
|
|
517
|
+
elements_skipped: r.elementsSkipped,
|
|
518
|
+
reading_order_ok: r.readingOrderOk ?? null,
|
|
519
|
+
hints_present: r.hintsPresent ?? null,
|
|
520
|
+
total_issues: r.totalIssues,
|
|
521
|
+
critical: r.critical,
|
|
522
|
+
important: r.important,
|
|
523
|
+
warning: r.warning,
|
|
524
|
+
issues: r.issues,
|
|
525
|
+
}, null, 2);
|
|
527
526
|
}
|
|
528
|
-
// `simctl keychain <device> biometric-enroll` / `biometric-match` do not
|
|
529
|
-
// exist - `keychain` supports only add-root-cert/add-cert/reset. The only
|
|
530
|
-
// available lever is the (undocumented) BiometricKit notification, which is
|
|
531
|
-
// posted via notifyutil inside the simulator. notifyutil exits 0 even when
|
|
532
|
-
// it cannot set or post the name, so its output has to be inspected: a
|
|
533
|
-
// "Failed with code N" line means the notification did not land, and that
|
|
534
|
-
// must be reported as a failure rather than as a simulated success.
|
|
535
527
|
case "ios_biometric": {
|
|
536
528
|
const d = iosDevice(args.device_id);
|
|
537
529
|
const action = args.match ? "match" : "nomatch";
|
|
@@ -1062,34 +1054,20 @@ async function handleAndroid(name, args, ctx = {}) {
|
|
|
1062
1054
|
const f = join(SCREENSHOT_DIR, `a11y_${Date.now()}.xml`);
|
|
1063
1055
|
run(`adb ${df} pull /sdcard/_mcp_a11y.xml ${shq(f)}`);
|
|
1064
1056
|
run(`adb ${df} shell rm /sdcard/_mcp_a11y.xml`);
|
|
1065
|
-
|
|
1066
|
-
const
|
|
1067
|
-
|
|
1068
|
-
|
|
1069
|
-
|
|
1070
|
-
|
|
1071
|
-
|
|
1072
|
-
|
|
1073
|
-
|
|
1074
|
-
|
|
1075
|
-
|
|
1076
|
-
|
|
1077
|
-
|
|
1078
|
-
|
|
1079
|
-
const bounds = node.match(/bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"/);
|
|
1080
|
-
if (clickable) {
|
|
1081
|
-
if (scope && rid && !rid.includes(scope)) { totalSkipped++; continue; }
|
|
1082
|
-
totalScanned++;
|
|
1083
|
-
if (!desc && !text) issues.push({ severity: "critical", issue: "Missing contentDescription", element: cls, resourceId: rid });
|
|
1084
|
-
if (!rid) issues.push({ severity: "warning", issue: "Missing resource-id (UI testing)", element: cls });
|
|
1085
|
-
if (bounds) {
|
|
1086
|
-
const w = parseInt(bounds[3]) - parseInt(bounds[1]);
|
|
1087
|
-
const h = parseInt(bounds[4]) - parseInt(bounds[2]);
|
|
1088
|
-
if (w < 48 || h < 48) issues.push({ severity: "important", issue: `Touch target too small: ${w}x${h}dp (min 48x48)`, element: cls, resourceId: rid });
|
|
1089
|
-
}
|
|
1090
|
-
}
|
|
1091
|
-
}
|
|
1092
|
-
return JSON.stringify({ scope: scope || "all", elements_scanned: totalScanned, elements_skipped: totalSkipped, total_issues: issues.length, critical: issues.filter(i => i.severity === "critical").length, important: issues.filter(i => i.severity === "important").length, warning: issues.filter(i => i.severity === "warning").length, issues }, null, 2);
|
|
1057
|
+
const xml = existsSync(f) ? readFileSync(f, "utf-8") : "";
|
|
1058
|
+
const r = auditAndroidDump({ xml, scope: args.scope || null });
|
|
1059
|
+
return JSON.stringify({
|
|
1060
|
+
scope: r.scope,
|
|
1061
|
+
measurable: r.measurable,
|
|
1062
|
+
reason: r.reason,
|
|
1063
|
+
elements_scanned: r.elementsScanned,
|
|
1064
|
+
elements_skipped: r.elementsSkipped,
|
|
1065
|
+
total_issues: r.totalIssues,
|
|
1066
|
+
critical: r.critical,
|
|
1067
|
+
important: r.important,
|
|
1068
|
+
warning: r.warning,
|
|
1069
|
+
issues: r.issues,
|
|
1070
|
+
}, null, 2);
|
|
1093
1071
|
}
|
|
1094
1072
|
case "android_launch_time": {
|
|
1095
1073
|
run(`adb ${df} shell am force-stop ${sanitizeId(args.package_name)}`);
|
|
@@ -1557,17 +1535,25 @@ const ISSUE_LIST = {
|
|
|
1557
1535
|
},
|
|
1558
1536
|
};
|
|
1559
1537
|
|
|
1538
|
+
// The counts are nullable on purpose. An audit that could not read the tree
|
|
1539
|
+
// reports null, never 0: a count of zero is a measurement, and none was taken.
|
|
1540
|
+
// `measurable` is the field to branch on; the counts are only meaningful when
|
|
1541
|
+
// it is true.
|
|
1560
1542
|
const ACCESSIBILITY_AUDIT_SCHEMA = {
|
|
1561
1543
|
type: "object",
|
|
1562
|
-
required: ["total_issues", "critical", "important", "warning", "issues"],
|
|
1544
|
+
required: ["measurable", "total_issues", "critical", "important", "warning", "issues"],
|
|
1563
1545
|
properties: {
|
|
1564
1546
|
scope: { type: "string" },
|
|
1547
|
+
measurable: { type: "boolean" },
|
|
1548
|
+
reason: { type: ["string", "null"] },
|
|
1565
1549
|
elements_scanned: { type: "integer" },
|
|
1566
1550
|
elements_skipped: { type: "integer" },
|
|
1567
|
-
|
|
1568
|
-
|
|
1569
|
-
|
|
1570
|
-
|
|
1551
|
+
reading_order_ok: { type: ["boolean", "null"] },
|
|
1552
|
+
hints_present: { type: ["integer", "null"] },
|
|
1553
|
+
total_issues: { type: ["integer", "null"] },
|
|
1554
|
+
critical: { type: ["integer", "null"] },
|
|
1555
|
+
important: { type: ["integer", "null"] },
|
|
1556
|
+
warning: { type: ["integer", "null"] },
|
|
1571
1557
|
issues: ISSUE_LIST,
|
|
1572
1558
|
},
|
|
1573
1559
|
};
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@mmerterden/multi-agent-toolkit-mcp",
|
|
3
|
-
"version": "3.
|
|
3
|
+
"version": "3.5.0",
|
|
4
4
|
"description": "MCP server for iOS Simulator, Android Emulator and headless web control. 86 tools: device automation (tap/swipe/type), accessibility audits, visual diff, crash logs, App Store / Play Store pre-submission compliance. Runs standalone over stdio with any MCP client.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "index.js",
|
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
},
|
|
11
11
|
"scripts": {
|
|
12
12
|
"start": "node index.js",
|
|
13
|
-
"test": "node --test tools/design-check/__tests__/design-check.test.mjs tools/design-check/__tests__/plan-determinism.test.mjs tools/ios-app-store-audit/__tests__/app-store-audit.test.mjs tools/ios-testflight/__tests__/testflight.test.mjs tools/ui-inspect/__tests__/ui-inspect.test.mjs tools/crash-logs/__tests__/crash-logs.test.mjs tools/launch-time/__tests__/launch-time.test.mjs tools/memory/__tests__/memory.test.mjs tools/offload/__tests__/offload.test.mjs __tests__/server-tools.test.mjs __tests__/injection.test.mjs",
|
|
13
|
+
"test": "node --test tools/design-check/__tests__/design-check.test.mjs tools/design-check/__tests__/plan-determinism.test.mjs tools/ios-app-store-audit/__tests__/app-store-audit.test.mjs tools/ios-testflight/__tests__/testflight.test.mjs tools/ui-inspect/__tests__/ui-inspect.test.mjs tools/crash-logs/__tests__/crash-logs.test.mjs tools/a11y/__tests__/a11y.test.mjs tools/launch-time/__tests__/launch-time.test.mjs tools/memory/__tests__/memory.test.mjs tools/offload/__tests__/offload.test.mjs __tests__/server-tools.test.mjs __tests__/injection.test.mjs",
|
|
14
14
|
"gates": "bash scripts/gates.sh"
|
|
15
15
|
},
|
|
16
16
|
"keywords": [
|
|
@@ -0,0 +1,335 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* a11y - accessibility audit scoring, split out of index.js so it can be tested.
|
|
3
|
+
*
|
|
4
|
+
* The audits used to return a full clean table for a tree they never read:
|
|
5
|
+
* `elements_scanned: 0, total_issues: 0, critical: 0, important: 0, warning: 0`.
|
|
6
|
+
* On iOS the tree comes from Simulator.app's host accessibility bridge, and that
|
|
7
|
+
* bridge stays empty until the Simulator window has been activated at least
|
|
8
|
+
* once, verified on a live simulator: 1 node before activation, 28 after. So
|
|
9
|
+
* "no issues" and "never looked" were the same answer, and the first is the one
|
|
10
|
+
* a reader believes.
|
|
11
|
+
*
|
|
12
|
+
* Every result therefore carries `measurable` and, when false, a `reason`. A
|
|
13
|
+
* result that could not be measured reports `totalIssues: null`, never 0.
|
|
14
|
+
*/
|
|
15
|
+
|
|
16
|
+
const IOS_INTERACTIVE_ROLES = new Set([
|
|
17
|
+
"AXButton",
|
|
18
|
+
"AXLink",
|
|
19
|
+
"AXTextField",
|
|
20
|
+
"AXTextArea",
|
|
21
|
+
"AXCheckBox",
|
|
22
|
+
"AXRadioButton",
|
|
23
|
+
"AXSlider",
|
|
24
|
+
"AXPopUpButton",
|
|
25
|
+
"AXMenuButton",
|
|
26
|
+
"AXSwitch",
|
|
27
|
+
]);
|
|
28
|
+
|
|
29
|
+
const IOS_MIN_TAP_PT = 44;
|
|
30
|
+
const ANDROID_MIN_TAP_DP = 48;
|
|
31
|
+
|
|
32
|
+
/**
|
|
33
|
+
* An accessibility tree with no children is not a screen without controls, it
|
|
34
|
+
* is a tree that was never populated. Callers must not score it.
|
|
35
|
+
*
|
|
36
|
+
* @param {object|null} tree - parsed ui-tree-dumper output
|
|
37
|
+
* @returns {boolean}
|
|
38
|
+
*/
|
|
39
|
+
export function isDegenerateTree(tree) {
|
|
40
|
+
if (!tree || typeof tree !== "object") return true;
|
|
41
|
+
if (tree.error) return true;
|
|
42
|
+
const kids = Array.isArray(tree.children) ? tree.children : [];
|
|
43
|
+
return kids.length === 0;
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
/**
|
|
47
|
+
* The device screen inside Simulator.app's window.
|
|
48
|
+
*
|
|
49
|
+
* The dumped window holds the simulator's own chrome as direct children -
|
|
50
|
+
* hardware buttons (Action, Volume Up, Sleep/Wake), the toolbar, the title
|
|
51
|
+
* text - and the iOS app underneath a single large AXGroup. Verified on a live
|
|
52
|
+
* simulator: seven chrome children beside one 402x873 AXGroup with the app in
|
|
53
|
+
* it. Auditing the window whole reports Apple's 17x35pt Volume button as an
|
|
54
|
+
* app tap-target violation, which is a false positive in exactly the place an
|
|
55
|
+
* accessibility report has to be trusted.
|
|
56
|
+
*
|
|
57
|
+
* @param {object} tree
|
|
58
|
+
* @returns {object|null} the screen subtree, or null when it cannot be found
|
|
59
|
+
*/
|
|
60
|
+
export function deviceScreenSubtree(tree) {
|
|
61
|
+
if (!tree || tree.role !== "AXWindow") return tree || null;
|
|
62
|
+
const groups = (tree.children || []).filter((c) => c && c.role === "AXGroup");
|
|
63
|
+
if (groups.length === 0) return null;
|
|
64
|
+
const area = (n) => (n.frame?.w || 0) * (n.frame?.h || 0);
|
|
65
|
+
return groups.reduce((best, c) => (area(c) > area(best) ? c : best), groups[0]);
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
/**
|
|
69
|
+
* Flatten the interactive elements in the order a screen reader would visit
|
|
70
|
+
* them, which is the order the accessibility tree presents them in.
|
|
71
|
+
*
|
|
72
|
+
* @param {object} node
|
|
73
|
+
* @param {Set<string>} roles
|
|
74
|
+
* @returns {object[]}
|
|
75
|
+
*/
|
|
76
|
+
function readingSequence(node, roles) {
|
|
77
|
+
const out = [];
|
|
78
|
+
const visit = (n) => {
|
|
79
|
+
if (!n || typeof n !== "object") return;
|
|
80
|
+
if (roles.has(n.role)) out.push(n);
|
|
81
|
+
(n.children || []).forEach(visit);
|
|
82
|
+
};
|
|
83
|
+
visit(node);
|
|
84
|
+
return out;
|
|
85
|
+
}
|
|
86
|
+
|
|
87
|
+
/**
|
|
88
|
+
* Group elements into visual rows.
|
|
89
|
+
*
|
|
90
|
+
* A screen is not a single top-to-bottom column. A home screen is a grid, a
|
|
91
|
+
* toolbar is a row, and comparing the reading order against a naive
|
|
92
|
+
* top-to-bottom sort reports every grid as broken. Two elements share a row
|
|
93
|
+
* when their vertical extents overlap by more than half the shorter one.
|
|
94
|
+
*
|
|
95
|
+
* @param {object[]} els
|
|
96
|
+
* @returns {object[][]} rows, top to bottom, each sorted left to right
|
|
97
|
+
*/
|
|
98
|
+
export function visualRows(els) {
|
|
99
|
+
const sorted = [...els].sort((a, b) => (a.frame?.y || 0) - (b.frame?.y || 0));
|
|
100
|
+
const rows = [];
|
|
101
|
+
for (const el of sorted) {
|
|
102
|
+
const y = el.frame?.y || 0;
|
|
103
|
+
const h = el.frame?.h || 0;
|
|
104
|
+
const row = rows.find((r) => {
|
|
105
|
+
const ry = r.top;
|
|
106
|
+
const rh = r.height;
|
|
107
|
+
const overlap = Math.min(y + h, ry + rh) - Math.max(y, ry);
|
|
108
|
+
return overlap > Math.min(h, rh) / 2;
|
|
109
|
+
});
|
|
110
|
+
if (row) {
|
|
111
|
+
row.items.push(el);
|
|
112
|
+
row.top = Math.min(row.top, y);
|
|
113
|
+
row.height = Math.max(row.height, h);
|
|
114
|
+
} else {
|
|
115
|
+
rows.push({ top: y, height: h, items: [el] });
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
return rows.map((r) => r.items.sort((a, b) => (a.frame?.x || 0) - (b.frame?.x || 0)));
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
/**
|
|
122
|
+
* Does the reading order follow the visual layout?
|
|
123
|
+
*
|
|
124
|
+
* The check is deliberately coarse: it compares the reading sequence against
|
|
125
|
+
* rows-then-columns, and reports the first element that arrives out of place.
|
|
126
|
+
* A screen can carry a deliberate reading order that differs from the visual
|
|
127
|
+
* one, so this is an `important` finding and never a `critical` one: it is a
|
|
128
|
+
* question worth asking, not a proven defect.
|
|
129
|
+
*
|
|
130
|
+
* @param {object[]} sequence - elements in reading order
|
|
131
|
+
* @param {boolean} rtl - right-to-left layout
|
|
132
|
+
* @returns {{ok: boolean, firstOutOfPlace: object|null, expectedIndex: number|null}}
|
|
133
|
+
*/
|
|
134
|
+
export function checkReadingOrder(sequence, rtl = false) {
|
|
135
|
+
if (sequence.length < 2) return { ok: true, firstOutOfPlace: null, expectedIndex: null };
|
|
136
|
+
const rows = visualRows(sequence);
|
|
137
|
+
const expected = rows.flatMap((r) => (rtl ? [...r].reverse() : r));
|
|
138
|
+
for (let i = 0; i < sequence.length; i++) {
|
|
139
|
+
if (sequence[i] !== expected[i]) {
|
|
140
|
+
return { ok: false, firstOutOfPlace: sequence[i], expectedIndex: expected.indexOf(sequence[i]) };
|
|
141
|
+
}
|
|
142
|
+
}
|
|
143
|
+
return { ok: true, firstOutOfPlace: null, expectedIndex: null };
|
|
144
|
+
}
|
|
145
|
+
|
|
146
|
+
/**
|
|
147
|
+
* @param {object} params
|
|
148
|
+
* @param {object|null} params.tree - parsed ui-tree-dumper output
|
|
149
|
+
* @param {string|null} [params.scope] - only audit identifiers with this prefix
|
|
150
|
+
* @returns {{measurable: boolean, reason: string|null, scope: string,
|
|
151
|
+
* elementsScanned: number, elementsSkipped: number,
|
|
152
|
+
* totalIssues: number|null, critical: number|null,
|
|
153
|
+
* important: number|null, warning: number|null, issues: object[]}}
|
|
154
|
+
*/
|
|
155
|
+
export function auditIosTree({ tree, scope = null, rtl = false }) {
|
|
156
|
+
if (isDegenerateTree(tree)) {
|
|
157
|
+
return unmeasurable(
|
|
158
|
+
scope,
|
|
159
|
+
tree && tree.error
|
|
160
|
+
? String(tree.error)
|
|
161
|
+
: "the accessibility tree came back empty; open Simulator.app and bring its window to the front, then retry - the host bridge does not populate until the window has been activated once",
|
|
162
|
+
);
|
|
163
|
+
}
|
|
164
|
+
|
|
165
|
+
const screen = deviceScreenSubtree(tree);
|
|
166
|
+
if (!screen) {
|
|
167
|
+
return unmeasurable(
|
|
168
|
+
scope,
|
|
169
|
+
"the window held no device screen group; the simulator may still be starting up",
|
|
170
|
+
);
|
|
171
|
+
}
|
|
172
|
+
|
|
173
|
+
const issues = [];
|
|
174
|
+
let elementsScanned = 0;
|
|
175
|
+
let elementsSkipped = 0;
|
|
176
|
+
const sequence = [];
|
|
177
|
+
|
|
178
|
+
const visit = (node, path = "") => {
|
|
179
|
+
if (!node || typeof node !== "object") return;
|
|
180
|
+
const loc = path ? `${path} > ${node.role}` : node.role;
|
|
181
|
+
const w = node.frame?.w || 0;
|
|
182
|
+
const h = node.frame?.h || 0;
|
|
183
|
+
|
|
184
|
+
if (IOS_INTERACTIVE_ROLES.has(node.role)) {
|
|
185
|
+
if (scope && node.identifier && !node.identifier.startsWith(scope)) {
|
|
186
|
+
elementsSkipped++;
|
|
187
|
+
(node.children || []).forEach((c) => visit(c, loc));
|
|
188
|
+
return;
|
|
189
|
+
}
|
|
190
|
+
elementsScanned++;
|
|
191
|
+
sequence.push(node);
|
|
192
|
+
// What a screen reader announces the element AS. Without it a control is
|
|
193
|
+
// read out as bare text and the user cannot tell it is actionable.
|
|
194
|
+
if (!node.roleDescription) {
|
|
195
|
+
issues.push({ severity: "important", issue: "No role announced (VoiceOver cannot say what this control is)", element: loc, identifier: node.identifier || null });
|
|
196
|
+
}
|
|
197
|
+
if (!node.title && !node.description && !node.value) {
|
|
198
|
+
issues.push({ severity: "critical", issue: "Missing accessibility label", element: loc, identifier: node.identifier || null });
|
|
199
|
+
}
|
|
200
|
+
if (!node.identifier) {
|
|
201
|
+
issues.push({ severity: "warning", issue: "Missing accessibility identifier (UI testing)", element: loc });
|
|
202
|
+
}
|
|
203
|
+
if (w > 0 && h > 0 && (w < IOS_MIN_TAP_PT || h < IOS_MIN_TAP_PT)) {
|
|
204
|
+
issues.push({ severity: "important", issue: `Tap target too small: ${Math.round(w)}x${Math.round(h)}pt (min ${IOS_MIN_TAP_PT}x${IOS_MIN_TAP_PT})`, element: loc, identifier: node.identifier || null });
|
|
205
|
+
}
|
|
206
|
+
}
|
|
207
|
+
(node.children || []).forEach((c) => visit(c, loc));
|
|
208
|
+
};
|
|
209
|
+
visit(screen);
|
|
210
|
+
|
|
211
|
+
// A populated window whose interactive elements were all filtered out is a
|
|
212
|
+
// scope that matched nothing, not a clean screen.
|
|
213
|
+
if (elementsScanned === 0) {
|
|
214
|
+
return unmeasurable(
|
|
215
|
+
scope,
|
|
216
|
+
scope
|
|
217
|
+
? `no interactive element matched scope "${scope}"`
|
|
218
|
+
: "the tree was readable but held no interactive elements to audit",
|
|
219
|
+
elementsSkipped,
|
|
220
|
+
);
|
|
221
|
+
}
|
|
222
|
+
|
|
223
|
+
const order = checkReadingOrder(sequence, rtl);
|
|
224
|
+
if (!order.ok) {
|
|
225
|
+
const el = order.firstOutOfPlace;
|
|
226
|
+
issues.push({
|
|
227
|
+
severity: "important",
|
|
228
|
+
issue: `Reading order does not follow the visual layout: this element is read at position ${sequence.indexOf(el) + 1} but sits at position ${order.expectedIndex + 1} on screen`,
|
|
229
|
+
element: el.role,
|
|
230
|
+
identifier: el.identifier || null,
|
|
231
|
+
label: el.title || el.description || el.value || null,
|
|
232
|
+
});
|
|
233
|
+
}
|
|
234
|
+
|
|
235
|
+
return {
|
|
236
|
+
measurable: true,
|
|
237
|
+
reason: null,
|
|
238
|
+
scope: scope || "all",
|
|
239
|
+
elementsScanned,
|
|
240
|
+
elementsSkipped,
|
|
241
|
+
readingOrderOk: order.ok,
|
|
242
|
+
hintsPresent: sequence.filter((n) => n.help).length,
|
|
243
|
+
...tally(issues),
|
|
244
|
+
};
|
|
245
|
+
}
|
|
246
|
+
|
|
247
|
+
/**
|
|
248
|
+
* @param {object} params
|
|
249
|
+
* @param {string} params.xml - uiautomator dump output
|
|
250
|
+
* @param {string|null} [params.scope] - only audit resource-ids containing this
|
|
251
|
+
* @returns {object} same shape as auditIosTree
|
|
252
|
+
*/
|
|
253
|
+
export function auditAndroidDump({ xml, scope = null }) {
|
|
254
|
+
const text = typeof xml === "string" ? xml : "";
|
|
255
|
+
if (!/<node\b/.test(text)) {
|
|
256
|
+
return unmeasurable(scope, "the uiautomator dump held no nodes; the dump failed or the screen was not ready");
|
|
257
|
+
}
|
|
258
|
+
|
|
259
|
+
const issues = [];
|
|
260
|
+
let elementsScanned = 0;
|
|
261
|
+
let elementsSkipped = 0;
|
|
262
|
+
|
|
263
|
+
for (const match of text.matchAll(/<node[^>]*>/g)) {
|
|
264
|
+
const node = match[0];
|
|
265
|
+
const attr = (name) => node.match(new RegExp(`${name}="([^"]*)"`))?.[1] || "";
|
|
266
|
+
if (!node.includes('clickable="true"')) continue;
|
|
267
|
+
|
|
268
|
+
const cls = attr("class");
|
|
269
|
+
const desc = attr("content-desc");
|
|
270
|
+
const rid = attr("resource-id");
|
|
271
|
+
const label = attr("text");
|
|
272
|
+
|
|
273
|
+
if (scope && rid && !rid.includes(scope)) {
|
|
274
|
+
elementsSkipped++;
|
|
275
|
+
continue;
|
|
276
|
+
}
|
|
277
|
+
elementsScanned++;
|
|
278
|
+
|
|
279
|
+
if (!desc && !label) issues.push({ severity: "critical", issue: "Missing contentDescription", element: cls, resourceId: rid || null });
|
|
280
|
+
if (!rid) issues.push({ severity: "warning", issue: "Missing resource-id (UI testing)", element: cls });
|
|
281
|
+
|
|
282
|
+
const bounds = node.match(/bounds="\[(\d+),(\d+)\]\[(\d+),(\d+)\]"/);
|
|
283
|
+
if (bounds) {
|
|
284
|
+
const w = parseInt(bounds[3], 10) - parseInt(bounds[1], 10);
|
|
285
|
+
const h = parseInt(bounds[4], 10) - parseInt(bounds[2], 10);
|
|
286
|
+
if (w < ANDROID_MIN_TAP_DP || h < ANDROID_MIN_TAP_DP) {
|
|
287
|
+
issues.push({ severity: "important", issue: `Touch target too small: ${w}x${h}dp (min ${ANDROID_MIN_TAP_DP}x${ANDROID_MIN_TAP_DP})`, element: cls, resourceId: rid || null });
|
|
288
|
+
}
|
|
289
|
+
}
|
|
290
|
+
}
|
|
291
|
+
|
|
292
|
+
if (elementsScanned === 0) {
|
|
293
|
+
return unmeasurable(
|
|
294
|
+
scope,
|
|
295
|
+
scope ? `no clickable element matched scope "${scope}"` : "the dump was readable but held no clickable elements to audit",
|
|
296
|
+
elementsSkipped,
|
|
297
|
+
);
|
|
298
|
+
}
|
|
299
|
+
|
|
300
|
+
return {
|
|
301
|
+
measurable: true,
|
|
302
|
+
reason: null,
|
|
303
|
+
scope: scope || "all",
|
|
304
|
+
elementsScanned,
|
|
305
|
+
elementsSkipped,
|
|
306
|
+
...tally(issues),
|
|
307
|
+
};
|
|
308
|
+
}
|
|
309
|
+
|
|
310
|
+
function tally(issues) {
|
|
311
|
+
const by = (s) => issues.filter((i) => i.severity === s).length;
|
|
312
|
+
return {
|
|
313
|
+
totalIssues: issues.length,
|
|
314
|
+
critical: by("critical"),
|
|
315
|
+
important: by("important"),
|
|
316
|
+
warning: by("warning"),
|
|
317
|
+
issues,
|
|
318
|
+
};
|
|
319
|
+
}
|
|
320
|
+
|
|
321
|
+
function unmeasurable(scope, reason, elementsSkipped = 0) {
|
|
322
|
+
return {
|
|
323
|
+
measurable: false,
|
|
324
|
+
reason,
|
|
325
|
+
scope: scope || "all",
|
|
326
|
+
elementsScanned: 0,
|
|
327
|
+
elementsSkipped,
|
|
328
|
+
// null rather than 0: a count of zero is a measurement, and none was taken.
|
|
329
|
+
totalIssues: null,
|
|
330
|
+
critical: null,
|
|
331
|
+
important: null,
|
|
332
|
+
warning: null,
|
|
333
|
+
issues: [],
|
|
334
|
+
};
|
|
335
|
+
}
|
package/ui-tree-dumper.swift
CHANGED
|
@@ -9,6 +9,12 @@ import Foundation
|
|
|
9
9
|
|
|
10
10
|
struct AXNode: Codable {
|
|
11
11
|
let role: String
|
|
12
|
+
// What a screen reader announces the element AS. Probed on a live simulator:
|
|
13
|
+
// AXRoleDescription carries "button" and friends, AXSubrole carries nothing
|
|
14
|
+
// at all through this bridge, so only the former is read.
|
|
15
|
+
let roleDescription: String?
|
|
16
|
+
// The iOS accessibilityHint bridges to AXHelp ("Double tap to open").
|
|
17
|
+
let help: String?
|
|
12
18
|
let title: String?
|
|
13
19
|
let value: String?
|
|
14
20
|
let description: String?
|
|
@@ -53,6 +59,8 @@ func dumpElement(_ element: AXUIElement, depth: Int, maxDepth: Int) -> AXNode? {
|
|
|
53
59
|
let value = getString(element, kAXValueAttribute)
|
|
54
60
|
let desc = getString(element, kAXDescriptionAttribute)
|
|
55
61
|
let identifier = getString(element, kAXIdentifierAttribute)
|
|
62
|
+
let roleDescription = getString(element, "AXRoleDescription")
|
|
63
|
+
let help = getString(element, "AXHelp")
|
|
56
64
|
let enabled = getBool(element, kAXEnabledAttribute)
|
|
57
65
|
let focused = getBool(element, kAXFocusedAttribute)
|
|
58
66
|
let frame = getFrame(element)
|
|
@@ -75,6 +83,8 @@ func dumpElement(_ element: AXUIElement, depth: Int, maxDepth: Int) -> AXNode? {
|
|
|
75
83
|
|
|
76
84
|
return AXNode(
|
|
77
85
|
role: role,
|
|
86
|
+
roleDescription: roleDescription,
|
|
87
|
+
help: help,
|
|
78
88
|
title: title,
|
|
79
89
|
value: value,
|
|
80
90
|
description: desc,
|
|
@@ -86,10 +96,22 @@ func dumpElement(_ element: AXUIElement, depth: Int, maxDepth: Int) -> AXNode? {
|
|
|
86
96
|
)
|
|
87
97
|
}
|
|
88
98
|
|
|
99
|
+
// The host accessibility bridge does not populate the iOS app's elements until
|
|
100
|
+
// the Simulator window has been activated at least once. Measured on a live
|
|
101
|
+
// simulator: 1 node before activation, 28 after, and it stays populated once
|
|
102
|
+
// the window has been backgrounded again. Without this the tree comes back
|
|
103
|
+
// holding only the window, which reads as a screen with no controls.
|
|
104
|
+
func activateSimulator(_ app: NSRunningApplication) {
|
|
105
|
+
if app.isActive { return }
|
|
106
|
+
app.activate(options: [])
|
|
107
|
+
Thread.sleep(forTimeInterval: 1.2)
|
|
108
|
+
}
|
|
109
|
+
|
|
89
110
|
func findSimulatorWindow() -> AXUIElement? {
|
|
90
111
|
let apps = NSWorkspace.shared.runningApplications
|
|
91
112
|
for app in apps {
|
|
92
113
|
if app.bundleIdentifier == "com.apple.iphonesimulator" {
|
|
114
|
+
activateSimulator(app)
|
|
93
115
|
let axApp = AXUIElementCreateApplication(app.processIdentifier)
|
|
94
116
|
var windows: AnyObject?
|
|
95
117
|
AXUIElementCopyAttributeValue(axApp, kAXWindowsAttribute as CFString, &windows)
|
|
@@ -105,7 +127,7 @@ func findSimulatorWindow() -> AXUIElement? {
|
|
|
105
127
|
let maxDepth = CommandLine.arguments.count > 1 ? Int(CommandLine.arguments[1]) ?? 10 : 10
|
|
106
128
|
|
|
107
129
|
guard let simWindow = findSimulatorWindow() else {
|
|
108
|
-
let error = ["error": "Simulator not running or no window
|
|
130
|
+
let error = ["error": "Simulator.app is not running or has no window. Booting a device with simctl is not enough - the accessibility bridge lives in the Simulator UI app. Run: open -a Simulator"]
|
|
109
131
|
let data = try! JSONSerialization.data(withJSONObject: error)
|
|
110
132
|
FileHandle.standardOutput.write(data)
|
|
111
133
|
exit(1)
|