digital-ai-testing-mcp 2.0.3 → 2.0.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -6,23 +6,28 @@
6
6
 
7
7
  # Digital.ai Testing — MCP Server
8
8
 
9
- An MCP (Model Context Protocol) server that connects AI assistants to a Digital.ai Testing device farm. The server exposes **202 tools**, **2 resources**, and **7 prompts** covering 27 capability areas: device management, test execution (including Espresso, XCUITest and Maestro test runs), app lifecycle, reporting and root-cause analysis, analytics, performance, project administration, interactive mobile inspection, interactive browser inspection, and more.
9
+ Talk to your Digital.ai Testing device farm from your AI assistant. This MCP (Model Context Protocol) server connects AI clients such as Claude, GitHub Copilot and Cursor to Digital.ai Testing, so you can ask in plain language to find devices, install apps, explore an app on a real device, generate tests, run test suites, investigate failures and compare performance.
10
+
11
+ It provides **202 tools**, **2 resources** and **7 prompts** across 27 capability areas: device management, test execution (including Espresso, XCUITest and Maestro test runs), app lifecycle, reporting and root-cause analysis, analytics, performance, project administration, interactive mobile inspection, interactive browser inspection, and more.
10
12
 
11
13
  ---
12
14
 
13
15
  ## Quick Start
14
16
 
17
+ **1. Check that Node.js 22 or later is installed** (npm comes with it). If it isn't, install the LTS version from https://nodejs.org first.
18
+
15
19
  ```bash
16
- # 0. Confirm Node.js 22+ is installed (installs npm alongside it)
17
20
  node --version
18
21
  npm --version
19
- # Don't have it? Install from https://nodejs.org (LTS) first.
22
+ ```
23
+
24
+ **2. Install the MCP server.**
20
25
 
21
- # 1. Install the MCP server
26
+ ```bash
22
27
  npm install -g digital-ai-testing-mcp
23
28
  ```
24
29
 
25
- Add to your AI client (Claude Desktop shown — see [Connecting AI Clients](#connecting-ai-clients) for VS Code, JetBrains, Copilot, and Cursor):
30
+ **3. Add it to your AI client** with your tenant URL and [access key](#access-keys). Claude Desktop is shown here; see [Connecting AI Clients](#connecting-ai-clients) for VS Code, JetBrains, Copilot and Cursor.
26
31
 
27
32
  ```json
28
33
  {
@@ -40,7 +45,7 @@ Add to your AI client (Claude Desktop shown — see [Connecting AI Clients](#con
40
45
  }
41
46
  ```
42
47
 
43
- Then ask: *"Show me the overall health of the device farm."*
48
+ **4. Restart the client and ask:** *"Show me the overall health of the device farm."*
44
49
 
45
50
  > Prefer a container? See [Option B — Pull from GitHub Container Registry](#option-b--pull-from-github-container-registry).
46
51
 
@@ -48,122 +53,51 @@ Then ask: *"Show me the overall health of the device farm."*
48
53
 
49
54
  ## Table of Contents
50
55
 
56
+ **Get started**
51
57
  - [Quick Start](#quick-start)
52
- - [Prerequisites](#prerequisites)
53
- - [Access Keys](#access-keys)
54
- - [Installation](#installation)
55
- - [Configuration](#configuration)
56
- - [Toolsets](#toolsets-reducing-context-size)
57
- - [Debug mode](#debug-mode-remediation-notes)
58
- - [Connecting AI Clients](#connecting-ai-clients)
59
- - [Claude Desktop](#claude-desktop)
60
- - [Claude Code (VS Code)](#claude-code-vs-code)
61
- - [Claude Code (JetBrains / Android Studio)](#claude-code-jetbrains--android-studio)
62
- - [GitHub Copilot (VS Code)](#github-copilot-vs-code)
63
- - [Cursor](#cursor)
64
- - [Example Prompts](#example-prompts)
65
- - [Capabilities](#capabilities) — full per-tool reference in [docs/tools.md](docs/tools.md)
66
- - [Workflow Reference](#workflow-reference)
67
- - [POC Lifecycle](#poc-lifecycle)
68
- - [Project Lifecycle](#project-lifecycle)
69
- - [Boilerplate Generation](#boilerplate-generation)
70
- - [Recommended agent guardrails](docs/recommended-agent-guardrails.md)
58
+ - [Installation](#installation) — [npm](#option-a--install-via-npm-recommended) · [Docker image](#option-b--pull-from-github-container-registry) · [from source](#option-c--build-from-source) · [Upgrading](#upgrading)
59
+ - [Connecting AI Clients](#connecting-ai-clients) — [Claude Desktop](#claude-desktop) · [Claude Code (VS Code)](#claude-code-vs-code) · [Claude Code (JetBrains)](#claude-code-jetbrains--android-studio) · [GitHub Copilot](#github-copilot-vs-code) · [Cursor](#cursor) · [Docker configs](#docker-client-configuration)
60
+ - [Access Keys](#access-keys) — access levels, finding your key, multiple profiles
61
+ - [Configuration](#configuration) — [Toolsets](#toolsets-reducing-context-size) · [Debug mode](#debug-mode-remediation-notes)
62
+
63
+ **Use it**
64
+ - [What You Can Do](#what-you-can-do) — [example prompts](#example-prompts) · [tasks at a glance](#tasks-at-a-glance) · [all capability areas](#all-capability-areas)
65
+ - [Creating Tests](#creating-tests) — [mobile](#mobile-app-tests-get_test_boilerplate) · [Test Orchestrator](#test-orchestrator-default-for-java-on-appium-server) · [web](#web-tests-get_web_test_boilerplate) · [validating a test](#check-a-test-before-you-deliver-it-validate_test_script)
66
+ - [Test Results and Root Cause Analysis](#test-results-and-root-cause-analysis)
71
67
  - [Performance Comparison Reports](#performance-comparison-reports)
72
- - [Reference](#reference)
73
- - [Response Format](#response-format)
74
- - [Understanding maxResults](#understanding-maxresults)
75
- - [List Filters & Sorting](#list-filters--sorting)
76
- - [Test Reporting Schema](#test-reporting-schema)
77
- - [Local File Transfer](#local-file-transfer-docker--remote-deployments-only)
68
+ - [Workflow Reference](#workflow-reference) — [POC lifecycle](#poc-lifecycle) · [Project lifecycle](#project-lifecycle)
78
69
  - [Safety Guards](#safety-guards)
79
- - [Development](#development)
70
+
71
+ **Help and reference**
80
72
  - [Troubleshooting](#troubleshooting)
73
+ - [Reference](#reference) — [response format](#response-format) · [maxResults](#understanding-maxresults) · [list filters](#list-filters--sorting) · [local file transfer (Docker)](#local-file-transfer-docker--remote-deployments-only)
81
74
  - [Known Limitations](#known-limitations)
75
+ - [Development](#development)
82
76
  - [License](#license)
83
77
 
84
78
  ---
85
79
 
86
- ## Prerequisites
87
-
88
- - **Node.js 22+** — required for [Option A](#option-a--install-via-npm-recommended) (recommended — no Docker needed)
89
- - **Docker** — required for [Option B](#option-b--pull-from-github-container-registry) or [Option C](#option-c--build-from-source) (container-based install)
90
- - **Digital.ai Testing account** with a valid access key
91
-
92
- ---
93
-
94
- ## Access Keys
95
-
96
- Your Digital.ai access key determines what the MCP server can do on your behalf. There are three access levels.
97
-
98
- | Access level | Access |
99
- |---|---|
100
- | **Cloud Admin** | All tools: device management, user provisioning, project administration, infrastructure, performance data |
101
- | **Project Admin** | Scoped to one project. Can manage device tags, view project admin settings, list users. v2 API tools (agents, regions, license data) return 403. |
102
- | **Project User** | Scoped to one project. Read/test operations only. Cannot manage tags or access admin settings; deleting reports requires the project's `allowUsersDeleteTests` setting. |
103
-
104
- > **The key format does not tell you the access level.** Older keys are long `eyJ…` tokens; since Digital.ai 24.11 newly generated keys use a shorter `aut_1_…` token, and that applies to Cloud Admins too. The MCP server therefore detects your access level from the API (`GET /api/v1/users/my-account-info` → `Admin` / `ProjectAdmin` / `User`) and never from the key's prefix. `get_server_info` and `list_environments` show what was detected for each profile.
105
-
106
- When a Cloud Admin tool is called with a project-level role, the MCP returns a plain-language error explaining what happened — and, if a Cloud Admin profile is known, a ready-to-use `switch_environment(...)` command.
107
-
108
- ### Finding your key
109
-
110
- 1. Log in to the Digital.ai Testing portal
111
- 2. Click your **name or avatar** in the top-right corner
112
- 3. Select **Access Key**
113
-
114
- The key shown is tied to the **project currently selected in the portal**. If you belong to multiple projects, each has a separate key.
115
-
116
- ### Using multiple keys
117
-
118
- Configure named profiles in `.env` to switch contexts at runtime without editing files:
119
-
120
- ```
121
- # Default connection (typically Cloud Admin credentials — either key format works)
122
- DIGITAL_AI_BASE_URL=https://your-tenant.experitest.com
123
- DIGITAL_AI_ACCESS_KEY=...your-cloud-admin-key...
124
-
125
- # Project-scoped profiles
126
- DAI_PROFILE_QA_URL=https://your-tenant.experitest.com
127
- DAI_PROFILE_QA_KEY=...your-qa-key...
128
-
129
- DAI_PROFILE_STAGING_URL=https://your-tenant.experitest.com
130
- DAI_PROFILE_STAGING_KEY=...your-staging-key...
131
- ```
132
-
133
- Then ask your AI:
134
- - *"What environments are configured?"* → `list_environments`
135
- - *"Switch to QA"* → `switch_environment("qa")`
136
- - *"Switch back to Cloud Admin"* → `switch_environment("default")`
137
-
138
- The switch takes effect immediately — no restart needed.
139
-
140
- > **Security:** Never commit `.env` to source control. Each team member should use their own key obtained from the portal under their own account.
80
+ ## Installation
141
81
 
142
- ---
82
+ **What you need:**
143
83
 
144
- ## Installation
84
+ - A **Digital.ai Testing account** with a valid [access key](#access-keys).
85
+ - **Node.js 22+** for [Option A](#option-a--install-via-npm-recommended) (recommended — no Docker needed), **or**
86
+ - **Docker** for [Option B](#option-b--pull-from-github-container-registry) or [Option C](#option-c--build-from-source) (container-based install).
145
87
 
146
88
  ### Option A — Install via npm (Recommended)
147
89
 
148
- No Docker required — runs directly under Node.js.
90
+ No Docker required — the server runs directly under Node.js.
149
91
 
150
92
  ```bash
151
93
  npm install -g digital-ai-testing-mcp
152
94
  ```
153
95
 
154
- This installs a `digital-ai-testing-mcp` command onto your `PATH`. Use it directly as the `"command"` in your AI client configuration below — no image name or `.env` file path needed; environment variables go directly in the client's `env` block instead (see [Configuration](#configuration)).
96
+ This puts a `digital-ai-testing-mcp` command on your `PATH`. Use it as the `"command"` in your [AI client configuration](#connecting-ai-clients). You don't need an image name or a `.env` file: settings go in the client's `env` block (see [Configuration](#configuration)).
155
97
 
156
- Set `MCP_DEPLOYMENT_MODE=local` alongside your credentials (already included in the config blocks below). This tells the server its filesystem is your own machine — tool descriptions and a handful of upload/download behaviors adjust accordingly (no Docker/volume-mount caveats, no path-locality warnings), instead of defaulting to the cautious Docker-deployment framing.
98
+ To run without a global install, use `"command": "npx"` with `"args": ["-y", "digital-ai-testing-mcp"]` instead — npx downloads and runs the latest published version.
157
99
 
158
- **Upgrading to the latest release:**
159
-
160
- ```bash
161
- npm install -g digital-ai-testing-mcp@latest
162
- ```
163
-
164
- Then restart your AI client (or reconnect the MCP server from its settings panel) so it respawns the process — npm installing a new version doesn't affect an already-running process. Confirm the upgrade worked by asking your AI to call `get_server_info`, which reports the running version.
165
-
166
- ---
100
+ Set `MCP_DEPLOYMENT_MODE=local` alongside your credentials (the client examples below already include it). It tells the server that its filesystem is your own machine, so tool descriptions and a few upload/download behaviors drop the Docker-specific caveats and path warnings.
167
101
 
168
102
  ### Option B — Pull from GitHub Container Registry
169
103
 
@@ -183,23 +117,15 @@ cp .env.example .env
183
117
  # Edit .env — set DIGITAL_AI_BASE_URL and DIGITAL_AI_ACCESS_KEY
184
118
  ```
185
119
 
186
- Use `ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest` as the image name in your AI client configuration below.
120
+ **Step 3 — Connect your client** using `ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest` as the image name — see [Docker client configuration](#docker-client-configuration).
187
121
 
188
- > Images are published when a GitHub Release is created — not on every commit. If `latest` is not available yet, use [Option C](#option-c--build-from-source) to build from source.
122
+ > Images are published when a GitHub Release is created — not on every commit. Each release is also tagged with its version (e.g. `2.0.3` and `2.0`) if you want to pin one. If `latest` is not available yet, use [Option C](#option-c--build-from-source) to build from source.
189
123
 
190
- **Upgrading to the latest release:**
191
-
192
- ```bash
193
- docker pull ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest
194
- ```
195
-
196
- Since the server is launched fresh each time via `docker run --rm` (see the client configs below), there's no running container to stop or remove first — the next time your AI client starts the server, it uses the freshly pulled image automatically. Confirm with `get_server_info`.
197
-
198
- ---
124
+ With Docker, tools that read or write local files work inside the container, not on your machine — see [Local File Transfer](#local-file-transfer-docker--remote-deployments-only).
199
125
 
200
126
  ### Option C — Build from source
201
127
 
202
- Required only for local development or modifying the server.
128
+ Needed only for local development or to modify the server.
203
129
 
204
130
  ```bash
205
131
  git clone https://github.com/digitalai-opensource/digital-ai-testing-mcp
@@ -209,106 +135,25 @@ cp .env.example .env
209
135
  docker build -t digital-ai-testing-mcp:latest .
210
136
  ```
211
137
 
212
- Use `digital-ai-testing-mcp:latest` as the image name in your AI client configuration below.
213
-
214
- **Upgrading to the latest release:**
215
-
216
- ```bash
217
- git pull
218
- docker build -t digital-ai-testing-mcp:latest .
219
- ```
220
-
221
- ---
222
-
223
- ## Configuration
224
-
225
- | Variable | Required | Default | Description |
226
- |---|---|---|---|
227
- | `DIGITAL_AI_BASE_URL` | ✅ | — | Your Digital.ai tenant URL |
228
- | `DIGITAL_AI_ACCESS_KEY` | ✅ | — | Access key — see [Access Keys](#access-keys) for the three access levels and key formats |
229
- | `MCP_SERVER_NAME` | Optional | `digital-ai-testing-mcp` | Server identity shown in the AI client |
230
- | `MCP_SERVER_VERSION` | Optional | version from `package.json` | Override the reported server version (rarely needed) |
231
- | `MCP_DEPLOYMENT_MODE` | Optional | `docker` | `docker` (default) or `local` — set to `local` when running via the npm package ([Option A](#option-a--install-via-npm-recommended)) so tool descriptions and upload/download behavior reflect that the server's filesystem is your own machine, not a container's |
232
- | `REQUEST_TIMEOUT_MS` | Optional | `30000` | API request timeout in milliseconds |
233
- | `UPLOAD_TIMEOUT_MS` | Optional | `120000` | File upload timeout in milliseconds |
234
- | `MCP_TOOLSETS` | Optional | all | Comma-separated toolsets to load in full: `devices`, `apps`, `reporting`, `performance`, `inspection`, `authoring`, `browsers`, `admin` (or `all`, or `core` for only the always-loaded core). See [Toolsets](#toolsets-reducing-context-size) |
235
- | `MCP_DEBUG_MODE` | Optional | `false` | `true` records remediation notes and a tool-call event log to help improve this MCP. See [Debug mode](#debug-mode-remediation-notes) |
236
- | `MCP_REMEDIATION_DIR` | Optional | — | Debug mode only: folder to write remediation files to (overrides project detection) |
237
- | `AXE_DEVTOOLS_API_KEY` | Optional | — | Deque Axe DevTools Mobile key used by `includeAxeScan` boilerplate |
238
- | `TEST_ORCHESTRATOR_JAR_URL` / `TEST_ORCHESTRATOR_JAR_SHA256` | Optional | pinned | Override the Test Orchestrator agent download location; both are required |
239
- | `ACCESS_PROBE_TIMEOUT_MS` | Optional | `8000` | Timeout for the access-level detection call |
240
-
241
- Additional `DAI_PROFILE_{NAME}_URL` / `DAI_PROFILE_{NAME}_KEY` pairs configure named profiles for multi-project or multi-environment use. See [Access Keys](#access-keys) and `.env.example` for examples.
242
-
243
- ### Toolsets (reducing context size)
244
-
245
- All 202 tool definitions add up to roughly 68K tokens of context. If your AI client is short on context — or runs several MCP servers — set `MCP_TOOLSETS` to the areas you use, e.g. `MCP_TOOLSETS=reporting` (about 25K tokens) or `MCP_TOOLSETS=inspection,authoring` (about 32K).
246
-
247
- Nothing becomes unavailable. Tools outside the chosen toolsets are still listed, with a one-line description. When the AI calls one, the server loads that whole toolset and replies with the tool's full description and parameter guidance, then the AI calls it again — so the detailed usage guidance is always seen before the tool runs. The AI can also load toolsets up front with `enable_toolset`. Leave `MCP_TOOLSETS` unset (the default) to load everything in full, exactly as before.
248
-
249
- | Toolset | Covers |
250
- |---|---|
251
- | `devices` | Devices, device groups, reservations, health, agents, regions |
252
- | `apps` | Application repository, file repository, provisioning profiles |
253
- | `reporting` | Test reports, analytics, root-cause analysis, test views, coverage |
254
- | `performance` | Performance transactions and comparisons, NV servers |
255
- | `inspection` | Live device and browser inspection, Android Auto / CarPlay, remote debug |
256
- | `authoring` | Test boilerplate, Espresso / XCUITest / Maestro test runs |
257
- | `browsers` | Browser listing and Selenium sessions |
258
- | `admin` | Users, projects, backups, usage reports, POC and project workflows |
259
-
260
- Always loaded in full: environment and connection tools (`get_server_info`, `switch_environment`, …), `enable_toolset`, and `validate_test_script`.
261
-
262
- ### Debug mode (remediation notes)
263
-
264
- Set `MCP_DEBUG_MODE=true` while evaluating or developing the MCP to collect feedback for improving it. With the default `false`, nothing changes.
265
-
266
- When it is on:
267
-
268
- - **The AI records notes.** The server asks the AI client to call `record_remediation_note` (a tool that exists only in debug mode) whenever it:
269
- - hits an unexpected error;
270
- - wastes calls because guidance was unclear;
271
- - is corrected or stopped by you;
272
- - gives up;
273
- - finds a better path than the one recommended;
274
- - spots a way to make things cleaner or cheaper.
275
-
276
- It records each note once the situation is resolved or abandoned, not in the middle of your task. Expected outcomes, such as deliberate negative tests or a safety check that worked as intended, are not recorded: they show nothing to improve.
277
- - **The server logs what it sees.** Every tool call is logged with its duration, response size and outcome (ok, error or guard), with flags for retries and repeated calls. This is objective data the AI can't under-report.
278
- - **The AI is told up front.** The debug-mode instructions come first in what the server sends at connect time, and `get_server_info` and `check_connectivity` show "Debug mode: ON".
279
- - **Errors get a reminder.** The first time a tool returns a given error or guard, the response starts with a one-line reminder to record a note if it was unexpected (in JSON responses, as a `_debugMode` field).
138
+ Use `digital-ai-testing-mcp:latest` as the image name in your [Docker client configuration](#docker-client-configuration).
280
139
 
281
- Where the files go: a `remediation/` folder in the root of **the project the AI is working in**, so each note sits next to the app and tests it's about.
140
+ ### Upgrading
282
141
 
283
- | Install | Notes | Event log |
142
+ | Install | Upgrade command | Then |
284
143
  |---|---|---|
285
- | npm (`MCP_DEPLOYMENT_MODE=local`) | The server writes `<project>/remediation/<yyyymmddhhmmss>-<id>.md` | The server writes `<project>/remediation/<yyyymmddhhmmss>-<id>.events.jsonl` |
286
- | Docker | Returned to the AI, which saves them to `remediation/` in its project if it can write files (Claude Code can; Claude Desktop needs a filesystem tool) | Kept in server memory; a summary is included in every note |
144
+ | npm (Option A) | `npm install -g digital-ai-testing-mcp@latest` | Restart your AI client, or reconnect the MCP server from its settings panel. Installing a new version doesn't affect a server that is already running. |
145
+ | GHCR image (Option B) | `docker pull ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest` | Nothing to stop or remove: the client configs start the server with `docker run --rm`, so the next start uses the new image. |
146
+ | From source (Option C) | `git pull` then `docker build -t digital-ai-testing-mcp:latest .` | Same as Option B. |
287
147
 
288
- With the npm install, the server finds the project in this order, and records the step it used as `location:` in each file:
289
- 1. `MCP_REMEDIATION_DIR`, if set (files go directly into that folder);
290
- 2. the workspace folder the AI client reports;
291
- 3. the folder the server was started in, if it looks like a project;
292
- 4. `~/remediation`.
293
-
294
- The `remediation/` folder contains its own `.gitignore` (`*`), so it is never committed to the project it lives in. There is one file per session; the `<id>` suffix keeps parallel sessions apart, and each file records the machine and user, so files collected from several machines can be merged. Keys, tokens, emails and signed-URL parameters are redacted before anything is written or returned.
148
+ To confirm the upgrade, ask your AI to call `get_server_info`, which reports the running version.
295
149
 
296
- Debug mode costs extra tokens (the instructions and the notes). Leave it off for normal use, and don't use debug sessions as token baselines.
297
-
298
- To process a batch, collect the `remediation/` folders you want to analyse, from one machine or several, and run `npm run remediation:digest -- --dir <folder> --dir <folder>` in a clone of this repository (the npm package doesn't include it). With no `--dir`, it reads `$MCP_REMEDIATION_DIR`, or else `./remediation` and `~/remediation`. It condenses every unprocessed file into one summary, so the development agent reads a few KB instead of every raw note:
299
- - notes by category, and issues that recur across sessions;
300
- - per-tool errors, guards, retries, response sizes and latency;
301
- - every note in a table.
302
-
303
- Useful options:
304
- - `--out <file>` writes the summary to a file; `--since YYYY-MM-DD` limits it to recent sessions; `--json` gives machine-readable output.
305
- - `--archive <session…>` or `--archive-all` moves handled files into `processed/<date>/` inside the folder they came from. Nothing is deleted.
150
+ [↑ Back to top](#table-of-contents)
306
151
 
307
152
  ---
308
153
 
309
154
  ## Connecting AI Clients
310
155
 
311
- The examples below default to the npm package (Option A) — no image name or `.env` file path needed; credentials go directly in the client's `env` block. Each client section also includes a **Docker config** (Options B/C) for anyone running the server as a container instead — replace `/ABSOLUTE/PATH/TO/.env` with the full path to your `.env` file, and swap the GHCR image name for `digital-ai-testing-mcp:latest` if you built from source (Option C).
156
+ The examples below use the npm install ([Option A](#option-a--install-via-npm-recommended)): credentials go directly in the client's `env` block. If you run the server in Docker (Options B/C), use the [Docker client configuration](#docker-client-configuration) instead.
312
157
 
313
158
  ### Claude Desktop
314
159
 
@@ -332,30 +177,15 @@ Find your config file:
332
177
  }
333
178
  ```
334
179
 
335
- > **Using Docker instead (Option B/C)?**
336
- > ```json
337
- > {
338
- > "mcpServers": {
339
- > "digital-ai-testing": {
340
- > "command": "docker",
341
- > "args": [
342
- > "run", "--rm", "-i",
343
- > "--env-file", "/ABSOLUTE/PATH/TO/.env",
344
- > "ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest"
345
- > ]
346
- > }
347
- > }
348
- > }
349
- > ```
350
- > **Built from source (Option C)?** Replace `ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest` with `digital-ai-testing-mcp:latest`.
351
-
352
180
  Restart Claude Desktop after editing — the tools appear automatically.
353
181
 
354
182
  ### Claude Code (VS Code)
355
183
 
356
184
  1. Open the Claude Code extension panel
357
185
  2. Go to **Settings → MCP Servers**
358
- 3. Add a new server with command `digital-ai-testing-mcp`, no args, and `DIGITAL_AI_BASE_URL`/`DIGITAL_AI_ACCESS_KEY`/`MCP_DEPLOYMENT_MODE=local` set in the server's environment variables — or, if running via Docker instead, the Docker command shown for Claude Desktop above
186
+ 3. Add a new server with command `digital-ai-testing-mcp`, no args, and `DIGITAL_AI_BASE_URL`/`DIGITAL_AI_ACCESS_KEY`/`MCP_DEPLOYMENT_MODE=local` set in the server's environment variables
187
+
188
+ Using Docker? Add the Docker command from [Docker client configuration](#docker-client-configuration) instead.
359
189
 
360
190
  ### Claude Code (JetBrains / Android Studio)
361
191
 
@@ -365,46 +195,18 @@ Install the [Claude Code](https://plugins.jetbrains.com/plugin/22828-claude-code
365
195
  claude mcp add digital-ai-testing --env DIGITAL_AI_BASE_URL=https://your-tenant.experitest.com --env DIGITAL_AI_ACCESS_KEY=your-access-key --env MCP_DEPLOYMENT_MODE=local -- digital-ai-testing-mcp
366
196
  ```
367
197
 
368
- This stores the server configuration in `~/.claude.json` scoped to the current project. Alternatively, use the Claude Code panel: **Settings → MCP Servers** and add the same command used for Claude Desktop above.
369
-
370
- > **Using Docker instead (Option B/C)?**
371
- > ```bash
372
- > # macOS / Linux
373
- > claude mcp add digital-ai-testing -- docker run --rm -i --env-file /absolute/path/to/.env ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest
374
- >
375
- > # Windows (use forward slashes — backslashes are stripped by Claude Code)
376
- > claude mcp add digital-ai-testing -- docker run --rm -i --env-file C:/projects/digital-ai-testing-mcp/.env ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest
377
- > ```
378
- > **Windows users on Docker:** Always use forward slashes (`C:/path/to/.env`), never backslashes — regardless of whether you register via the CLI command above or through the Claude Code panel (**Settings → MCP Servers**). Backslashes are silently stripped when Claude Code writes the configuration to `~/.claude.json`, resulting in a broken path and a cryptic `-32000` reconnection error.
379
- > **Built from source (Option C)?** Replace the GHCR image name with `digital-ai-testing-mcp:latest`.
198
+ This stores the server configuration in `~/.claude.json`, scoped to the current project. Alternatively, use the Claude Code panel: **Settings → MCP Servers** and add the same command used for [Claude Desktop](#claude-desktop).
380
199
 
381
200
  Restart the Claude Code panel after adding the server — the tools appear automatically.
382
201
 
202
+ Using Docker? See [Docker client configuration](#docker-client-configuration) — and on Windows, use forward slashes in the `.env` path.
203
+
383
204
  ### GitHub Copilot (VS Code)
384
205
 
385
206
  GitHub Copilot supports MCP tools in **Agent mode** only. Register the server in one of two scopes:
386
207
 
387
- **User settings** — available in all workspaces. Open `settings.json` (`Ctrl+Shift+P` → "Preferences: Open User Settings (JSON)"):
388
-
389
- ```json
390
- {
391
- "mcp": {
392
- "servers": {
393
- "digital-ai-testing": {
394
- "command": "digital-ai-testing-mcp",
395
- "args": [],
396
- "env": {
397
- "DIGITAL_AI_BASE_URL": "https://your-tenant.experitest.com",
398
- "DIGITAL_AI_ACCESS_KEY": "your-access-key",
399
- "MCP_DEPLOYMENT_MODE": "local"
400
- }
401
- }
402
- }
403
- }
404
- }
405
- ```
406
-
407
- **Workspace settings** — scoped to one project. Create `.vscode/mcp.json`:
208
+ - **User settings** — available in all workspaces. Open the user MCP configuration (`Ctrl+Shift+P` → "MCP: Open User Configuration") and add the same `servers` entry shown for the workspace file below. (Older VS Code versions used an `"mcp": { "servers": { … } }` block in the user `settings.json` instead.)
209
+ - **Workspace settings** — scoped to one project. Create `.vscode/mcp.json`:
408
210
 
409
211
  ```json
410
212
  {
@@ -424,22 +226,12 @@ GitHub Copilot supports MCP tools in **Agent mode** only. Register the server in
424
226
 
425
227
  Committing `.vscode/mcp.json` to source control shares the server configuration with the entire team automatically.
426
228
 
427
- > **Using Docker instead (Option B/C)?** in either settings file above, replace the `"digital-ai-testing"` entry's `command`/`args`/`env` with:
428
- > ```json
429
- > {
430
- > "command": "docker",
431
- > "args": [
432
- > "run", "--rm", "-i",
433
- > "--env-file", "/ABSOLUTE/PATH/TO/.env",
434
- > "ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest"
435
- > ]
436
- > }
437
- > ```
438
-
439
229
  To use the tools: open Copilot Chat (`Ctrl+Alt+I`), switch the mode dropdown to **Agent**, and type your request.
440
230
 
441
231
  > GitHub Copilot on the web (`github.com/copilot`) does not support external MCP servers. VS Code is required.
442
232
 
233
+ Using Docker? See [Docker client configuration](#docker-client-configuration).
234
+
443
235
  ### Cursor
444
236
 
445
237
  Cursor supports MCP in **Agent mode**. Add the server in **Cursor Settings → MCP** (or `Cursor Settings → Features → MCP`):
@@ -462,28 +254,213 @@ Cursor supports MCP in **Agent mode**. Add the server in **Cursor Settings → M
462
254
 
463
255
  Alternatively, create a `.cursor/mcp.json` file in your project root with the same `mcpServers` object — this scopes the server to that workspace and can be committed to share it with your team.
464
256
 
465
- > **Using Docker instead (Option B/C)?**
466
- > ```json
467
- > {
468
- > "mcpServers": {
469
- > "digital-ai-testing": {
470
- > "command": "docker",
471
- > "args": [
472
- > "run", "--rm", "-i",
473
- > "--env-file", "/ABSOLUTE/PATH/TO/.env",
474
- > "ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest"
475
- > ]
476
- > }
477
- > }
478
- > }
479
- > ```
480
- > **Built from source (Option C)?** Replace the GHCR image name with `digital-ai-testing-mcp:latest`.
481
-
482
- > Cursor is available on macOS, Windows, and Linux. It is the recommended option for iOS developers on macOS where Xcode is the primary IDE but does not natively support MCP.
257
+ Cursor is available on macOS, Windows, and Linux. It is the recommended option for iOS developers on macOS, where Xcode is the primary IDE but does not natively support MCP.
258
+
259
+ Using Docker? See [Docker client configuration](#docker-client-configuration).
260
+
261
+ ### Docker client configuration
262
+
263
+ For [Option B](#option-b--pull-from-github-container-registry) or [Option C](#option-c--build-from-source), the client starts the container instead of the `digital-ai-testing-mcp` command, and credentials come from your `.env` file. In every example:
264
+
265
+ - Replace `/ABSOLUTE/PATH/TO/.env` with the full path to your `.env` file.
266
+ - **Built from source (Option C)?** Replace `ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest` with `digital-ai-testing-mcp:latest`.
267
+
268
+ **Claude Desktop and Cursor** (`mcpServers` format):
269
+
270
+ ```json
271
+ {
272
+ "mcpServers": {
273
+ "digital-ai-testing": {
274
+ "command": "docker",
275
+ "args": [
276
+ "run", "--rm", "-i",
277
+ "--env-file", "/ABSOLUTE/PATH/TO/.env",
278
+ "ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest"
279
+ ]
280
+ }
281
+ }
282
+ }
283
+ ```
284
+
285
+ **GitHub Copilot** — in either configuration file, replace the `"digital-ai-testing"` entry's `command`/`args`/`env` with:
286
+
287
+ ```json
288
+ {
289
+ "command": "docker",
290
+ "args": [
291
+ "run", "--rm", "-i",
292
+ "--env-file", "/ABSOLUTE/PATH/TO/.env",
293
+ "ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest"
294
+ ]
295
+ }
296
+ ```
297
+
298
+ **Claude Code** (JetBrains / Android Studio, or VS Code) — from the project root, or add the same command under **Settings → MCP Servers**:
299
+
300
+ ```bash
301
+ # macOS / Linux
302
+ claude mcp add digital-ai-testing -- docker run --rm -i --env-file /absolute/path/to/.env ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest
303
+
304
+ # Windows (use forward slashes — backslashes are stripped by Claude Code)
305
+ claude mcp add digital-ai-testing -- docker run --rm -i --env-file C:/projects/digital-ai-testing-mcp/.env ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest
306
+ ```
307
+
308
+ > **Windows users on Docker:** always use forward slashes (`C:/path/to/.env`), never backslashes — whether you register via the CLI command above or through the Claude Code panel. Claude Code silently strips backslashes when it writes the configuration to `~/.claude.json`, which breaks the path and causes a cryptic `-32000` reconnection error ([how to fix it](#failed-to-reconnect-to-digital-ai-testing--32000)).
309
+
310
+ [↑ Back to top](#table-of-contents)
311
+
312
+ ---
313
+
314
+ ## Access Keys
315
+
316
+ Your Digital.ai access key decides what the MCP server can do on your behalf. There are three access levels:
317
+
318
+ | Access level | What it can do |
319
+ |---|---|
320
+ | **Cloud Admin** | All tools: device management, user provisioning, project administration, infrastructure, performance data |
321
+ | **Project Admin** | Scoped to one project. Everything a Project User can do, plus: manage device tags, create reservations, set project notes, view project admin settings, list its own project's users. v2 API tools (agents, regions, license data, usage reports) return 403. |
322
+ | **Project User** | Scoped to one project. Day-to-day testing: devices, app upload/install, reserving devices for itself, test runs, reports, test views, performance data, and inspection sessions. Cannot manage tags or access admin settings; deleting reports requires the project's `allowUsersDeleteTests` setting. |
323
+
324
+ > **You can't tell the access level from the key's format.** Older keys are long `eyJ…` tokens; since Digital.ai 24.11, newly generated keys use a shorter `aut_1_…` token — for Cloud Admins too. So the server asks the API for your role (`GET /api/v1/users/my-account-info` → `Admin` / `ProjectAdmin` / `User`) and never guesses from the key's prefix. `get_server_info` shows the level detected for the active profile, and `list_environments` shows it for every profile. Per-tool role requirements are in [docs/tools.md](docs/tools.md).
325
+
326
+ If you call a Cloud Admin tool with a project-level role, the server returns a plain-language explanation — and, if a Cloud Admin profile is configured, a ready-to-use `switch_environment(...)` command.
327
+
328
+ ### Finding your key
329
+
330
+ 1. Log in to the Digital.ai Testing portal
331
+ 2. Click your **name or avatar** in the top-right corner
332
+ 3. Select **Access Key**
333
+
334
+ The key shown is tied to the **project currently selected in the portal**. If you belong to multiple projects, each has a separate key.
335
+
336
+ ### Using multiple keys
337
+
338
+ Set up named profiles to switch between keys (for example, Cloud Admin and a project) without editing files. With the npm install, add the variables to the `env` block of your AI client's configuration; with Docker, add them to your `.env` file:
339
+
340
+ ```
341
+ # Default connection (typically Cloud Admin credentials — either key format works)
342
+ DIGITAL_AI_BASE_URL=https://your-tenant.experitest.com
343
+ DIGITAL_AI_ACCESS_KEY=...your-cloud-admin-key...
344
+
345
+ # Project-scoped profiles
346
+ DAI_PROFILE_QA_URL=https://your-tenant.experitest.com
347
+ DAI_PROFILE_QA_KEY=...your-qa-key...
348
+
349
+ DAI_PROFILE_STAGING_URL=https://your-tenant.experitest.com
350
+ DAI_PROFILE_STAGING_KEY=...your-staging-key...
351
+ ```
352
+
353
+ Then ask your AI:
354
+ - *"What environments are configured?"* → `list_environments`
355
+ - *"Switch to QA"* → `switch_environment("qa")`
356
+ - *"Switch back to Cloud Admin"* → `switch_environment("default")`
357
+
358
+ The switch takes effect immediately — no restart needed.
359
+
360
+ > **Security:** Never commit `.env` to source control. Each team member should use their own key, obtained from the portal under their own account.
361
+
362
+ [↑ Back to top](#table-of-contents)
363
+
364
+ ---
365
+
366
+ ## Configuration
367
+
368
+ Set these as environment variables: in the client's `env` block (npm install) or in your `.env` file (Docker).
369
+
370
+ | Variable | Required | Default | Description |
371
+ |---|---|---|---|
372
+ | `DIGITAL_AI_BASE_URL` | ✅ | — | Your Digital.ai tenant URL |
373
+ | `DIGITAL_AI_ACCESS_KEY` | ✅ | — | Access key — see [Access Keys](#access-keys) for the three access levels and key formats |
374
+ | `MCP_DEPLOYMENT_MODE` | Optional | `docker` | `docker` (default) or `local`. Set `local` when running via the npm package ([Option A](#option-a--install-via-npm-recommended)) so tool descriptions and upload/download behavior reflect that the server's filesystem is your own machine, not a container's |
375
+ | `MCP_TOOLSETS` | Optional | all | Comma-separated toolsets to load in full: `devices`, `apps`, `reporting`, `performance`, `inspection`, `authoring`, `browsers`, `admin` (or `all`, or `core` for only the always-loaded core). See [Toolsets](#toolsets-reducing-context-size) |
376
+ | `MCP_DEBUG_MODE` | Optional | `false` | `true` records remediation notes and a tool-call event log to help improve this MCP. See [Debug mode](#debug-mode-remediation-notes) |
377
+ | `MCP_REMEDIATION_DIR` | Optional | — | Debug mode only: folder to write remediation files to (overrides project detection) |
378
+ | `REQUEST_TIMEOUT_MS` | Optional | `30000` | API request timeout in milliseconds |
379
+ | `UPLOAD_TIMEOUT_MS` | Optional | `120000` | File upload timeout in milliseconds |
380
+ | `ACCESS_PROBE_TIMEOUT_MS` | Optional | `8000` | Timeout for the access-level detection call |
381
+ | `AXE_DEVTOOLS_API_KEY` | Optional | — | Deque Axe DevTools Mobile key used by `includeAxeScan` boilerplate |
382
+ | `TEST_ORCHESTRATOR_JAR_URL` / `TEST_ORCHESTRATOR_JAR_SHA256` | Optional | pinned | Override the Test Orchestrator agent download location; both are required |
383
+ | `MCP_SERVER_NAME` | Optional | `digital-ai-testing-mcp` | Server identity shown in the AI client |
384
+ | `MCP_SERVER_VERSION` | Optional | version from `package.json` | Override the reported server version (rarely needed) |
385
+
386
+ Additional `DAI_PROFILE_{NAME}_URL` / `DAI_PROFILE_{NAME}_KEY` pairs configure named profiles for multi-project or multi-environment use. See [Using multiple keys](#using-multiple-keys) and `.env.example` for examples.
387
+
388
+ ### Toolsets (reducing context size)
389
+
390
+ All 202 tool definitions together take up roughly 70K tokens of your AI's context. If your AI client is short on context — or runs several MCP servers — set `MCP_TOOLSETS` to the areas you use, e.g. `MCP_TOOLSETS=reporting` (about 26K tokens), `MCP_TOOLSETS=inspection,authoring` (about 34K), or `MCP_TOOLSETS=core` (about 17K — only the always-loaded core in full).
391
+
392
+ Nothing becomes unavailable. Tools outside the chosen toolsets are still listed, with a one-line description. When the AI calls one, the server loads that whole toolset and replies with the tool's full description and parameter guidance, then the AI calls it again — so it always sees the detailed usage guidance before the tool runs. The AI can also load toolsets up front with `enable_toolset`. Leave `MCP_TOOLSETS` unset (the default) to load everything in full.
393
+
394
+ | Toolset | Covers |
395
+ |---|---|
396
+ | `devices` | Devices, device groups, reservations, health, agents, regions |
397
+ | `apps` | Application repository, file repository, provisioning profiles |
398
+ | `reporting` | Test reports, analytics, root-cause analysis, test views, coverage |
399
+ | `performance` | Performance transactions and comparisons, NV servers |
400
+ | `inspection` | Live device and browser inspection, Android Auto / CarPlay, remote debug |
401
+ | `authoring` | Test boilerplate, Espresso / XCUITest / Maestro test runs |
402
+ | `browsers` | Browser listing and Selenium sessions |
403
+ | `admin` | Users, projects, backups, usage reports, POC and project workflows |
404
+
405
+ Always loaded in full: environment and connection tools (`get_server_info`, `switch_environment`, …), `enable_toolset`, and `validate_test_script`.
406
+
407
+ ### Debug mode (remediation notes)
408
+
409
+ Set `MCP_DEBUG_MODE=true` while evaluating or developing the MCP to collect feedback for improving it. With the default `false`, nothing changes.
410
+
411
+ When it is on:
412
+
413
+ - **The AI records notes.** The server asks the AI client to call `record_remediation_note` (a tool that exists only in debug mode) whenever it:
414
+ - hits an unexpected error;
415
+ - wastes calls because guidance was unclear;
416
+ - is corrected or stopped by you;
417
+ - gives up;
418
+ - finds a better path than the one recommended;
419
+ - spots a way to make things cleaner or cheaper.
420
+
421
+ It records each note once the situation is resolved or abandoned, not in the middle of your task. Expected outcomes, such as deliberate negative tests or a safety check that worked as intended, are not recorded: they show nothing to improve.
422
+ - **The server logs what it sees.** Every tool call is logged with its duration, response size and outcome (ok, error or guard), with flags for retries and repeated calls. This is objective data the AI can't under-report.
423
+ - **The AI is told up front.** The debug-mode instructions come first in what the server sends at connect time, and `get_server_info` and `check_connectivity` show "Debug mode: ON".
424
+ - **Errors get a reminder.** The first time a tool returns a given error or guard, the response starts with a one-line reminder to record a note if it was unexpected (in JSON responses, as a `_debugMode` field).
425
+
426
+ Debug mode costs extra tokens (the instructions and the notes). Leave it off for normal use, and don't use debug sessions as token baselines.
427
+
428
+ **Where the files go:** a `remediation/` folder in the root of **the project the AI is working in**, so each note sits next to the app and tests it's about.
429
+
430
+ | Install | Notes | Event log |
431
+ |---|---|---|
432
+ | npm (`MCP_DEPLOYMENT_MODE=local`) | The server writes `<project>/remediation/<yyyymmddhhmmss>-<id>.md` | The server writes `<project>/remediation/<yyyymmddhhmmss>-<id>.events.jsonl` |
433
+ | Docker | Returned to the AI, which saves them to `remediation/` in its project if it can write files (Claude Code can; Claude Desktop needs a filesystem tool) | Kept in server memory; a summary is included in every note |
434
+
435
+ With the npm install, the server finds the project in this order, and records the step it used as `location:` in each file:
436
+ 1. `MCP_REMEDIATION_DIR`, if set (files go directly into that folder);
437
+ 2. the workspace folder the AI client reports;
438
+ 3. the folder the server was started in, if it looks like a project;
439
+ 4. `~/remediation`.
440
+
441
+ The `remediation/` folder contains its own `.gitignore` (`*`), so it is never committed to the project it lives in. There is one file per session; the `<id>` suffix keeps parallel sessions apart, and each file records the machine and user, so files collected from several machines can be merged. Keys, tokens, emails and signed-URL parameters are redacted before anything is written or returned.
442
+
443
+ <details>
444
+ <summary><strong>Summarizing a batch of notes</strong></summary>
445
+
446
+ Collect the `remediation/` folders you want to analyse, from one machine or several, and run `npm run remediation:digest -- --dir <folder> --dir <folder>` in a clone of this repository (the npm package doesn't include it). With no `--dir`, it reads `$MCP_REMEDIATION_DIR`, or else `./remediation` and `~/remediation`. It condenses every unprocessed file into one summary, so the development agent reads a few KB instead of every raw note:
447
+ - notes by category, and issues that recur across sessions;
448
+ - per-tool errors, guards, retries, response sizes and latency;
449
+ - every note in a table.
450
+
451
+ Useful options:
452
+ - `--out <file>` writes the summary to a file; `--since YYYY-MM-DD` limits it to recent sessions; `--json` gives machine-readable output.
453
+ - `--archive <session…>` or `--archive-all` moves handled files into `processed/<date>/` inside the folder they came from. Nothing is deleted.
454
+
455
+ </details>
456
+
457
+ [↑ Back to top](#table-of-contents)
483
458
 
484
459
  ---
485
460
 
486
- ## Example Prompts
461
+ ## What You Can Do
462
+
463
+ ### Example prompts
487
464
 
488
465
  Once connected, talk to the server in plain language — no tool names needed:
489
466
 
@@ -498,13 +475,45 @@ Once connected, talk to the server in plain language — no tool names needed:
498
475
 
499
476
  **[→ docs/examples.md](docs/examples.md)** has 200+ prompts organized by scenario: farm health, device groups, regression sign-off, test analytics, performance, usage reports, interactive inspection, POC lifecycle, and more.
500
477
 
501
- ---
478
+ ### Tasks at a glance
479
+
480
+ | I want to… | Where to look |
481
+ |---|---|
482
+ | Check farm health, find a device, reserve it, manage groups and tags | [Devices](docs/tools.md#devices), [Device Groups](docs/tools.md#device-groups), [Health & Diagnostics](docs/tools.md#health--diagnostics), [Reservations](docs/tools.md#reservations) |
483
+ | Upload, install or find the latest build of an app; manage test files and iOS profiles | [Applications](docs/tools.md#applications), [Repository](docs/tools.md#repository), [Provisioning Profiles](docs/tools.md#provisioning-profiles) |
484
+ | Explore an app or web page live with the AI | [Interactive inspection](#interactive-inspection) below |
485
+ | Create an automated test | [Creating Tests](#creating-tests) |
486
+ | Run Espresso, XCUITest or Maestro suites on the platform | [Test runs](#test-runs-espresso-xcuitest-maestro) below |
487
+ | Find out why tests failed, and track pass rates and trends | [Test Results and Root Cause Analysis](#test-results-and-root-cause-analysis) |
488
+ | Compare performance between versions, devices or networks | [Performance Comparison Reports](#performance-comparison-reports) |
489
+ | Export or summarize platform usage *(Cloud Admin)* | [Usage reports](#usage-reports) below |
490
+ | Set up or wind down a POC or project; manage users | [Workflow Reference](#workflow-reference), [Users](docs/tools.md#users), [Projects](docs/tools.md#projects) |
491
+
492
+ #### Interactive inspection
493
+
494
+ An inspection session opens a live connection to a real device or browser so the AI can see the screen and interact with it — useful for exploring an app and capturing real element IDs before writing a test.
495
+
496
+ - **Mobile** ([tools](docs/tools.md#mobile-inspection-sessions)) — Android and iOS, on both Appium Grid and Appium Server projects: screenshots, element discovery, the full gesture set, keys, app/device control, Android Auto / CarPlay projection and mock biometric authentication. The session shares a watch-only live view URL so you can follow along. iOS behaves differently in a few places ([limitation 29](docs/limitations.md#29-ios-inspection-sessions-differ-from-android)), and Appium Grid projects support fewer commands ([limitation 18](docs/limitations.md#18-appium-grid-projects-support-fewer-commands-than-appium-server)).
497
+ - **Browser** ([tools](docs/tools.md#web-inspection-sessions)) — navigate, discover elements (including inside Shadow DOM), verify CSS selectors and generate browser-neutral Selenium scripts.
498
+ - **Remote debug** ([tools](docs/tools.md#remote-debug)) — `get_remote_debug_command` makes a cloud device appear as if it were plugged into your machine (over ADB for Android, usbmuxd for iOS), so Android Studio, Xcode and command-line ADB can use it — for example for shell commands or file push. Install the app **before** connecting (see [Troubleshooting](#common-symptoms)).
499
+
500
+ Example prompts: [Interactive Inspection](docs/examples.md#interactive-inspection-ai-driven-test-building), [Android Auto / CarPlay](docs/examples.md#android-auto--carplay), [Web Browser Inspection](docs/examples.md#web-browser-inspection-ai-driven-browser-test-building).
502
501
 
503
- ## Capabilities
502
+ #### Test runs (Espresso, XCUITest, Maestro)
504
503
 
505
- 202 tools across 27 capability domains. The complete per-tool reference — descriptions, filters, auth requirements, and usage notes — lives in **[docs/tools.md](docs/tools.md)**.
504
+ The platform runs these suites itself: `execute_test_run` starts a run, `get_test_run_status` follows it, `cancel_test_run` stops it, and `get_test_run_command` gives a command to start it from your own machine. `generate_maestro_flow` builds a Maestro flow from structured steps. Maestro is Android-only. Espresso and Maestro runs are verified; XCUITest uses the same API but is not yet verified end to end. See [Test Runs](docs/tools.md#test-runs-espresso--xcuitest--maestro) and [example prompts](docs/examples.md#test-runs-espresso-xcuitest-maestro).
506
505
 
507
- | Domain | Tools | Highlights |
506
+ Appium and Selenium tests are different: you run them from your own Appium or Selenium client (IDE, CI scripts), and the server manages the devices, apps, reservations and results they use.
507
+
508
+ #### Usage reports
509
+
510
+ Cloud Admins can export the platform's usage-report CSVs — device reservations, users/devices/browser usage, user statistics, license usage — or have `summarize_usage_report` aggregate one in memory and return just the summary, with no file written. Usage data is kept for 2 years (platform 25.9+); earlier periods come back empty or partial. A large unfiltered export runs only after you explicitly agree (`confirmLargeExport`, see [Safety Guards](#safety-guards)). See [Usage Reports](docs/tools.md#usage-reports), [limitation 21](docs/limitations.md#21-usage-reports-2-year-retention-large-exports) and [example prompts](docs/examples.md#usage-reports-platform-administration--finops).
511
+
512
+ ### All capability areas
513
+
514
+ 202 tools across 27 capability areas. The complete per-tool reference — descriptions, filters, access requirements and usage notes — is in **[docs/tools.md](docs/tools.md)**.
515
+
516
+ | Area | Tools | Highlights |
508
517
  |---|---|---|
509
518
  | [Devices](docs/tools.md#devices) | 19 | List/query devices, find available, release, reboot, tags, Mobile Studio |
510
519
  | [Device Groups](docs/tools.md#device-groups) | 9 | Create groups, move devices, control project access |
@@ -516,7 +525,7 @@ Once connected, talk to the server in plain language — no tool names needed:
516
525
  | [Projects](docs/tools.md#projects) | 17 | Create/configure projects, settings, app assignment |
517
526
  | [Provisioning Profiles](docs/tools.md#provisioning-profiles) | 7 | iOS signing profile lifecycle |
518
527
  | [Backup](docs/tools.md#backup) | 1 | Trigger a system backup |
519
- | [Health & Diagnostics](docs/tools.md#health--diagnostics) | 11 | Farm health, readiness checks, license utilization |
528
+ | [Health & Diagnostics](docs/tools.md#health--diagnostics) | 7 | Farm snapshot, iOS readiness, agent status, active sessions, reporter storage, license limits and utilization |
520
529
  | [Coverage Analytics](docs/tools.md#coverage-analytics) | 2 | Tested-vs-available device gap analysis |
521
530
  | [Reporting](docs/tools.md#reporting) | 23 | Search/filter test reports, root-cause analysis, failure summarization, stability, trends, logs, video download (byte ranges), share links, attachment download command, cleanup |
522
531
  | [Test Runs](docs/tools.md#test-runs-espresso--xcuitest--maestro) | 5 | Run Espresso, XCUITest and Maestro suites on the platform; status, cancel, local command; generate Maestro flows |
@@ -525,27 +534,265 @@ Once connected, talk to the server in plain language — no tool names needed:
525
534
  | [Agents](docs/tools.md#agents) | 2 | Host machine status *(Cloud Admin)* |
526
535
  | [Regions](docs/tools.md#regions) | 2 | Region status and infrastructure topology *(Cloud Admin)* |
527
536
  | [NV Servers](docs/tools.md#nv-servers) | 2 | Network Virtualization servers *(Cloud Admin)* |
528
- | [Environment Management](docs/tools.md#environment-management) | 2 | Named connection profiles, runtime switching |
537
+ | [Server, Environments & Toolsets](docs/tools.md#server-environments--toolsets) | 6 | Server info, connectivity and workflow readiness checks, named connection profiles and runtime switching, toolset loading (plus `record_remediation_note` in debug mode) |
529
538
  | [Workflows](docs/tools.md#workflows) | 6 | POC and project lifecycle orchestration |
530
- | [Boilerplate Generation](docs/tools.md#boilerplate-generation) | 4 | Ready-to-run Appium and Selenium test scripts in 4 languages, Test Orchestrator agent install; validate scripts before delivery |
531
- | [Remote Debug](docs/tools.md#remote-debug) | 1 | Connect a cloud device as a local ADB device |
539
+ | [Boilerplate Generation](docs/tools.md#boilerplate-generation) | 4 | Ready-to-run Appium and Selenium test projects (Java JUnit 5, Java TestNG, Node.js, Python), Test Orchestrator agent install; validate scripts before delivery |
540
+ | [Remote Debug](docs/tools.md#remote-debug) | 1 | Connect a cloud Android or iOS device as if it were attached locally (ADB / usbmuxd) |
532
541
  | [Mobile Inspection Sessions](docs/tools.md#mobile-inspection-sessions) | 24 | AI-driven live device interaction — screenshots, element discovery, full gesture set, keys, app/device control, Android Auto / CarPlay projection, mock biometric authentication |
533
542
  | [Web Inspection Sessions](docs/tools.md#web-inspection-sessions) | 8 | AI-driven live browser interaction — navigate, Shadow DOM element discovery, CSS selector verification, browser-neutral Selenium script generation |
534
543
  | [Performance Comparison](docs/tools.md#performance-comparison) | 4 | Two-set Speed Index comparison with confound detection, MAD outlier exclusion, and fresh-sample generation |
535
- | [Usage Reports](docs/tools.md#usage-reports) | 3 | Platform usage-report CSV export and in-memory aggregation — device reservations, users/devices/browser usage, license usage; multi-section CSVs with a User Tag column, about 2 years of retention; large unfiltered exports run only after you explicitly agree (`confirmLargeExport`) *(Cloud Admin)* |
544
+ | [Usage Reports](docs/tools.md#usage-reports) | 3 | Platform usage-report CSV export and in-memory aggregation — device reservations, users/devices/browser usage, user statistics, license usage; multi-section CSVs with a User Tag column, 2-year retention (platform 25.9+); large unfiltered exports run only after you explicitly agree (`confirmLargeExport`) *(Cloud Admin)* |
536
545
  | [Resources & Prompts](docs/tools.md#resources--prompts) | — | 2 ambient resources, 7 guided prompts |
537
546
 
547
+ [↑ Back to top](#table-of-contents)
548
+
549
+ ---
550
+
551
+ ## Creating Tests
552
+
553
+ ### Two ways to create a test
554
+
555
+ The server supports two ways of creating a test, for both mobile and web, and the AI is instructed to pick between them based on context:
556
+
557
+ **Autonomous** — `get_test_boilerplate` (mobile) / `get_web_test_boilerplate` (web)
558
+ - **When:** your intent is specific — a standard flow ("create a login test") or step-level detail — AND the selectors are known (app source in the workspace, or a prior inspection session).
559
+ - **Your involvement:** none required — the AI writes the test from the boilerplate, the source and best practices.
560
+ - **Output:** in an IDE, a local test automation project; in a chat-only client, portable project files shown inline.
561
+
562
+ **Interactive** — `start_inspection_session` / `start_browser_inspection_session`, or the `collaborative_test_creation` / `collaborative_web_test_creation` prompts
563
+ - **When:** the request is vague ("I want a test for app X"), the AI has no access to the app's source (chat-only, or an IDE without this app's code), or you want to watch or drive on a live device or browser.
564
+ - **Your involvement:** collaborative — the AI builds the test step by step with you (browser sessions relay screenshots; mobile sessions share a live device view URL).
565
+ - **Output:** the same, plus selectors verified against the real app or page along the way.
566
+
567
+ **A mix is often best:** even in autonomous mode, the AI can open a short inspection session — with no involvement from you — to capture or check element IDs or CSS selectors against the real build. The AI is explicitly instructed to **never make up selectors**: when in doubt it checks live, and when the mode itself is unclear it asks.
568
+
569
+ > **Stronger guarantees:** the server sends this policy to the AI automatically (in its MCP `instructions`), and both boilerplate tools refuse to produce code for a real app or URL without verified selectors. But not every client passes server `instructions` on to the AI, and no MCP server can stop an AI from writing a test file by hand — so you can add a client-side reinforcement for a guarantee that doesn't depend on the client. See **[Recommended agent guardrails](docs/recommended-agent-guardrails.md)**.
570
+
571
+ ### Mobile app tests (`get_test_boilerplate`)
572
+
573
+ Generates a complete, pre-configured Appium test project. The Digital.ai server URL and access key are filled in from the active connection profile — switch profiles with `switch_environment` first if you want the script to carry a project-scoped key instead of your admin key.
574
+
575
+ **Recommended workflow** for building a new test:
576
+
577
+ 1. `get_application_info` — confirm package name, launch activity, and app ID
578
+ 2. `find_available_device` — select a healthy device and read its `region`
579
+ 3. `install_application` — install the app on that device before starting a session
580
+ 4. `start_inspection_session` — capture real element IDs with `get_element_tree` / `find_elements`
581
+ 5. `get_test_boilerplate(appId, region, …)` — generates code while the inspection session is live, or with `confirmSelectorsVerified: true`
582
+ 6. [`validate_test_script`](#check-a-test-before-you-deliver-it-validate_test_script) — before saving or presenting the test
583
+ 7. Run the test, then `stop_inspection_session` and `release_device` — explicit cleanup
584
+
585
+ > **Inspection gate:** when you target a real app (`appId`/`packageName`/`bundleIdentifier`), this tool returns **no code** — just a structured `{ status: "blocked", reason: "no_verified_selectors" }` error — UNLESS a live inspection session exists in the MCP server, or you set `confirmSelectorsVerified: true` (for selectors already captured another way: rdb/UIAutomator, `open_mobile_studio`, or authoritative source). The built-in ExperiBank demo (no app identifiers) is never gated. This way you can never receive a placeholder scaffold that could be passed off as a finished test.
586
+
587
+ **Finding element locators:** use an inspection session first — `start_inspection_session` → `get_element_tree` / `find_elements` gives the AI the element tree, screenshots and gestures, and is the only way to capture verified selectors inside the MCP. `open_mobile_studio` (the platform's browser-based UI Inspector) is a visual alternative that needs no local tooling. For direct device access (ADB shell commands and file push on Android, Xcode on iOS), use `get_remote_debug_command` to connect the device locally.
588
+
589
+ > ⚠️ The generated script is a **finished product**, not a discovery tool — don't run it to find out what's on screen. Running it without known element selectors creates Incomplete sessions visible to the whole team. Discover selectors with an inspection session (or `open_mobile_studio`) first.
590
+
591
+ <details>
592
+ <summary><strong>Parameters</strong></summary>
593
+
594
+ | Parameter | Values | Default |
595
+ |---|---|---|
596
+ | `platform` | `android` \| `ios` | _(required)_ |
597
+ | `language` | `java-junit5` \| `java-testng` \| `nodejs` \| `python` | _(required)_ |
598
+ | `appId` | Numeric app ID from `list_applications` | Recommended |
599
+ | `deviceCategory` | `PHONE` \| `TABLET` | `PHONE` |
600
+ | `testName` | Any string | `"My First Mobile Test"` |
601
+ | `packageName` | Android package name (e.g. `com.mycompany.app`) | _(optional)_ |
602
+ | `mainActivity` | Android main activity (e.g. `.MainActivity`) | _(optional)_ |
603
+ | `bundleIdentifier` | iOS bundle ID (e.g. `com.mycompany.app`) | _(optional)_ |
604
+ | `region` | Region code from `find_available_device` response (e.g. `US2`, `SG1`) | _(optional, strongly recommended)_ |
605
+ | `projectType` | `standalone-gradle` \| `standalone-maven` \| `android-gradle-submodule` | `standalone-gradle` |
606
+ | `includePerformanceTransactions` | `true` \| `false` | `false` |
607
+ | `includeAxeScan` | `true` \| `false` | `false` |
608
+ | `confirmSelectorsVerified` | `true` \| `false` | _(optional)_ |
609
+ | `orchestration` | `auto` \| `on` \| `off` | `auto` |
610
+ | `orchestrationMaxRetries` | `0`–`5` | `2` (`0` for placeholder or performance tests) |
611
+ | `automotiveProjection` | Android Auto / CarPlay resolution | _(optional)_ |
612
+ | `outputFormat` | `json` \| `human` | `json` |
613
+
614
+ </details>
615
+
616
+ **Generated files.** `projectType` controls the Java output layout only (it is ignored for Node.js and Python): `standalone-gradle` produces `src/test/java/` with both `build.gradle` and `pom.xml`; `standalone-maven` produces `pom.xml` only; `android-gradle-submodule` puts the files under `e2e-tests/` (with `e2e-tests/build.gradle`) for embedding in an existing Android Studio project.
617
+
618
+ | Language | Files |
619
+ |---|---|
620
+ | `java-junit5` / `java-testng` | `src/test/java/LocalAndroidTest.java` or `LocaliOSTest.java` + build file(s) per `projectType`; orchestrated projects add `orchestrator/config.template.yml`, `orchestrator/.gitignore` and `lib/.gitignore` |
621
+ | `nodejs` | `package.json` + `wdio.conf.js` + `AndroidNative.js` / `iOSNative.js` + `setup-steps.sh` (WebdriverIO 7.40 for Appium Grid projects, WebdriverIO 9 for Appium Server) |
622
+ | `python` | `AndroidNative.py` / `iOSNative.py` + `requirements.txt` |
623
+
624
+ All boilerplate uses the ExperiBank demo app as its starting point. ExperiBank is available on most farm devices, so you can verify connectivity before switching to your own app.
625
+
626
+ **Performance transactions:** pass `includePerformanceTransactions: true` to wrap the test body in `startPerformanceTransaction` / `endPerformanceTransaction` calls. The start argument is the NV network profile (defaults to `"Monitor"` — observe without throttling); the end argument is the transaction name. The platform records CPU, memory, battery and Speed Index metrics; results appear in the reporter Transactions tab about 1 minute after `endPerformanceTransaction` and can be queried with `list_transactions`. Prerequisite: an NV server must be ONLINE and tunnel-connected in the device's region (`list_nv_servers`). A throttling profile turns on network shaping immediately; `"Monitor"` is pass-through and carries no ANR (app-not-responding) risk.
627
+
628
+ **Accessibility scanning:** pass `includeAxeScan: true` to add a Deque Axe DevTools Mobile accessibility scan (a `mobile: axeScan` executeScript call). It sets the required `appium:automationName` capability automatically (`AxeUiAutomator2` for Android, `AxeXCUITest` for iOS) and pins `appiumVersion` to 2.16.2 (the Axe driver is unavailable on Appium Server 3.3.0+; don't raise it). The key comes from `AXE_DEVTOOLS_API_KEY`; if that's unset, a placeholder is written.
629
+
630
+ Both flags can be combined — the Axe scan runs inside the performance transaction.
631
+
632
+ #### Test Orchestrator (default for Java on Appium Server)
633
+
634
+ For **Java (JUnit 5 / TestNG) on an Appium Server project**, the generated project is set up for the **Digital.ai Test Orchestrator** (the Smart Agent): failed tests are retried automatically, each attempt is reported separately, the real pass/fail is synced to Reporter, and runs are grouped by Build ID — with no change to the test code. Appium Grid, Python and Node.js projects get the classic boilerplate. Pass `orchestration: "off"` to get the classic boilerplate for Java too.
635
+
636
+ 1. **`get_test_boilerplate`** adds conditional agent wiring to `build.gradle` / `pom.xml`, an `orchestrator/config.template.yml` (retries default to 2; 0 for placeholder or performance tests), plus `orchestrator/.gitignore` and `lib/.gitignore` so the rendered key and the agent JAR can never be committed. No root `.gitignore` is written (it would overwrite yours).
637
+ 2. **`install_test_orchestrator_agent(projectDir)`** downloads the agent on demand, verifies its SHA-256 and places it at `lib/smart-agent.jar`. Under Docker it returns a download command (bash and PowerShell) that does the same and deletes the file on a checksum mismatch. If the download fails, the response gives the URL, the checksum and the target path for a manual install.
638
+ 3. **Set `DIGITAL_AI_ACCESS_KEY` and run `gradle test` / `mvn test`.** The agent attaches only when the JAR is present **and** the key is set; the key is rendered into the git-ignored `orchestrator/rendered/` at build time, never into a tracked file, and Java 17 is required only in that case. Otherwise the project builds and runs exactly like the classic boilerplate (Gradle prints why the agent was not attached). Set `BUILD_ID` to group a CI build's results.
639
+
640
+ Requirements: a **Java 17 or 21** JDK with a current truststore, JUnit 5 or TestNG 7+, Appium java-client 8–10. See [Limitations](docs/limitations.md#17-test-orchestrator-known-issues) for known issues.
641
+
642
+ The agent JAR is **not** shipped with this server. By default it is downloaded from the [Digital.ai Test Orchestrator sample repository](https://github.com/raheekhandigitalai/Digital.ai-Testing-Test-Orchestrator), pinned to a fixed commit and verified by SHA-256 (see `resources/test-orchestrator/agent.json`). To use a different location — for example an internal mirror — set `TEST_ORCHESTRATOR_JAR_URL` **and** `TEST_ORCHESTRATOR_JAR_SHA256` (both required; a URL without a checksum is refused).
643
+
644
+ ### Web tests (`get_web_test_boilerplate`)
645
+
646
+ Generates a complete, pre-configured Selenium test project for browser testing. The Digital.ai server URL and access key are filled in from the active connection profile.
647
+
648
+ **Recommended workflow** for building a new web test:
649
+
650
+ 1. `list_available_browsers` — confirm which browsers and OS combinations are available in your region
651
+ 2. `start_browser_inspection_session` — open a live browser session on the target URL
652
+ 3. `get_page_dom` — extract the rendered DOM, including shadow trees; note element IDs and `data-testid` values
653
+ 4. `find_web_elements` / `tap_element` / `type_into_element` — interact with elements to verify selectors work
654
+ 5. `take_inspection_screenshot` — relay screenshots at each key step for verification
655
+ 6. `stop_browser_inspection_session`
656
+ 7. `get_web_test_boilerplate(confirmSelectorsVerified: true)` — generate the verified script (once the session is stopped, pass `shadowDomSupport: "always"` if `get_page_dom` reported shadow DOM)
657
+
658
+ > **Inspection gate:** when you target a real URL (`url` provided), this tool returns **no code** — just a structured `{ status: "blocked", reason: "no_verified_selectors" }` error — UNLESS a live browser inspection session exists in the MCP server, or you set `confirmSelectorsVerified: true` (for selectors captured in a prior `start_browser_inspection_session` session). This mirrors the mobile gate.
659
+
660
+ **Browser-neutral by default:** when `targetBrowser` is omitted, the generated script reads the browser name at runtime from the `BROWSER` environment variable, so the same script runs on Chrome, Firefox, Microsoft Edge, Safari and Opera without changes — just change the variable or your CI matrix. Specify `targetBrowser` only when you need browser-specific options (e.g. Safari-only session setup).
661
+
662
+ **Shadow DOM support:** with `always`, the generated script includes a `shadowQuery(host, cssSelector)` helper; with `auto` (the default), it is included only while a browser inspection session is open in the MCP server; `never` omits it. The helper reaches into shadow roots using `executeScript`. Use it for apps built with Web Components, Lit, Polymer or other shadow-DOM frameworks. `get_page_dom` detects shadow DOM automatically during inspection and reports `hasShadowDom: true` when present.
663
+
664
+ <details>
665
+ <summary><strong>Parameters</strong></summary>
666
+
667
+ | Parameter | Values | Default |
668
+ |---|---|---|
669
+ | `language` | `java-junit5` \| `java-testng` \| `nodejs` \| `python` | _(required)_ |
670
+ | `url` | URL of the page being tested | _(optional)_ |
671
+ | `testName` | Any string | `"WebTest"` |
672
+ | `targetBrowser` | `chrome` \| `firefox` \| `MicrosoftEdge` \| `safari` \| `opera` (names as `list_available_browsers` returns them; `edge` is accepted for `MicrosoftEdge`) | _(optional)_ |
673
+ | `shadowDomSupport` | `auto` \| `always` \| `never` | `auto` |
674
+ | `confirmSelectorsVerified` | `true` \| `false` | _(optional)_ |
675
+ | `outputFormat` | `json` \| `human` | `json` |
676
+
677
+ </details>
678
+
679
+ ### Check a test before you deliver it (`validate_test_script`)
680
+
681
+ A final check for the case the boilerplate gate can't catch: a test written by hand with guessed selectors. Pass the full `scriptContent` before presenting or saving any test. It looks for unreplaced `<…>` placeholder selectors, the deliberate scaffold fail-guard, placeholder or made-up credentials, and known fabricated resource IDs. If it finds a serious problem it returns `isError` with a `fail` verdict, so a test that can't work isn't delivered as finished. A `pass` is necessary but not sufficient: it confirms the obvious placeholders are gone, not that the selectors are real — a live inspection is still the authoritative source.
682
+
683
+ | Parameter | Values | Default |
684
+ |---|---|---|
685
+ | `scriptContent` | The full test script text | _(required)_ |
686
+ | `fileName` | Label for the result | _(optional)_ |
687
+ | `outputFormat` | `json` \| `human` | `json` |
688
+
689
+ [↑ Back to top](#table-of-contents)
690
+
691
+ ---
692
+
693
+ ## Test Results and Root Cause Analysis
694
+
695
+ The reporting tools search test reports, summarize pass/fail results and explain failures. Field names, filter properties and ID types are in the [Test reporting schema](docs/tools.md#test-reporting-schema); more prompts are in [Root Cause Analysis](docs/examples.md#root-cause-analysis-why-tests-are-failing) and [Functional Test Analytics](docs/examples.md#functional-test-analytics).
696
+
697
+ Two things to know:
698
+ - **The list doesn't include failure details.** `list_test_reports` returns summary records; call `get_test_report` on a test to get its cause, error category and stack trace.
699
+ - **Filter by date with `startDate`/`endDate`**, not with a `start_time` filter (the platform rejects that).
700
+
701
+ **Common queries:**
702
+
703
+ ```
704
+ # Recent failures
705
+ list_test_reports
706
+ filter: [{"property":"status","operator":"=","value":"Failed"}]
707
+ sort: [{"property":"start_time","descending":true}]
708
+ limit: 20
709
+
710
+ # Date range
711
+ list_test_reports
712
+ startDate: "2026-05-01T00:00:00Z"
713
+ endDate: "2026-05-15T23:59:59Z"
714
+
715
+ # Pass/fail by OS
716
+ get_grouped_test_reports
717
+ groupBy: ["device.os"]
718
+ pivotBy: ["status"]
719
+
720
+ # Project summary
721
+ get_project_test_summary
722
+ startDate: "2026-05-01T00:00:00Z"
723
+ projectName: "My Project"
724
+ ```
725
+
726
+ **Root cause analysis of a failure:**
727
+
728
+ ```
729
+ # 1. Find the failures
730
+ list_test_reports
731
+ filter: [{"property":"status","operator":"=","value":"Failed"}]
732
+
733
+ # 2. Diagnose — structured cause without any download
734
+ get_test_report(testId: 69)
735
+ → cause, errorCategory, errorClassification, errorDetail (stack trace)
736
+
737
+ # 2b. Platform root-cause analysis (read-only)
738
+ get_root_cause_analysis(uuid: "…") # or reportUrl / testId
739
+
740
+ # 3. Full Appium log as text, when the structured fields aren't enough
741
+ get_test_log(uuid: "…", logType: "appium")
742
+
743
+ # 4. Pull the binary artifacts (session video .mp4, full ZIP) to YOUR machine.
744
+ # With the npm install, call download_test_video / download_test_attachments
745
+ # directly; under Docker/remote, generate a local command instead:
746
+ get_test_attachments_download_command(uuid: "…", localPlatform: "macos")
747
+ → a curl command you run locally; the file never transits the container
748
+ ```
749
+
750
+ Tests missing from the results? See [Troubleshooting](#common-symptoms) — a project with its own reporter needs `projectName`.
751
+
752
+ [↑ Back to top](#table-of-contents)
753
+
754
+ ---
755
+
756
+ ## Performance Comparison Reports
757
+
758
+ Compare performance (Speed Index, CPU, memory, battery, duration) between two sets of conditions — app v1 vs v2, device A vs device B, OS version X vs Y, region to region, two network profiles, or two automation scripts. Works with all access levels — Cloud Admin sees all projects; project-level keys (Project Admin and Project User) see only their own project's transactions.
759
+
760
+ The headline metric is reported three ways every time — **trimmed mean, median, and raw mean** — so a small or noisy sample can't hide behind a single number.
761
+
762
+ | Tool | Purpose |
763
+ |---|---|
764
+ | `compare_performance_transactions` | Two-set comparison; trimmed-mean/median/mean per side + delta and % change. MAD outliers excluded by default (≥4 samples/side). Pass `comparisonAxis` to embed a confound check. |
765
+ | `assess_comparison_confounds` | Verdict — **clean / caveated / confounded** — by flagging any dimension other than the declared axis (device model, OS, OS version, network profile, project, transaction name) that varies across or within the sides, plus missing telemetry and sample imbalance. |
766
+ | `detect_performance_outliers` | Robust median/MAD outlier flagging on a single set; returns the kept set and recommended exclusions/re-runs. |
767
+ | `performance_transaction_control` | Phase 2: generate fresh samples inside an inspection session — `start` (with an NV profile) → run the verified flow → `end` (names the record). Records appear in the reporter ~1 min later. |
768
+
769
+ > **Why the confound check matters:** comparing two transactions with Speed Index 1015 vs 1000 looks like a 1.5% version delta — but if they ran on *different device models, different OS versions, and different projects*, and one has no CPU/memory telemetry, `assess_comparison_confounds` returns **confounded** and names each cause, so the delta is never misread as a version regression.
770
+
771
+ The **`performance_comparison_report` prompt** runs the full workflow end to end: define the axis → scrub confounds → negotiate sample size → **require explicit plan confirmation with a time estimate** → run the series with outlier-driven re-runs → report the delta with root-cause reasoning. True host/background interference can't be observed directly; the report approximates device quietness (idle, healthy status, no concurrent reservation) and states that limit explicitly.
772
+
773
+ To record new performance samples from a generated test, see [performance transactions](#mobile-app-tests-get_test_boilerplate) in Creating Tests. More prompts: [Performance Testing Analytics](docs/examples.md#performance-testing-analytics).
774
+
775
+ [↑ Back to top](#table-of-contents)
776
+
538
777
  ---
539
778
 
540
779
  ## Workflow Reference
541
780
 
781
+ Six workflow tools set up and wind down POC and project environments. They don't make changes themselves: each returns a guided, numbered plan that the AI then carries out with the individual tools (`create_device_group`, `create_user`, …), pausing for your confirmation at the points the plan specifies. The plans use Cloud Admin tools (user, group and project administration), so run them with a Cloud Admin profile.
782
+
783
+ | Goal | POC | General project |
784
+ |---|---|---|
785
+ | Set up | [`create_poc`](#create_poc--onboard-a-new-poc-environment) | [`setup_project`](#setup_project--provision-a-new-project-environment) |
786
+ | Release devices and users, keep the project | [`close_poc`](#close_poc--wind-down-without-deleting) | [`close_project_resources`](#close_project_resources--release-resources-without-deleting) |
787
+ | Delete everything | [`delete_poc`](#delete_poc--full-teardown) | [`teardown_project`](#teardown_project--full-project-deletion) |
788
+
542
789
  ### POC Lifecycle
543
790
 
544
- Three tools cover the full POC lifecycle. All derive the project name (`"<Customer> POC"`) and device tag (`"<customername>poc"`) from `customerName` — pass the same value consistently across all three tools.
791
+ Three tools cover the full POC lifecycle. All derive the project name (`"<Customer> POC"`) and device tag (`"<customername>poc"`) from `customerName` — pass the same value to all three tools.
545
792
 
546
793
  #### `create_poc` — Onboard a new POC environment
547
794
 
548
- Collects all parameters upfront, presents a confirmation summary to the operator, then executes 10 sequential steps:
795
+ Collects all parameters up front, checks that the app binaries are in the repository, presents a confirmation summary to the operator (Step 0), then runs 10 steps in order:
549
796
 
550
797
  1. Create (or reuse) a device group named `<Customer> POC`
551
798
  2. Locate the Default device group
@@ -558,7 +805,7 @@ Collects all parameters upfront, presents a confirmation summary to the operator
558
805
  9. Create users, assign to the POC project, remove from Default, and tag each account with the POC tag. Cloud Admin access is never granted through this workflow.
559
806
  10. Assign ExperiBank (or the app identified by `androidPackageName` / `iosBundleId`) to the POC project — if a binary isn't in the repository, the summary reports it as a manual upload and the workflow continues
560
807
 
561
- If any step fails, the workflow stops, reports which steps completed with all created resource IDs, and offers resume (re-invoke — existing resources are detected and reused) or unwind (`delete_poc`) paths.
808
+ If any step fails, the workflow stops, reports which steps completed with all created resource IDs, and offers to resume (call it again — existing resources are detected and reused) or unwind (`delete_poc`). Steps 1 and 6 check for an existing group/project by name before creating, so it is safe to re-run after a partial failure.
562
809
 
563
810
  **Required parameters:**
564
811
 
@@ -582,10 +829,6 @@ If any step fails, the workflow stops, reports which steps completed with all cr
582
829
  | `androidPackageName` | `"com.experitest.ExperiBank"` |
583
830
  | `iosBundleId` | `"com.experitest.ExperiBank"` |
584
831
 
585
- Steps 1 and 6 check for an existing group/project by name before creating — safe to re-run after a partial failure.
586
-
587
- ---
588
-
589
832
  #### `close_poc` — Wind down without deleting
590
833
 
591
834
  Reverses the device and user changes from `create_poc` while leaving the project and device group intact:
@@ -598,27 +841,23 @@ Reverses the device and user changes from `create_poc` while leaving the project
598
841
 
599
842
  Only `customerName` is required.
600
843
 
601
- > The Digital.ai REST API does not expose a user lock or disable endpoint. Deleting the account via `delete_user` is the only API-available way to remove access for users provisioned solely for the POC.
602
-
603
- ---
844
+ > The Digital.ai REST API has no endpoint to lock or disable a user. Deleting the account via `delete_user` is the only way the API offers to remove access for users created only for the POC.
604
845
 
605
846
  #### `delete_poc` — Full teardown
606
847
 
607
848
  Performs the same wind-down as `close_poc`, then permanently deletes the device group and the project. Requires `confirmDeletion: true`.
608
849
 
609
- Before any destructive action, the workflow gathers a full inventory (project ID, group ID, device list, user breakdown) and presents it for explicit confirmation. Unlike `close_poc`, user processing is batch-confirmed via this upfront inventory rather than per-user prompts.
850
+ Before any destructive action, the workflow gathers a full inventory (project ID, group ID, device list, user breakdown) and presents it for explicit confirmation. Unlike `close_poc`, user processing is confirmed once for the whole batch via this inventory rather than user by user.
610
851
 
611
852
  **Best practice:** Run `close_poc` first — it winds down the POC while preserving the project and device group. Note that `close_poc` is not fully reversible: device moves and tag removal can be undone, but it permanently deletes POC-only user accounts (each deletion requires per-user operator confirmation). Use `delete_poc` only once the project data is confirmed no longer needed. Calling `delete_poc` without `confirmDeletion: true` returns a safe explanation and suggests `close_poc` as the alternative.
612
853
 
613
- ---
614
-
615
854
  ### Project Lifecycle
616
855
 
617
- Three tools mirror the POC lifecycle for general project management. They apply the same multi-project user protection rules: users who belong to other projects are revoked from this project only, never deleted from the platform.
856
+ Three tools mirror the POC lifecycle for general project management. They follow the same rule for users who belong to several projects: those users are removed from this project only, never deleted from the platform.
618
857
 
619
858
  #### `setup_project` — Provision a new project environment
620
859
 
621
- At the start, the workflow asks whether to create a **simple** project record only, or a **full** environment (device group, device allocation, user provisioning, app assignment).
860
+ If you call it with only a project name (no `region`, `deviceCount` or `users`), the workflow first asks whether to create a **simple** project record only, or a **full** environment (device group, device allocation, user provisioning, app assignment) — and for a full environment, collects those parameters. If any of them are supplied, it goes straight to the full setup and asks you to confirm the parameters.
622
861
 
623
862
  Full setup steps:
624
863
 
@@ -637,12 +876,12 @@ Full setup steps:
637
876
  | Parameter | Description |
638
877
  |---|---|
639
878
  | `projectName` | The name for the new project |
640
- | `region` | Target region for device selection |
641
879
 
642
880
  **Optional parameters:**
643
881
 
644
882
  | Parameter | Default | Description |
645
883
  |---|---|---|
884
+ | `region` | _(none)_ | Target region for device selection — needed for a full setup; without it, devices are selected from all regions |
646
885
  | `deviceCount` | `6` | Total devices to allocate |
647
886
  | `iosCount` | `ceil(deviceCount / 2)` | iOS device count |
648
887
  | `androidCount` | `floor(deviceCount / 2)` | Android device count |
@@ -654,177 +893,100 @@ Full setup steps:
654
893
  | `iosBundleId` | `"com.experitest.ExperiBank"` | iOS bundle ID used to find the app |
655
894
  | `projectMemo` | _(none)_ | Notes to record on the project |
656
895
 
657
- ---
658
-
659
896
  #### `close_project_resources` — Release resources without deleting
660
897
 
661
- Wind down the project environment while leaving the project record intact:
898
+ Winds down the project environment while leaving the project record intact:
662
899
 
663
900
  1. Locate the project and its associated device group
664
901
  2. List the devices in the project group
665
902
  3. Remove the project-derived tag from each device
666
903
  4. Return devices to the Default group
667
- 5. Process users: **delete accounts with no other project memberships; revoke project access only for multi-project users** (each action individually confirmed by the operator)
904
+ 5. Process the users carrying the project tag (applied by `setup_project`): **delete accounts with no other project memberships; revoke project access only for multi-project users** (each action individually confirmed by the operator)
668
905
 
669
906
  Only `projectName` is required. Active sessions and installed apps on the devices are not touched — release sessions with `release_orphaned_sessions` and remove apps with `uninstall_application` separately if needed.
670
907
 
671
- ---
672
-
673
908
  #### `teardown_project` — Full project deletion
674
909
 
675
910
  Performs all `close_project_resources` steps, then permanently deletes the device group and the project. Requires `confirmDeletion: true`.
676
911
 
677
- The workflow presents a full inventory summary before any irreversible action. The same multi-project user protection applies.
912
+ The workflow presents a full inventory summary before any irreversible action; as with `delete_poc`, user processing is confirmed once for the whole batch via this inventory rather than user by user. Users who belong to other projects are protected in the same way.
678
913
 
679
914
  **Best practice:** Run `close_project_resources` first, then confirm the project is no longer needed before calling `teardown_project`.
680
915
 
681
- ---
682
-
683
- ## Boilerplate Generation
684
-
685
- ### Two ways to create a test
686
-
687
- The server supports two test-creation modes for both mobile and web, and the agent is instructed to route between them based on context:
688
-
689
- | | **Autonomous** (`get_test_boilerplate` / `get_web_test_boilerplate`) | **Interactive** (`start_inspection_session` / `start_browser_inspection_session` / `collaborative_test_creation` / `collaborative_web_test_creation` prompt) |
690
- |---|---|---|
691
- | **When** | Intent is specific — a standardized flow ("create a login test") or step-level detail — AND selectors are derivable (app source in the workspace, or a prior inspection session) | Request is vague ("I want a test for app X"), no source access (chat-only, or IDE without this app's code), or the user wants to watch/drive on a live device or browser |
692
- | **User involvement** | None required — the agent writes the test from boilerplate + source knowledge + best practices | Collaborative — the agent builds the test step by step with the user (browser sessions relay screenshots; mobile sessions share a live device view URL) |
693
- | **Output** | In an IDE: a local test automation project. Chat-only: portable project files presented inline | Same, plus selectors verified against the real app or page along the way |
694
-
695
- **Hybrid is often best:** even in autonomous mode, the agent can open a short inspection session — no user involvement — to capture or verify element IDs or CSS selectors against the real build. The agent is explicitly instructed to **never fabricate selectors** — when in doubt it verifies live, and when the mode itself is ambiguous it asks.
696
-
697
- > **Stronger guarantees:** the server ships this routing policy automatically in its MCP `instructions`, and both boilerplate tools structurally refuse to emit code for a real app or URL without verified selectors. Because not every client surfaces server `instructions` — and no MCP server can intercept a file an agent writes by hand — you can install a client-side reinforcement for a stronger, client-independent guarantee. See **[Recommended agent guardrails](docs/recommended-agent-guardrails.md)**.
698
-
699
- ### `get_test_boilerplate`
700
-
701
- Generates a complete, pre-configured Appium test script. The Digital.ai server URL and access key are pre-filled from the active connection profile — switch profiles with `switch_environment` first to generate scripts carrying a project-scoped key instead of your admin key.
702
-
703
- #### Test Orchestrator (default for Java on Appium Server)
704
-
705
- For **Java (JUnit 5 / TestNG) on an Appium Server project**, the generated project is wired for the **Digital.ai Test Orchestrator** (the Smart Agent): failed tests are retried automatically, each attempt is reported separately, the real pass/fail is synced to Reporter, and runs are grouped by Build ID — with no change to the test code. Appium Grid, Python and NodeJS projects get the classic boilerplate. Pass `orchestration: "off"` to get the classic boilerplate for Java too.
706
-
707
- | Step | What happens |
708
- |---|---|
709
- | 1. `get_test_boilerplate` | Adds conditional agent wiring to `build.gradle` / `pom.xml`, an `orchestrator/config.template.yml` (retries default to 2; 0 for placeholder or performance tests), plus `orchestrator/.gitignore` and `lib/.gitignore` so the rendered key and the agent JAR can never be committed. No root `.gitignore` is written (it would overwrite yours). |
710
- | 2. `install_test_orchestrator_agent(projectDir)` | Downloads the agent on demand, verifies its SHA-256 and places it at `lib/smart-agent.jar`. Under Docker it returns a download command (bash and PowerShell) that does the same and deletes the file on a checksum mismatch. If the download fails, the response gives the URL, the checksum and the target path for a manual install. |
711
- | 3. Set `DIGITAL_AI_ACCESS_KEY` and run `gradle test` / `mvn test` | The agent attaches only when the JAR is present **and** the key is set; the key is rendered into the git-ignored `orchestrator/rendered/` at build time, never into a tracked file, and Java 17 is required only in that case. Otherwise the project builds and runs exactly like the classic boilerplate (Gradle prints why the agent was not attached). Set `BUILD_ID` to group a CI build's results. |
712
-
713
- Requirements: a **Java 17 or 21** JDK with a current truststore, JUnit 5 or TestNG 7+, Appium java-client 8–10. See [Limitations](docs/limitations.md#17-test-orchestrator-known-issues) for known issues.
714
-
715
- The agent JAR is **not** shipped with this server. By default it is downloaded from the [Digital.ai Test Orchestrator sample repository](https://github.com/raheekhandigitalai/Digital.ai-Testing-Test-Orchestrator), pinned to a fixed commit and verified by SHA-256 (see `resources/test-orchestrator/agent.json`). To use a different location — for example an internal mirror — set `TEST_ORCHESTRATOR_JAR_URL` **and** `TEST_ORCHESTRATOR_JAR_SHA256` (both required; a URL without a checksum is refused).
716
-
717
- | Parameter | Values | Default |
718
- |---|---|---|
719
- | `platform` | `android` \| `ios` | _(required)_ |
720
- | `language` | `java-junit5` \| `java-testng` \| `nodejs` \| `python` | _(required)_ |
721
- | `appId` | Numeric app ID from `list_applications` | Recommended |
722
- | `deviceCategory` | `PHONE` \| `TABLET` | `PHONE` |
723
- | `testName` | Any string | `"My First Mobile Test"` |
724
- | `packageName` | Android package name (e.g. `com.mycompany.app`) | _(optional)_ |
725
- | `mainActivity` | Android main activity (e.g. `.MainActivity`) | _(optional)_ |
726
- | `bundleIdentifier` | iOS bundle ID (e.g. `com.mycompany.app`) | _(optional)_ |
727
- | `region` | Region code from `find_available_device` response (e.g. `US2`, `SG1`) | _(optional, strongly recommended)_ |
728
- | `projectType` | `standalone-gradle` \| `standalone-maven` \| `android-gradle-submodule` | `standalone-gradle` |
729
- | `includePerformanceTransactions` | `true` \| `false` | `false` |
730
- | `includeAxeScan` | `true` \| `false` | `false` |
731
- | `confirmSelectorsVerified` | `true` \| `false` | _(optional)_ |
732
- | `orchestration` | `auto` \| `on` \| `off` | `auto` |
733
- | `orchestrationMaxRetries` | `0`–`5` | `2` (`0` for placeholder or performance tests) |
734
- | `automotiveProjection` | Android Auto / CarPlay resolution | _(optional)_ |
735
- | `outputFormat` | `json` \| `human` | `json` |
736
-
737
- > **Inspection gate:** when you target a real app (`appId`/`packageName`/`bundleIdentifier`), this tool returns **no code** and a structured `{ status: "blocked", reason: "no_verified_selectors" }` error UNLESS a live inspection session exists in the MCP process, or you set `confirmSelectorsVerified: true` (escape hatch for selectors already captured via rdb/UIAutomator, `open_mobile_studio`, or authoritative source). The built-in ExperiBank demo (no app identifiers) is never gated. This makes it structurally impossible to receive a placeholder scaffold and pass it off as a finished test.
738
-
739
- **Recommended workflow** for building a new test:
740
-
741
- 1. `get_application_info` — confirm package name, launch activity, and app ID
742
- 2. `find_available_device` — select a healthy device and read its `region`
743
- 3. `start_inspection_session` (or `open_mobile_studio`) — capture real element IDs
744
- 4. `get_test_boilerplate(appId, region, …)` — generates code while the inspection session is live, or with `confirmSelectorsVerified: true`
745
- 5. `validate_test_script` — before saving or presenting the test
746
- 6. Run the test, then `release_device` — explicit cleanup
747
-
748
- **Performance transactions:** Pass `includePerformanceTransactions: true` to bracket the test body with `startPerformanceTransaction` / `endPerformanceTransaction` calls. The start arg is the NV network profile (defaults to `"Monitor"` — observe without throttling); the end arg is the transaction name. The platform records CPU, memory, battery, and Speed Index metrics; results appear in the reporter Transactions tab (~1 min after `endPerformanceTransaction`) and are queryable via `list_transactions`. Pre-requisite: an NV server must be ONLINE and tunnel-connected in the device region (`list_nv_servers`). Note: a throttling profile activates NV shaping immediately; `"Monitor"` is pass-through and carries no ANR risk.
916
+ [↑ Back to top](#table-of-contents)
749
917
 
750
- **Accessibility scanning:** Pass `includeAxeScan: true` to inject a Deque Axe DevTools Mobile accessibility scan (`mobile: axeScan` executeScript call). Sets the required `appium:automationName` capability automatically (`AxeUiAutomator2` for Android, `AxeXCUITest` for iOS). Also pins `appiumVersion` to 2.16.2 (the Axe driver is unavailable on Appium Server 3.3.0+; don't raise it). The key comes from `AXE_DEVTOOLS_API_KEY`; if that's unset, a placeholder is written.
918
+ ---
751
919
 
752
- Both flags can be combined — the Axe scan runs inside the performance transaction boundary.
920
+ ## Safety Guards
753
921
 
754
- **Element locators:** Use `open_mobile_studio` first — it is the platform's native UI Inspector and requires no local tooling. For scenarios requiring direct ADB access (file push, shell commands), use `get_remote_debug_command` to connect the device locally.
922
+ All destructive operations require `confirmDeletion: true`. Without it:
755
923
 
756
- > ⚠️ The generated script is an **end-product artifact** — do not execute it as a discovery step. Running it without known element selectors creates Incomplete sessions visible to the whole team. Discover selectors with `open_mobile_studio` first.
924
+ > ⚠️ Safety guard triggered. "Delete user 42" is a destructive operation that cannot be undone. To confirm you want to proceed, include `confirmDeletion: true` in your request. No changes were made.
757
925
 
758
- ### `validate_test_script`
926
+ The first call describes what will be deleted. The second call — with `confirmDeletion: true` — carries it out. This prevents accidental data loss when an AI assistant misreads your intent. `release_orphaned_sessions` works the same way: without `confirmDeletion: true` (or with `dryRun: true`) it only previews the devices it would release.
759
927
 
760
- Delivery backstop for the case the `get_test_boilerplate` gate cannot catch — a test written by hand with guessed selectors. Pass the full `scriptContent` before presenting or saving any test; it scans for unreplaced `<…>` placeholder selectors, the deliberate scaffold fail-guard, placeholder/fabricated credentials, and known fabricated resource IDs. Returns `isError` with a `fail` verdict when any high-severity pattern is found, so a non-functional test cannot be delivered as finished. A `pass` is necessary but not sufficient — it confirms obvious placeholders are gone, not that selectors are real; a live inspection remains the authoritative source.
928
+ Two other actions have their own confirmation flags:
761
929
 
762
- | Parameter | Values | Default |
763
- |---|---|---|
764
- | `scriptContent` | The full test script text | _(required)_ |
765
- | `fileName` | Label for the result | _(optional)_ |
766
- | `outputFormat` | `json` \| `human` | `json` |
930
+ - **`confirmPublicShare`** — `share_test_report` creates a public link anyone can open without logging in, and it cannot be revoked early (see [limitation 22](docs/limitations.md#22-public-report-links-cannot-be-revoked)). Without the flag it returns a preview only.
931
+ - **`confirmLargeExport`** — usage-report tools stop before an unfiltered export spanning more than 31 days (it can reach hundreds of MB) and ask. Only you should approve such an export, not the AI on its own (see [limitation 21](docs/limitations.md#21-usage-reports-2-year-retention-large-exports)).
767
932
 
768
- `projectType` controls the Java output layout only (ignored for Node.js and Python): `standalone-gradle` produces `src/test/java/` with both `build.gradle` and `pom.xml`; `standalone-maven` produces `pom.xml` only; `android-gradle-submodule` scopes files under `e2e-tests/` for embedding in an existing Android Studio project.
933
+ More checks run before any request leaves the server:
769
934
 
770
- **Files generated per language:**
935
+ - **Permission check first:** report-deletion tools (`delete_test_reports*`, `cleanup_inspection_sessions`, `cleanup_browser_inspection_sessions`) check your detected access level first. A project-level role is refused up front when its project's `allowUsersDeleteTests` setting is readable and false, with a message telling you to have a Cloud Admin enable it (or switch to a Cloud Admin profile) — instead of an unexplained 403.
936
+ - **Path validation:** tools that read or write local files require an absolute path and refuse path traversal (`..`). Upload tools also refuse credential-file names (`.env*`, SSH private keys), so a misdirected request cannot publish secrets to the cloud repository.
771
937
 
772
- | Language | Files |
773
- |---|---|
774
- | `java-junit5` / `java-testng` | `AndroidNative.java` or `iOSNative.java` + `build.gradle` + `pom.xml` |
775
- | `nodejs` | `wdio.conf.js` + test file + setup shell steps (WebdriverIO 7.40 for Appium Grid projects, WebdriverIO 9 for Appium Server) |
776
- | `python` | Test file + `requirements.txt` |
938
+ [↑ Back to top](#table-of-contents)
777
939
 
778
- All boilerplate defaults to the ExperiBank demo app as the starting point. ExperiBank is available on most farm devices and can verify connectivity before switching to your own app.
940
+ ---
779
941
 
780
- ### `get_web_test_boilerplate`
942
+ ## Troubleshooting
781
943
 
782
- Generates a complete, pre-configured Selenium test script for browser testing. The Digital.ai server URL and access key are pre-filled from the active connection profile.
944
+ ### Common symptoms
783
945
 
784
- | Parameter | Values | Default |
946
+ | Symptom | Cause | Fix |
785
947
  |---|---|---|
786
- | `language` | `java-junit5` \| `java-testng` \| `nodejs` \| `python` | _(required)_ |
787
- | `url` | URL of the page being tested | _(optional)_ |
788
- | `testName` | Any string | `"WebTest"` |
789
- | `targetBrowser` | `chrome` \| `firefox` \| `MicrosoftEdge` \| `safari` \| `opera` (names as `list_available_browsers` returns them; `edge` is accepted for `MicrosoftEdge`) | _(optional)_ |
790
- | `shadowDomSupport` | `auto` \| `always` \| `never` | `auto` |
791
- | `confirmSelectorsVerified` | `true` \| `false` | _(optional)_ |
792
- | `outputFormat` | `json` \| `human` | `json` |
793
-
794
- > **Inspection gate:** when you target a real URL (`url` provided), this tool returns **no code** and a structured `{ status: "blocked", reason: "no_verified_selectors" }` error UNLESS a live browser inspection session exists in the MCP process, or you set `confirmSelectorsVerified: true` (escape hatch for selectors captured via a prior `start_browser_inspection_session` session). This mirrors the mobile boilerplate gate.
795
-
796
- **Browser-neutral by default:** when `targetBrowser` is omitted, the generated script reads the browser name at runtime from an environment variable (`BROWSER`). This means the same script runs on Chrome, Firefox, Microsoft Edge, Safari and Opera without modification — just change the env var or CI matrix. Specify `targetBrowser` only when you need browser-specific options (e.g. Safari-only session setup).
948
+ | `403 Forbidden` on admin tools (agents, regions, license) | Active profile has a project-level role (Project Admin or Project User) — v2 endpoints require Cloud Admin access | Ask the AI: *"switch to my Cloud Admin profile"* — it will call `list_environments` and pick the right one. See [Using multiple keys](#using-multiple-keys) |
949
+ | Report delete tools refuse with "Reporter deletes are disabled for project …" | The project's `allowUsersDeleteTests` setting is false, so project-level roles can't delete reports | Ask a Cloud Admin to enable it for the project, or switch to a Cloud Admin profile |
950
+ | `install_application` returns 400 on a device you can see | Device is reserved via an rdb (remote debug) session | Install **first**, then run `get_remote_debug_command` — not the other way around |
951
+ | `install_application` returns 400 (no rdb involved) | App not assigned to a project containing the target device | Call `assign_app_to_project` first |
952
+ | Repeatable `NoSuchElementException` while sibling tests pass | Device health, not test code — device stuck in a wrong state or offline-but-pooled | Run `get_device_health_summary`; scope the deviceQuery with `@region='<healthy-region>'` |
953
+ | Tests missing from `list_test_reports` results | Project has its own reporter instance — unscoped queries search the default scope | Pass `projectName` (exact name from `list_projects`) |
954
+ | rdb fails: `validation error / Failed to reserve device` | With a project-level role, the serial lookup returned an internal device ID instead of the real serial (the `get_remote_debug_command` response warns about this) | Ask the AI: *"switch to Cloud Admin"* → regenerate the rdb script → switch back. See [limitation 13](docs/limitations.md#13-get_remote_debug_command-requires-cloud-admin-for-reliable-serial-resolution) |
955
+ | Device query returns nothing for `@manufacturer` / `@tag` | These fields are silently ignored server-side | Use the `manufacturer`/`tags` parameters on `find_available_device` — they filter client-side |
956
+ | A downloaded file isn't on your machine, or an upload can't find your file (Docker) | Under Docker, file tools use the container's filesystem | See [Local File Transfer](#local-file-transfer-docker--remote-deployments-only) |
797
957
 
798
- **Shadow DOM support:** when `shadowDomSupport` is `auto` (the default) or `always`, the generated script includes a `shadowQuery(host, cssSelector)` helper that traverses into shadow roots using `executeScript`. Use this for apps built with Web Components, Lit, Polymer, or other shadow-DOM frameworks. `get_page_dom` detects shadow DOM automatically during inspection and reports `hasShadowDom: true` when present.
958
+ ### `Failed to reconnect to digital-ai-testing: -32000`
799
959
 
800
- **Recommended workflow for building a new web test:**
960
+ On Windows with Docker, this usually means the `--env-file` path was stored with its backslashes stripped. Open `~/.claude.json`, find the `mcpServers` entry for this project, and check that the env file path looks right (e.g. `C:/projects/digital-ai-testing-mcp/.env`). If it shows something like `C:projectsdigital-ai-testing-mcp.env`, the backslashes were removed. Fix the path in the file directly (use forward slashes), then run `/mcp` in Claude Code to reconnect.
801
961
 
802
- 1. `list_available_browsers` — confirm which browsers and OS combinations are available in your region
803
- 2. `start_browser_inspection_session` — open a live browser session on the target URL
804
- 3. `get_page_dom` — extract the rendered DOM, including shadow trees; note element IDs and `data-testid` values
805
- 4. `find_web_elements` / `tap_element` / `type_into_element` — interact with elements to verify selectors work
806
- 5. `take_inspection_screenshot` — relay screenshots at each key step for verification
807
- 6. `stop_browser_inspection_session`
808
- 7. `get_web_test_boilerplate(confirmSelectorsVerified: true)` — generate the verified script
962
+ ### Workflow tools failing?
809
963
 
810
- ---
964
+ Run this sequence:
811
965
 
812
- ## Performance Comparison Reports
966
+ ```
967
+ 1. get_server_info — confirm tool count (expect 202; 203 with MCP_DEBUG_MODE=true) and active profile
968
+ 2. check_workflow_readiness — which dependency tools are present or missing
969
+ 3. check_connectivity — confirm the backend API is reachable
970
+ ```
813
971
 
814
- Compare performance (Speed Index, CPU, memory, battery, duration) between two sets of conditions — app v1 vs v2, device A vs device B, OS version X vs Y, region to region, two network profiles, or two automation scripts. Works with all access levels — Cloud Admin sees all projects; project-level keys (Project Admin and Project User) see only their own project's transactions.
972
+ `check_workflow_readiness` returns a structured report:
815
973
 
816
- The headline metric is reported three ways every time — **trimmed mean, median, and raw mean** — so a small or noisy sample can't hide behind a single number.
974
+ ```json
975
+ {
976
+ "allWorkflowsReady": true,
977
+ "registeredToolCount": 202,
978
+ "workflows": {
979
+ "create_poc": { "ready": true, "missingRead": [], "missingWrite": [] },
980
+ "setup_project": { "ready": true, "missingRead": [], "missingWrite": [] }
981
+ }
982
+ }
983
+ ```
817
984
 
818
- | Tool | Purpose |
819
- |---|---|
820
- | `compare_performance_transactions` | Two-set comparison; trimmed-mean/median/mean per side + delta and % change. MAD outliers excluded by default (≥4 samples/side). Pass `comparisonAxis` to embed a confound check. |
821
- | `assess_comparison_confounds` | Verdict — **clean / caveated / confounded** — by flagging any dimension other than the declared axis (device model, OS, OS version, network profile, project, transaction name) that varies across or within the sides, plus missing telemetry and sample imbalance. |
822
- | `detect_performance_outliers` | Robust median/MAD outlier flagging on a single set; returns the kept set and recommended exclusions/re-runs. |
823
- | `performance_transaction_control` | Phase 2: generate fresh samples inside an inspection session — `start` (with an NV profile) → run the verified flow → `end` (names the record). Records appear in the reporter ~1 min later. |
985
+ If `ready` is `false`, `missingRead` and `missingWrite` list exactly which tools are absent. The most common cause is an outdated install — see [Upgrading](#upgrading).
824
986
 
825
- > **Why the confound check matters:** comparing two transactions with Speed Index 1015 vs 1000 looks like a 1.5% version delta — but if they ran on *different device models, different OS versions, and different projects*, and one has no CPU/memory telemetry, `assess_comparison_confounds` returns **confounded** and names each cause, so the delta is never misread as a version regression.
987
+ The server also logs a readiness check at startup (visible in your terminal, or `docker logs`/your AI client's MCP server logs if running in a container): `Workflow readiness: all workflows ready ✓`, or `⚠️ DEGRADED: create_poc — missing: ...` when tools are absent.
826
988
 
827
- The **`performance_comparison_report` prompt** orchestrates the full workflow end to end: define the axis → scrub confounds → negotiate sample size → **require explicit plan confirmation with a time estimate** → run the series with outlier-driven re-runs → report the delta with root-cause reasoning. True host/background interference is not directly observable; the report approximates device quietness (idle, healthy status, no concurrent reservation) and states that limit explicitly.
989
+ [↑ Back to top](#table-of-contents)
828
990
 
829
991
  ---
830
992
 
@@ -839,7 +1001,7 @@ All data-returning tools accept an `outputFormat` parameter:
839
1001
  | `"json"` | ✅ Yes | Chaining tool calls — IDs and values pass directly to the next tool without parsing |
840
1002
  | `"human"` | No | Displaying results to an operator — formatted prose with status indicators |
841
1003
 
842
- The default is `"json"` because the primary consumers are AI agents that need to extract IDs for follow-up calls. Agents can render JSON for human display; they cannot reliably parse prose for IDs.
1004
+ The default is `"json"` because the main consumers are AI agents that need to pick out IDs for follow-up calls. An agent can turn JSON into readable output for you; it can't reliably pull IDs out of prose.
843
1005
 
844
1006
  ```
845
1007
  # Example: pass device group IDs without string manipulation
@@ -852,13 +1014,13 @@ get_devices_in_group { groupId: "7" }
852
1014
  add_devices_to_group { groupId: "12", deviceIds: ["abc123"] }
853
1015
  ```
854
1016
 
855
- Tools that only perform mutations (create, delete, install, etc.) return simple success strings and do not have an `outputFormat` parameter.
1017
+ Tools that only make changes (create, delete, install, etc.) return simple success messages and have no `outputFormat` parameter. One exception to the `json` default: `get_page_dom` defaults to `human` (a readable element list); pass `outputFormat: "json"` to chain its results.
856
1018
 
857
1019
  ### Understanding maxResults
858
1020
 
859
- The Digital.ai API returns complete datasets — there is no server-side pagination for most endpoints. The MCP caps responses at `maxResults` (default: 50, max: 500) to keep response sizes manageable for AI assistants.
1021
+ The Digital.ai API returns complete datasets — most endpoints have no server-side paging. The MCP caps list responses at `maxResults` (default: 50, max: 500) to keep responses a manageable size for AI assistants. `list_test_reports` uses `limit` (same default and maximum) plus `page`, and `find_elements` details at most 20 matches by default (see [limitation 19](docs/limitations.md#19-find_elements-details-only-the-first-20-matches)).
860
1022
 
861
- Use filters to narrow results rather than raising `maxResults`. When results are truncated, you will see:
1023
+ Use filters to narrow results rather than raising `maxResults`. When results are cut off, you will see:
862
1024
 
863
1025
  > ⚠️ Showing 50 of 312 results. Use filters to narrow results, or increase maxResults (max 500) to see more.
864
1026
 
@@ -873,110 +1035,21 @@ Use filters to narrow results rather than raising `maxResults`. When results are
873
1035
  | `list_projects` | name (partial match) | name, id |
874
1036
  | `list_project_users` | username (partial), role (exact) | username, role |
875
1037
  | `list_applications` | nameContains, osType, packageName, bundleIdentifier, uniqueName, fileType, isForSimulator | applicationName, releaseVersion, buildVersion, createdAt (default), osType, fileType |
876
- | `list_test_reports` | status, name, has_attachment, success; startDate/endDate (date range) | start_time |
877
-
878
- All list tools accept `sortOrder: "asc" | "desc"` (default `"asc"`, except `list_applications`, which defaults to `"desc"`, newest first).
879
-
880
- `list_test_reports` sorting is performed by the platform for every access level. If the platform ever refuses it for a credential, the server transparently falls back to a client-side scan (up to 5,000 records) so "newest first" results stay correct, just slower on large report sets.
881
-
882
- ### Test Reporting Schema
883
-
884
- Report fields use **snake_case** to match the API response:
885
-
886
- | Field | Type | Description |
887
- |---|---|---|
888
- | `uuid` | string | Unique test execution identifier |
889
- | `test_id` | number | Numeric reporter ID |
890
- | `name` | string | Test name |
891
- | `status` | string | `Passed`, `Failed`, `Incomplete`, `Skipped`, `Error`, `Healed` |
892
- | `success` | boolean | True when status is Passed |
893
- | `start_time` | string | ISO 8601 execution start timestamp |
894
- | `duration` | number \| null | Milliseconds (`null` for in-progress sessions) |
895
- | `project_id` | number | Owning project |
896
- | `has_attachment` | string | `"Y"` or `"N"` |
897
-
898
- **Diagnostic fields — `get_test_report` (single record) only, NOT present on `list_test_reports`:**
899
-
900
- | Field | Type | Description |
901
- |---|---|---|
902
- | `cause` | string | Human-readable failure cause (e.g. `"no such element - …"`) |
903
- | `errorCategory` | string | Broad bucket (e.g. `assertion`) |
904
- | `errorClassification` | string | Specific class (e.g. `element_not_found`) |
905
- | `errorDetail` | string | Full error object including stack trace |
906
- | `subTestCount` / `failedSubTestCount` | number | Sub-test totals for merged reports |
907
-
908
- To diagnose a failure, list with `list_test_reports`, then call `get_test_report(testId)` per test — the list view does not carry the diagnostic fields (the backend omits them there).
909
-
910
- **Confirmed working filter properties:** `status`, `name` (with `contains` for substring match), `has_attachment`, `success`, `test_id`, `project_id`, `device.os` (case-sensitive: `"Android"` / `"iOS"`), `duration`, `attachment_count`, `attachments_size`, `status_code`. Operators: `=`, `>`, `<`, `>=`, `<=`, `contains`.
911
-
912
- **CSRF-blocked filters** — return 401 regardless of key type: `start_time`, `create_time`, `uuid`. Use the `startDate`/`endDate` parameters on `list_test_reports` for date-range filtering instead — these are applied client-side after fetching.
913
-
914
- **Reporter ID types:**
915
-
916
- | Identifier | Source | How to retrieve |
917
- |---|---|---|
918
- | `uuid` | List results, report URLs (`/reporter/video-report/<uuid>`) | `get_test_report(uuid: "...")` or `get_test_report(reportUrl: "...")` — preferred, globally unique |
919
- | `test_id` | List results | `get_test_report(testId: N)` — unique only within a project; the same number can be a different test in another project |
920
- | `report_api_id` | Returned by session-start tools | `get_test_by_report_id(reportApiId: "...")` — wait ~60 s after session close |
921
-
922
- **Examples:**
923
-
924
- ```
925
- # Recent failures
926
- list_test_reports
927
- filter: [{"property":"status","operator":"=","value":"Failed"}]
928
- sort: [{"property":"start_time","descending":true}]
929
- limit: 20
930
-
931
- # Date range
932
- list_test_reports
933
- startDate: "2026-05-01T00:00:00Z"
934
- endDate: "2026-05-15T23:59:59Z"
935
-
936
- # Pass/fail by OS
937
- get_grouped_test_reports
938
- groupBy: ["device.os"]
939
- pivotBy: ["status"]
940
-
941
- # Project summary
942
- get_project_test_summary
943
- startDate: "2026-05-01T00:00:00Z"
944
- projectName: "My Project"
945
- ```
946
-
947
- #### Root cause analysis of a failure
948
-
949
- ```
950
- # 1. Find the failures
951
- list_test_reports
952
- filter: [{"property":"status","operator":"=","value":"Failed"}]
953
-
954
- # 2. Diagnose — structured cause without any download
955
- get_test_report(testId: 69)
956
- → cause, errorCategory, errorClassification, errorDetail (stack trace)
957
-
958
- # 2b. Platform root-cause analysis (read-only)
959
- get_root_cause_analysis(uuid: "…") # or reportUrl / testId
1038
+ | `list_test_reports` | `filter` array (see [Test reporting schema](docs/tools.md#test-reporting-schema)), `searchValue` (name substring), startDate/endDate (date range), projectId/projectName | `sort` array, e.g. `start_time` descending |
960
1039
 
961
- # 3. Full Appium log as text, when the structured fields aren't enough
962
- get_test_log(uuid: "…", logType: "appium")
1040
+ The list tools with `sortBy` also accept `sortOrder: "asc" | "desc"` (default `"asc"`, except `list_applications`, which defaults to `"desc"`, newest first).
963
1041
 
964
- # 4. Pull the binary artifacts (session video .mp4, full ZIP) to YOUR machine.
965
- # With the npm install, call download_test_video / download_test_attachments
966
- # directly; under Docker/remote, generate a local command instead:
967
- get_test_attachments_download_command(uuid: "…", localPlatform: "macos")
968
- → a curl command you run locally; the file never transits the container
969
- ```
1042
+ `list_test_reports` sorting is done by the platform for every access level. If the platform ever refuses it for a credential, the server automatically falls back to scanning up to 5,000 records itself, so "newest first" results stay correct, just slower on large report sets.
970
1043
 
971
1044
  ### Local File Transfer (Docker / remote deployments only)
972
1045
 
973
- **If you installed via the recommended [npm package](#option-a--install-via-npm-recommended) (Option A) and run the server on your own machine, skip this section** — `download_*`/`upload_*` tools already read/write your local filesystem directly, no workaround needed.
1046
+ **If you installed the [npm package](#option-a--install-via-npm-recommended) (Option A) and run the server on your own machine, skip this section** — the `download_*`/`upload_*` tools already read and write your local files directly.
974
1047
 
975
- The rest of this section applies only to Docker/remote deployment (Option B or C). There, any tool that reads or writes a local file path operates on the **MCP server's own filesystem** — which, in the published Docker image or a remote deployment, is **not** your machine. A `download_*` tool writes into the container; an `upload_*` tool reads from it. Neither is reachable from your local shell unless you volume-mount a shared directory.
1048
+ The rest of this section applies only to Docker or remote deployments (Option B or C). There, any tool that reads or writes a local file path works on the **MCP server's own filesystem** — which, in the published Docker image or a remote deployment, is **not** your machine. A `download_*` tool writes into the container; an `upload_*` tool reads from it. Neither is reachable from your local shell unless you mount a shared folder into the container.
976
1049
 
977
- For those deployments, every file-transfer tool has a **command-generator sibling** that emits a `curl` / PowerShell command you run locally, so the bytes move directly between your machine and the platform — never through the container:
1050
+ For those deployments, every file-transfer tool has a **command-generator** counterpart that produces a `curl` / PowerShell command you run locally, so the file moves directly between your machine and the platform — never through the container:
978
1051
 
979
- | Direction | Server-side tool (shared FS only) | Local-command sibling (Docker/remote) |
1052
+ | Direction | Server-side tool (shared filesystem only) | Local-command counterpart (Docker/remote) |
980
1053
  |---|---|---|
981
1054
  | Upload app | `upload_application_file` | `get_application_upload_command` |
982
1055
  | Upload repo file | `upload_repository_file` | `get_repository_upload_command` |
@@ -987,38 +1060,50 @@ For those deployments, every file-transfer tool has a **command-generator siblin
987
1060
  | Download profile | `download_provisioning_profile` | `get_provisioning_profile_download_command` |
988
1061
  | Download app language files | `extract_app_language_files` | `get_app_language_files_download_command` |
989
1062
  | Download iOS app container | `download_ios_app_container` | `get_ios_app_container_download_command` |
1063
+ | Download usage report CSV | `download_usage_report` | `get_usage_report_download_command` (or `summarize_usage_report`, which returns an aggregated summary with no file at all) |
1064
+ | Install Test Orchestrator agent | `install_test_orchestrator_agent` | the same tool — under Docker it returns bash and PowerShell download commands instead of writing the file |
990
1065
 
991
- For test **logs** specifically, `get_test_log` is simpler still — it returns the Appium/device log text inline in the response, no command or download needed. The generated commands embed the active access key in plaintext, so run them immediately and don't save the output.
992
-
993
- ---
1066
+ For test **logs**, `get_test_log` is simpler still — it returns the Appium/device log text directly in the response, with no command or download needed. The generated commands contain the active access key in plain text, so run them right away and don't save the output. Background: [limitation 15](docs/limitations.md#15-local-file-io-runs-on-the-mcp-servers-filesystem-not-yours).
994
1067
 
995
- ## Safety Guards
1068
+ [↑ Back to top](#table-of-contents)
996
1069
 
997
- All destructive operations require `confirmDeletion: true`. Without it:
1070
+ ---
998
1071
 
999
- > ⚠️ Safety guard triggered. "Delete user 42" is a destructive operation that cannot be undone. Include `confirmDeletion: true` to proceed. No changes were made.
1072
+ ## Known Limitations
1000
1073
 
1001
- The first call describes exactly what will be deleted. The second call — with `confirmDeletion: true` — executes. This prevents accidental data loss when an AI assistant misinterprets intent.
1074
+ The ones you're most likely to run into:
1002
1075
 
1003
- Two further guards run before any request leaves the server:
1076
+ 1. **Appium and Selenium tests aren't started by the server.** You launch them from your own Appium or Selenium client (IDE, CI scripts); the server manages the devices, apps, reservations and results. Espresso, XCUITest and Maestro suites are different — the platform runs them itself (see [Test runs](#test-runs-espresso-xcuitest-maestro)).
1077
+ 2. **iOS inspection sessions differ from Android:** no clearing app data, no clipboard on Appium Grid devices, and back navigation uses the navigation-bar button (iOS has no Back button). See [limitation 29](docs/limitations.md#29-ios-inspection-sessions-differ-from-android).
1078
+ 3. **Appium Grid projects support fewer commands than Appium Server** — for example no `-ios predicate string` locator, no Android Auto / CarPlay projection, no pinch/zoom. See [limitation 18](docs/limitations.md#18-appium-grid-projects-support-fewer-commands-than-appium-server).
1079
+ 4. **Under Docker, file tools work on the container's filesystem, not yours.** Use the command-generator tools instead — see [Local File Transfer](#local-file-transfer-docker--remote-deployments-only) and [limitation 15](docs/limitations.md#15-local-file-io-runs-on-the-mcp-servers-filesystem-not-yours).
1080
+ 5. **Usage reports go back 2 years** (platform 25.9+), and large unfiltered exports need your explicit approval. See [limitation 21](docs/limitations.md#21-usage-reports-2-year-retention-large-exports).
1081
+ 6. **There's no way to disable or lock a user.** Removing access for a user created only for a POC means deleting the account — which is why `close_poc` asks for confirmation per user.
1082
+ 7. **Deleting test reports with a project-level key depends on a per-project setting** (`allowUsersDeleteTests`). Cloud Admin can always delete. When the setting is off, the tools refuse up front for a Project Admin and explain how to get it enabled; a Project User cannot read the setting, so the platform returns its own 403.
1004
1083
 
1005
- - **Auth pre-flight:** report-deletion tools (`delete_test_reports*`, `cleanup_inspection_sessions`) check the API-detected access level first. A project-level role is refused up front only when its project's `allowUsersDeleteTests` setting is false, with a message telling you to have a Cloud Admin enable it (or switch to a Cloud Admin profile) instead of an opaque 403.
1006
- - **Upload path validation:** tools that read local files for upload refuse relative paths, path traversal, and credential-file names (`.env*`, SSH private keys) — a misdirected request cannot publish secrets to the cloud repository.
1084
+ See [docs/limitations.md](docs/limitations.md) for the full list.
1007
1085
 
1008
1086
  ---
1009
1087
 
1010
1088
  ## Development
1011
1089
 
1090
+ To build and run the server from a clone (see [Option C](#option-c--build-from-source)), and for how to contribute, see [CONTRIBUTING.md](CONTRIBUTING.md). Security reports: [SECURITY.md](SECURITY.md).
1091
+
1012
1092
  ```bash
1013
1093
  npm install
1014
1094
  npm run build # compile TypeScript to dist/
1015
1095
  npm run dev # tsx watch for live reload
1096
+ npm run typecheck
1097
+ npm run lint
1016
1098
  ```
1017
1099
 
1018
- **Running tests** — most suites require a live `.env` with valid credentials and call the real Digital.ai API. The exception is `test:tools`, which exercises registered tool handlers (guards, auth gates, path validation) through an in-memory MCP transport with no live API access:
1100
+ **Running tests** — the per-domain suites (devices, users, reporting, …) need a `.env` with valid credentials and call the real Digital.ai API. `test:tools`, `test:utils` and the other unit suites run offline; `test:tools` exercises registered tool handlers (guards, access checks, path validation) through an in-memory MCP transport with no live API access.
1101
+
1102
+ <details>
1103
+ <summary><strong>All test scripts</strong></summary>
1019
1104
 
1020
1105
  ```bash
1021
- npm run test # all tests
1106
+ npm run test # all tests (includes the live suites)
1022
1107
  npm run test:tools # tool-layer guard/gate regression tests (offline)
1023
1108
  npm run test:devices # device management
1024
1109
  npm run test:users # user management
@@ -1032,90 +1117,23 @@ npm run test:device-groups # device groups
1032
1117
  npm run test:repository # file repository
1033
1118
  npm run test:provisioning # iOS provisioning profiles
1034
1119
  npm run test:health # environment health
1035
- npm run test:utils # utility functions
1120
+ npm run test:utils # utility functions (offline)
1036
1121
  npm run test:transactions # performance transactions
1037
1122
  npm run test:analytics # analytics tools
1038
1123
  npm run test:infrastructure # agents / regions / NV servers / sessions
1039
1124
  npm run test:live # live API-behavior probes (requires .env; not part of npm test)
1040
- npm run test:fidelity # agent tool-choice evaluation (builds first)
1125
+ npm run test:fidelity # agent tool-choice evaluation (builds first; spends model tokens)
1041
1126
  npm run remediation:digest # summarize debug-mode remediation notes
1042
1127
  ```
1043
1128
 
1044
1129
  See `package.json` for the full list of scripts.
1045
1130
 
1046
- ---
1047
-
1048
- ## Troubleshooting
1049
-
1050
- ### Common symptoms
1051
-
1052
- | Symptom | Cause | Fix |
1053
- |---|---|---|
1054
- | `403 Forbidden` on admin tools (agents, regions, license) | Active profile is a project-level key — v2 endpoints require Cloud Admin access | Ask Agent: *"switch to my Cloud Admin profile"* — it will call `list_environments` and pick the right one |
1055
- | Report delete tools refuse with "Reporter deletes are disabled for project …" | The project's `allowUsersDeleteTests` setting is false, so project-level roles can't delete reports | Ask a Cloud Admin to enable it for the project, or switch to a Cloud Admin profile |
1056
- | `install_application` returns 400 on a device you can see | Device is reserved via an rdb (remote debug) session | Install **first**, then run `get_remote_debug_command` — not the other way around |
1057
- | `install_application` returns 400 (no rdb involved) | App not assigned to a project containing the target device | Call `assign_app_to_project` first |
1058
- | Repeatable `NoSuchElementException` while sibling tests pass | Device health, not test code — device stuck in a wrong state or offline-but-pooled | Run `get_device_health_summary`; scope the deviceQuery with `@region='<healthy-region>'` |
1059
- | Tests missing from `list_test_reports` results | Project has its own reporter instance — unscoped queries search the default scope | Pass `projectName` (exact name from `list_projects`) |
1060
- | rdb fails: `validation error / Failed to reserve device` | Project-level key resolved an internal device ID instead of the real serial | Ask Agent: *"switch to Cloud Admin"* → regenerate the rdb script → switch back |
1061
- | Device query returns nothing for `@manufacturer` / `@tag` | These fields are silently ignored server-side | Use the `manufacturer`/`tags` parameters on `find_available_device` — they filter client-side |
1062
-
1063
- ### `Failed to reconnect to digital-ai-testing: -32000`
1064
-
1065
- On Windows this usually means the `--env-file` path was stored with backslashes stripped. Open `~/.claude.json`, find the `mcpServers` entry for this project, and check that the env file path looks correct (e.g. `C:/projects/digital-ai-testing-mcp/.env`). If it shows something like `C:projectsdigital-ai-testing-mcp.env`, the backslashes were eaten. Fix the path in the file directly (use forward slashes), then run `/mcp` in Claude Code to reconnect.
1066
-
1067
- ### Workflow tools failing?
1068
-
1069
- Run this sequence:
1070
-
1071
- ```
1072
- 1. get_server_info — confirm tool count (expect 202; 203 with MCP_DEBUG_MODE=true) and active profile
1073
- 2. check_workflow_readiness — which dependency tools are present or missing
1074
- 3. check_connectivity — confirm the backend API is reachable
1075
- ```
1076
-
1077
- `check_workflow_readiness` returns a structured report:
1078
-
1079
- ```json
1080
- {
1081
- "allWorkflowsReady": true,
1082
- "registeredToolCount": 202,
1083
- "workflows": {
1084
- "create_poc": { "ready": true, "missingRead": [], "missingWrite": [] },
1085
- "setup_project": { "ready": true, "missingRead": [], "missingWrite": [] }
1086
- }
1087
- }
1088
- ```
1089
-
1090
- If `ready` is `false`, `missingRead` and `missingWrite` list exactly which tools are absent. The most common cause is a stale build — update it:
1091
-
1092
- ```bash
1093
- # npm install (Option A)
1094
- npm install -g digital-ai-testing-mcp@latest
1095
-
1096
- # Docker (Option B/C)
1097
- docker pull ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest # or: docker build -t digital-ai-testing-mcp:latest .
1098
- ```
1099
-
1100
- The server also logs a readiness check at startup (visible in your terminal, or `docker logs`/your AI client's MCP server logs if running in a container): `Workflow readiness: all workflows ready ✓`, or `⚠️ DEGRADED: create_poc — missing: ...` when tools are absent.
1101
-
1102
- ---
1103
-
1104
- ## Known Limitations
1105
-
1106
- The four most commonly encountered:
1107
-
1108
- 1. **No API to trigger Appium or Selenium test execution.** Those tests launch from Appium clients (IDE, CI scripts); the server manages devices, apps, reservations, and results. Espresso, XCUITest and Maestro suites are different: the platform runs them itself via `execute_test_run` / `get_test_run_status` / `cancel_test_run` / `get_test_run_command` (Maestro is Android-only). Espresso and Maestro runs are verified; XCUITest uses the same API but is not yet verified end to end. Interactive [mobile inspection sessions](docs/tools.md#mobile-inspection-sessions) and [browser inspection sessions](docs/tools.md#web-inspection-sessions) are also live WebDriver sessions for element discovery on Android, iOS, and desktop browsers.
1109
- 2. **Inspection sessions support Android and iOS** on both the legacy Appium Grid (JWP) and Appium Server (W3C/OSS) projects. iOS caveats: no clear-app-data, no clipboard on Grid devices, and back navigation uses the nav-bar button (iOS has no Back button).
1110
- 3. **No user disable/lock endpoint.** Removing access for a user provisioned solely for a POC means deleting the account — which is why `close_poc` requires per-user confirmation.
1111
- 4. **Deleting test reports with a project-level key depends on a per-project setting** (`allowUsersDeleteTests`). Cloud Admin can always delete. When the setting is off, the tools refuse up front and explain how to get it enabled.
1112
-
1113
- See [docs/limitations.md](docs/limitations.md) for the full list.
1131
+ </details>
1114
1132
 
1115
1133
  ---
1116
1134
 
1117
1135
  ## License
1118
-
1136
+
1119
1137
  This project is licensed under the Apache License, Version 2.0. See [LICENSE](./LICENSE) for the full license text.
1120
-
1138
+
1121
1139
  Use of this software is also subject to the [Supplemental Terms of Use](./SUPPLEMENTAL_TERMS.md), which address AI client responsibilities, agentic action risks, liability limitations, and other obligations specific to the Digital.ai MCP Server for Testing. By using this software, you agree to both the Apache 2.0 License and the Supplemental Terms.