digital-ai-testing-mcp 2.0.2 → 2.0.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +641 -623
- package/dist/api/client.d.ts.map +1 -1
- package/dist/api/client.js +2 -4
- package/dist/api/client.js.map +1 -1
- package/dist/api/webdriver.d.ts.map +1 -1
- package/dist/api/webdriver.js +7 -0
- package/dist/api/webdriver.js.map +1 -1
- package/dist/tools/agent-tools.d.ts.map +1 -1
- package/dist/tools/agent-tools.js +3 -1
- package/dist/tools/agent-tools.js.map +1 -1
- package/dist/tools/debug-tools.d.ts +10 -0
- package/dist/tools/debug-tools.d.ts.map +1 -1
- package/dist/tools/debug-tools.js +30 -17
- package/dist/tools/debug-tools.js.map +1 -1
- package/dist/tools/reporting-tools.d.ts.map +1 -1
- package/dist/tools/reporting-tools.js +9 -2
- package/dist/tools/reporting-tools.js.map +1 -1
- package/dist/tools/web-inspection-tools.d.ts.map +1 -1
- package/dist/tools/web-inspection-tools.js +15 -2
- package/dist/tools/web-inspection-tools.js.map +1 -1
- package/dist/utils/api-error.d.ts +3 -0
- package/dist/utils/api-error.d.ts.map +1 -0
- package/dist/utils/api-error.js +47 -0
- package/dist/utils/api-error.js.map +1 -0
- package/dist/utils/remediation.d.ts.map +1 -1
- package/dist/utils/remediation.js +6 -1
- package/dist/utils/remediation.js.map +1 -1
- package/dist/utils/response-formatter.js +1 -1
- package/dist/utils/response-formatter.js.map +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -6,23 +6,28 @@
|
|
|
6
6
|
|
|
7
7
|
# Digital.ai Testing — MCP Server
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
Talk to your Digital.ai Testing device farm from your AI assistant. This MCP (Model Context Protocol) server connects AI clients such as Claude, GitHub Copilot and Cursor to Digital.ai Testing, so you can ask in plain language to find devices, install apps, explore an app on a real device, generate tests, run test suites, investigate failures and compare performance.
|
|
10
|
+
|
|
11
|
+
It provides **202 tools**, **2 resources** and **7 prompts** across 27 capability areas: device management, test execution (including Espresso, XCUITest and Maestro test runs), app lifecycle, reporting and root-cause analysis, analytics, performance, project administration, interactive mobile inspection, interactive browser inspection, and more.
|
|
10
12
|
|
|
11
13
|
---
|
|
12
14
|
|
|
13
15
|
## Quick Start
|
|
14
16
|
|
|
17
|
+
**1. Check that Node.js 22 or later is installed** (npm comes with it). If it isn't, install the LTS version from https://nodejs.org first.
|
|
18
|
+
|
|
15
19
|
```bash
|
|
16
|
-
# 0. Confirm Node.js 22+ is installed (installs npm alongside it)
|
|
17
20
|
node --version
|
|
18
21
|
npm --version
|
|
19
|
-
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
**2. Install the MCP server.**
|
|
20
25
|
|
|
21
|
-
|
|
26
|
+
```bash
|
|
22
27
|
npm install -g digital-ai-testing-mcp
|
|
23
28
|
```
|
|
24
29
|
|
|
25
|
-
Add to your AI client (Claude Desktop shown
|
|
30
|
+
**3. Add it to your AI client** with your tenant URL and [access key](#access-keys). Claude Desktop is shown here; see [Connecting AI Clients](#connecting-ai-clients) for VS Code, JetBrains, Copilot and Cursor.
|
|
26
31
|
|
|
27
32
|
```json
|
|
28
33
|
{
|
|
@@ -40,7 +45,7 @@ Add to your AI client (Claude Desktop shown — see [Connecting AI Clients](#con
|
|
|
40
45
|
}
|
|
41
46
|
```
|
|
42
47
|
|
|
43
|
-
|
|
48
|
+
**4. Restart the client and ask:** *"Show me the overall health of the device farm."*
|
|
44
49
|
|
|
45
50
|
> Prefer a container? See [Option B — Pull from GitHub Container Registry](#option-b--pull-from-github-container-registry).
|
|
46
51
|
|
|
@@ -48,122 +53,51 @@ Then ask: *"Show me the overall health of the device farm."*
|
|
|
48
53
|
|
|
49
54
|
## Table of Contents
|
|
50
55
|
|
|
56
|
+
**Get started**
|
|
51
57
|
- [Quick Start](#quick-start)
|
|
52
|
-
- [
|
|
53
|
-
- [
|
|
54
|
-
- [
|
|
55
|
-
- [Configuration](#configuration)
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
- [
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
- [Claude Code (JetBrains / Android Studio)](#claude-code-jetbrains--android-studio)
|
|
62
|
-
- [GitHub Copilot (VS Code)](#github-copilot-vs-code)
|
|
63
|
-
- [Cursor](#cursor)
|
|
64
|
-
- [Example Prompts](#example-prompts)
|
|
65
|
-
- [Capabilities](#capabilities) — full per-tool reference in [docs/tools.md](docs/tools.md)
|
|
66
|
-
- [Workflow Reference](#workflow-reference)
|
|
67
|
-
- [POC Lifecycle](#poc-lifecycle)
|
|
68
|
-
- [Project Lifecycle](#project-lifecycle)
|
|
69
|
-
- [Boilerplate Generation](#boilerplate-generation)
|
|
70
|
-
- [Recommended agent guardrails](docs/recommended-agent-guardrails.md)
|
|
58
|
+
- [Installation](#installation) — [npm](#option-a--install-via-npm-recommended) · [Docker image](#option-b--pull-from-github-container-registry) · [from source](#option-c--build-from-source) · [Upgrading](#upgrading)
|
|
59
|
+
- [Connecting AI Clients](#connecting-ai-clients) — [Claude Desktop](#claude-desktop) · [Claude Code (VS Code)](#claude-code-vs-code) · [Claude Code (JetBrains)](#claude-code-jetbrains--android-studio) · [GitHub Copilot](#github-copilot-vs-code) · [Cursor](#cursor) · [Docker configs](#docker-client-configuration)
|
|
60
|
+
- [Access Keys](#access-keys) — access levels, finding your key, multiple profiles
|
|
61
|
+
- [Configuration](#configuration) — [Toolsets](#toolsets-reducing-context-size) · [Debug mode](#debug-mode-remediation-notes)
|
|
62
|
+
|
|
63
|
+
**Use it**
|
|
64
|
+
- [What You Can Do](#what-you-can-do) — [example prompts](#example-prompts) · [tasks at a glance](#tasks-at-a-glance) · [all capability areas](#all-capability-areas)
|
|
65
|
+
- [Creating Tests](#creating-tests) — [mobile](#mobile-app-tests-get_test_boilerplate) · [Test Orchestrator](#test-orchestrator-default-for-java-on-appium-server) · [web](#web-tests-get_web_test_boilerplate) · [validating a test](#check-a-test-before-you-deliver-it-validate_test_script)
|
|
66
|
+
- [Test Results and Root Cause Analysis](#test-results-and-root-cause-analysis)
|
|
71
67
|
- [Performance Comparison Reports](#performance-comparison-reports)
|
|
72
|
-
- [Reference](#reference)
|
|
73
|
-
- [Response Format](#response-format)
|
|
74
|
-
- [Understanding maxResults](#understanding-maxresults)
|
|
75
|
-
- [List Filters & Sorting](#list-filters--sorting)
|
|
76
|
-
- [Test Reporting Schema](#test-reporting-schema)
|
|
77
|
-
- [Local File Transfer](#local-file-transfer-docker--remote-deployments-only)
|
|
68
|
+
- [Workflow Reference](#workflow-reference) — [POC lifecycle](#poc-lifecycle) · [Project lifecycle](#project-lifecycle)
|
|
78
69
|
- [Safety Guards](#safety-guards)
|
|
79
|
-
|
|
70
|
+
|
|
71
|
+
**Help and reference**
|
|
80
72
|
- [Troubleshooting](#troubleshooting)
|
|
73
|
+
- [Reference](#reference) — [response format](#response-format) · [maxResults](#understanding-maxresults) · [list filters](#list-filters--sorting) · [local file transfer (Docker)](#local-file-transfer-docker--remote-deployments-only)
|
|
81
74
|
- [Known Limitations](#known-limitations)
|
|
75
|
+
- [Development](#development)
|
|
82
76
|
- [License](#license)
|
|
83
77
|
|
|
84
78
|
---
|
|
85
79
|
|
|
86
|
-
##
|
|
87
|
-
|
|
88
|
-
- **Node.js 22+** — required for [Option A](#option-a--install-via-npm-recommended) (recommended — no Docker needed)
|
|
89
|
-
- **Docker** — required for [Option B](#option-b--pull-from-github-container-registry) or [Option C](#option-c--build-from-source) (container-based install)
|
|
90
|
-
- **Digital.ai Testing account** with a valid access key
|
|
91
|
-
|
|
92
|
-
---
|
|
93
|
-
|
|
94
|
-
## Access Keys
|
|
95
|
-
|
|
96
|
-
Your Digital.ai access key determines what the MCP server can do on your behalf. There are three access levels.
|
|
97
|
-
|
|
98
|
-
| Access level | Access |
|
|
99
|
-
|---|---|
|
|
100
|
-
| **Cloud Admin** | All tools: device management, user provisioning, project administration, infrastructure, performance data |
|
|
101
|
-
| **Project Admin** | Scoped to one project. Can manage device tags, view project admin settings, list users. v2 API tools (agents, regions, license data) return 403. |
|
|
102
|
-
| **Project User** | Scoped to one project. Read/test operations only. Cannot manage tags or access admin settings; deleting reports requires the project's `allowUsersDeleteTests` setting. |
|
|
103
|
-
|
|
104
|
-
> **The key format does not tell you the access level.** Older keys are long `eyJ…` tokens; since Digital.ai 24.11 newly generated keys use a shorter `aut_1_…` token, and that applies to Cloud Admins too. The MCP server therefore detects your access level from the API (`GET /api/v1/users/my-account-info` → `Admin` / `ProjectAdmin` / `User`) and never from the key's prefix. `get_server_info` and `list_environments` show what was detected for each profile.
|
|
105
|
-
|
|
106
|
-
When a Cloud Admin tool is called with a project-level role, the MCP returns a plain-language error explaining what happened — and, if a Cloud Admin profile is known, a ready-to-use `switch_environment(...)` command.
|
|
107
|
-
|
|
108
|
-
### Finding your key
|
|
109
|
-
|
|
110
|
-
1. Log in to the Digital.ai Testing portal
|
|
111
|
-
2. Click your **name or avatar** in the top-right corner
|
|
112
|
-
3. Select **Access Key**
|
|
113
|
-
|
|
114
|
-
The key shown is tied to the **project currently selected in the portal**. If you belong to multiple projects, each has a separate key.
|
|
115
|
-
|
|
116
|
-
### Using multiple keys
|
|
117
|
-
|
|
118
|
-
Configure named profiles in `.env` to switch contexts at runtime without editing files:
|
|
119
|
-
|
|
120
|
-
```
|
|
121
|
-
# Default connection (typically Cloud Admin credentials — either key format works)
|
|
122
|
-
DIGITAL_AI_BASE_URL=https://your-tenant.experitest.com
|
|
123
|
-
DIGITAL_AI_ACCESS_KEY=...your-cloud-admin-key...
|
|
124
|
-
|
|
125
|
-
# Project-scoped profiles
|
|
126
|
-
DAI_PROFILE_QA_URL=https://your-tenant.experitest.com
|
|
127
|
-
DAI_PROFILE_QA_KEY=...your-qa-key...
|
|
128
|
-
|
|
129
|
-
DAI_PROFILE_STAGING_URL=https://your-tenant.experitest.com
|
|
130
|
-
DAI_PROFILE_STAGING_KEY=...your-staging-key...
|
|
131
|
-
```
|
|
132
|
-
|
|
133
|
-
Then ask your AI:
|
|
134
|
-
- *"What environments are configured?"* → `list_environments`
|
|
135
|
-
- *"Switch to QA"* → `switch_environment("qa")`
|
|
136
|
-
- *"Switch back to Cloud Admin"* → `switch_environment("default")`
|
|
137
|
-
|
|
138
|
-
The switch takes effect immediately — no restart needed.
|
|
139
|
-
|
|
140
|
-
> **Security:** Never commit `.env` to source control. Each team member should use their own key obtained from the portal under their own account.
|
|
80
|
+
## Installation
|
|
141
81
|
|
|
142
|
-
|
|
82
|
+
**What you need:**
|
|
143
83
|
|
|
144
|
-
|
|
84
|
+
- A **Digital.ai Testing account** with a valid [access key](#access-keys).
|
|
85
|
+
- **Node.js 22+** for [Option A](#option-a--install-via-npm-recommended) (recommended — no Docker needed), **or**
|
|
86
|
+
- **Docker** for [Option B](#option-b--pull-from-github-container-registry) or [Option C](#option-c--build-from-source) (container-based install).
|
|
145
87
|
|
|
146
88
|
### Option A — Install via npm (Recommended)
|
|
147
89
|
|
|
148
|
-
No Docker required — runs directly under Node.js.
|
|
90
|
+
No Docker required — the server runs directly under Node.js.
|
|
149
91
|
|
|
150
92
|
```bash
|
|
151
93
|
npm install -g digital-ai-testing-mcp
|
|
152
94
|
```
|
|
153
95
|
|
|
154
|
-
This
|
|
96
|
+
This puts a `digital-ai-testing-mcp` command on your `PATH`. Use it as the `"command"` in your [AI client configuration](#connecting-ai-clients). You don't need an image name or a `.env` file: settings go in the client's `env` block (see [Configuration](#configuration)).
|
|
155
97
|
|
|
156
|
-
|
|
98
|
+
To run without a global install, use `"command": "npx"` with `"args": ["-y", "digital-ai-testing-mcp"]` instead — npx downloads and runs the latest published version.
|
|
157
99
|
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
```bash
|
|
161
|
-
npm install -g digital-ai-testing-mcp@latest
|
|
162
|
-
```
|
|
163
|
-
|
|
164
|
-
Then restart your AI client (or reconnect the MCP server from its settings panel) so it respawns the process — npm installing a new version doesn't affect an already-running process. Confirm the upgrade worked by asking your AI to call `get_server_info`, which reports the running version.
|
|
165
|
-
|
|
166
|
-
---
|
|
100
|
+
Set `MCP_DEPLOYMENT_MODE=local` alongside your credentials (the client examples below already include it). It tells the server that its filesystem is your own machine, so tool descriptions and a few upload/download behaviors drop the Docker-specific caveats and path warnings.
|
|
167
101
|
|
|
168
102
|
### Option B — Pull from GitHub Container Registry
|
|
169
103
|
|
|
@@ -183,23 +117,15 @@ cp .env.example .env
|
|
|
183
117
|
# Edit .env — set DIGITAL_AI_BASE_URL and DIGITAL_AI_ACCESS_KEY
|
|
184
118
|
```
|
|
185
119
|
|
|
186
|
-
|
|
120
|
+
**Step 3 — Connect your client** using `ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest` as the image name — see [Docker client configuration](#docker-client-configuration).
|
|
187
121
|
|
|
188
|
-
> Images are published when a GitHub Release is created — not on every commit. If `latest` is not available yet, use [Option C](#option-c--build-from-source) to build from source.
|
|
122
|
+
> Images are published when a GitHub Release is created — not on every commit. Each release is also tagged with its version (e.g. `2.0.3` and `2.0`) if you want to pin one. If `latest` is not available yet, use [Option C](#option-c--build-from-source) to build from source.
|
|
189
123
|
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
```bash
|
|
193
|
-
docker pull ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest
|
|
194
|
-
```
|
|
195
|
-
|
|
196
|
-
Since the server is launched fresh each time via `docker run --rm` (see the client configs below), there's no running container to stop or remove first — the next time your AI client starts the server, it uses the freshly pulled image automatically. Confirm with `get_server_info`.
|
|
197
|
-
|
|
198
|
-
---
|
|
124
|
+
With Docker, tools that read or write local files work inside the container, not on your machine — see [Local File Transfer](#local-file-transfer-docker--remote-deployments-only).
|
|
199
125
|
|
|
200
126
|
### Option C — Build from source
|
|
201
127
|
|
|
202
|
-
|
|
128
|
+
Needed only for local development or to modify the server.
|
|
203
129
|
|
|
204
130
|
```bash
|
|
205
131
|
git clone https://github.com/digitalai-opensource/digital-ai-testing-mcp
|
|
@@ -209,106 +135,25 @@ cp .env.example .env
|
|
|
209
135
|
docker build -t digital-ai-testing-mcp:latest .
|
|
210
136
|
```
|
|
211
137
|
|
|
212
|
-
Use `digital-ai-testing-mcp:latest` as the image name in your
|
|
213
|
-
|
|
214
|
-
**Upgrading to the latest release:**
|
|
215
|
-
|
|
216
|
-
```bash
|
|
217
|
-
git pull
|
|
218
|
-
docker build -t digital-ai-testing-mcp:latest .
|
|
219
|
-
```
|
|
220
|
-
|
|
221
|
-
---
|
|
222
|
-
|
|
223
|
-
## Configuration
|
|
224
|
-
|
|
225
|
-
| Variable | Required | Default | Description |
|
|
226
|
-
|---|---|---|---|
|
|
227
|
-
| `DIGITAL_AI_BASE_URL` | ✅ | — | Your Digital.ai tenant URL |
|
|
228
|
-
| `DIGITAL_AI_ACCESS_KEY` | ✅ | — | Access key — see [Access Keys](#access-keys) for the three access levels and key formats |
|
|
229
|
-
| `MCP_SERVER_NAME` | Optional | `digital-ai-testing-mcp` | Server identity shown in the AI client |
|
|
230
|
-
| `MCP_SERVER_VERSION` | Optional | version from `package.json` | Override the reported server version (rarely needed) |
|
|
231
|
-
| `MCP_DEPLOYMENT_MODE` | Optional | `docker` | `docker` (default) or `local` — set to `local` when running via the npm package ([Option A](#option-a--install-via-npm-recommended)) so tool descriptions and upload/download behavior reflect that the server's filesystem is your own machine, not a container's |
|
|
232
|
-
| `REQUEST_TIMEOUT_MS` | Optional | `30000` | API request timeout in milliseconds |
|
|
233
|
-
| `UPLOAD_TIMEOUT_MS` | Optional | `120000` | File upload timeout in milliseconds |
|
|
234
|
-
| `MCP_TOOLSETS` | Optional | all | Comma-separated toolsets to load in full: `devices`, `apps`, `reporting`, `performance`, `inspection`, `authoring`, `browsers`, `admin` (or `all`, or `core` for only the always-loaded core). See [Toolsets](#toolsets-reducing-context-size) |
|
|
235
|
-
| `MCP_DEBUG_MODE` | Optional | `false` | `true` records remediation notes and a tool-call event log to help improve this MCP. See [Debug mode](#debug-mode-remediation-notes) |
|
|
236
|
-
| `MCP_REMEDIATION_DIR` | Optional | — | Debug mode only: folder to write remediation files to (overrides project detection) |
|
|
237
|
-
| `AXE_DEVTOOLS_API_KEY` | Optional | — | Deque Axe DevTools Mobile key used by `includeAxeScan` boilerplate |
|
|
238
|
-
| `TEST_ORCHESTRATOR_JAR_URL` / `TEST_ORCHESTRATOR_JAR_SHA256` | Optional | pinned | Override the Test Orchestrator agent download location; both are required |
|
|
239
|
-
| `ACCESS_PROBE_TIMEOUT_MS` | Optional | `8000` | Timeout for the access-level detection call |
|
|
240
|
-
|
|
241
|
-
Additional `DAI_PROFILE_{NAME}_URL` / `DAI_PROFILE_{NAME}_KEY` pairs configure named profiles for multi-project or multi-environment use. See [Access Keys](#access-keys) and `.env.example` for examples.
|
|
242
|
-
|
|
243
|
-
### Toolsets (reducing context size)
|
|
244
|
-
|
|
245
|
-
All 202 tool definitions add up to roughly 68K tokens of context. If your AI client is short on context — or runs several MCP servers — set `MCP_TOOLSETS` to the areas you use, e.g. `MCP_TOOLSETS=reporting` (about 25K tokens) or `MCP_TOOLSETS=inspection,authoring` (about 32K).
|
|
246
|
-
|
|
247
|
-
Nothing becomes unavailable. Tools outside the chosen toolsets are still listed, with a one-line description. When the AI calls one, the server loads that whole toolset and replies with the tool's full description and parameter guidance, then the AI calls it again — so the detailed usage guidance is always seen before the tool runs. The AI can also load toolsets up front with `enable_toolset`. Leave `MCP_TOOLSETS` unset (the default) to load everything in full, exactly as before.
|
|
248
|
-
|
|
249
|
-
| Toolset | Covers |
|
|
250
|
-
|---|---|
|
|
251
|
-
| `devices` | Devices, device groups, reservations, health, agents, regions |
|
|
252
|
-
| `apps` | Application repository, file repository, provisioning profiles |
|
|
253
|
-
| `reporting` | Test reports, analytics, root-cause analysis, test views, coverage |
|
|
254
|
-
| `performance` | Performance transactions and comparisons, NV servers |
|
|
255
|
-
| `inspection` | Live device and browser inspection, Android Auto / CarPlay, remote debug |
|
|
256
|
-
| `authoring` | Test boilerplate, Espresso / XCUITest / Maestro test runs |
|
|
257
|
-
| `browsers` | Browser listing and Selenium sessions |
|
|
258
|
-
| `admin` | Users, projects, backups, usage reports, POC and project workflows |
|
|
259
|
-
|
|
260
|
-
Always loaded in full: environment and connection tools (`get_server_info`, `switch_environment`, …), `enable_toolset`, and `validate_test_script`.
|
|
261
|
-
|
|
262
|
-
### Debug mode (remediation notes)
|
|
263
|
-
|
|
264
|
-
Set `MCP_DEBUG_MODE=true` while evaluating or developing the MCP to collect feedback for improving it. With the default `false`, nothing changes.
|
|
265
|
-
|
|
266
|
-
When it is on:
|
|
267
|
-
|
|
268
|
-
- **The AI records notes.** The server asks the AI client to call `record_remediation_note` (a tool that exists only in debug mode) whenever it:
|
|
269
|
-
- hits an unexpected error;
|
|
270
|
-
- wastes calls because guidance was unclear;
|
|
271
|
-
- is corrected or stopped by you;
|
|
272
|
-
- gives up;
|
|
273
|
-
- finds a better path than the one recommended;
|
|
274
|
-
- spots a way to make things cleaner or cheaper.
|
|
275
|
-
|
|
276
|
-
It records each note once the situation is resolved or abandoned, not in the middle of your task. Expected outcomes, such as deliberate negative tests or a safety check that worked as intended, are not recorded: they show nothing to improve.
|
|
277
|
-
- **The server logs what it sees.** Every tool call is logged with its duration, response size and outcome (ok, error or guard), with flags for retries and repeated calls. This is objective data the AI can't under-report.
|
|
278
|
-
- **The AI is told up front.** The debug-mode instructions come first in what the server sends at connect time, and `get_server_info` and `check_connectivity` show "Debug mode: ON".
|
|
279
|
-
- **Errors get a reminder.** The first time a tool returns a given error or guard, the response starts with a one-line reminder to record a note if it was unexpected (in JSON responses, as a `_debugMode` field).
|
|
138
|
+
Use `digital-ai-testing-mcp:latest` as the image name in your [Docker client configuration](#docker-client-configuration).
|
|
280
139
|
|
|
281
|
-
|
|
140
|
+
### Upgrading
|
|
282
141
|
|
|
283
|
-
| Install |
|
|
142
|
+
| Install | Upgrade command | Then |
|
|
284
143
|
|---|---|---|
|
|
285
|
-
| npm (
|
|
286
|
-
|
|
|
144
|
+
| npm (Option A) | `npm install -g digital-ai-testing-mcp@latest` | Restart your AI client, or reconnect the MCP server from its settings panel. Installing a new version doesn't affect a server that is already running. |
|
|
145
|
+
| GHCR image (Option B) | `docker pull ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest` | Nothing to stop or remove: the client configs start the server with `docker run --rm`, so the next start uses the new image. |
|
|
146
|
+
| From source (Option C) | `git pull` then `docker build -t digital-ai-testing-mcp:latest .` | Same as Option B. |
|
|
287
147
|
|
|
288
|
-
|
|
289
|
-
1. `MCP_REMEDIATION_DIR`, if set (files go directly into that folder);
|
|
290
|
-
2. the workspace folder the AI client reports;
|
|
291
|
-
3. the folder the server was started in, if it looks like a project;
|
|
292
|
-
4. `~/remediation`.
|
|
293
|
-
|
|
294
|
-
The `remediation/` folder contains its own `.gitignore` (`*`), so it is never committed to the project it lives in. There is one file per session; the `<id>` suffix keeps parallel sessions apart, and each file records the machine and user, so files collected from several machines can be merged. Keys, tokens, emails and signed-URL parameters are redacted before anything is written or returned.
|
|
148
|
+
To confirm the upgrade, ask your AI to call `get_server_info`, which reports the running version.
|
|
295
149
|
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
To process a batch, collect the `remediation/` folders you want to analyse, from one machine or several, and run `npm run remediation:digest -- --dir <folder> --dir <folder>` in a clone of this repository (the npm package doesn't include it). With no `--dir`, it reads `$MCP_REMEDIATION_DIR`, or else `./remediation` and `~/remediation`. It condenses every unprocessed file into one summary, so the development agent reads a few KB instead of every raw note:
|
|
299
|
-
- notes by category, and issues that recur across sessions;
|
|
300
|
-
- per-tool errors, guards, retries, response sizes and latency;
|
|
301
|
-
- every note in a table.
|
|
302
|
-
|
|
303
|
-
Useful options:
|
|
304
|
-
- `--out <file>` writes the summary to a file; `--since YYYY-MM-DD` limits it to recent sessions; `--json` gives machine-readable output.
|
|
305
|
-
- `--archive <session…>` or `--archive-all` moves handled files into `processed/<date>/` inside the folder they came from. Nothing is deleted.
|
|
150
|
+
[↑ Back to top](#table-of-contents)
|
|
306
151
|
|
|
307
152
|
---
|
|
308
153
|
|
|
309
154
|
## Connecting AI Clients
|
|
310
155
|
|
|
311
|
-
The examples below
|
|
156
|
+
The examples below use the npm install ([Option A](#option-a--install-via-npm-recommended)): credentials go directly in the client's `env` block. If you run the server in Docker (Options B/C), use the [Docker client configuration](#docker-client-configuration) instead.
|
|
312
157
|
|
|
313
158
|
### Claude Desktop
|
|
314
159
|
|
|
@@ -332,30 +177,15 @@ Find your config file:
|
|
|
332
177
|
}
|
|
333
178
|
```
|
|
334
179
|
|
|
335
|
-
> **Using Docker instead (Option B/C)?**
|
|
336
|
-
> ```json
|
|
337
|
-
> {
|
|
338
|
-
> "mcpServers": {
|
|
339
|
-
> "digital-ai-testing": {
|
|
340
|
-
> "command": "docker",
|
|
341
|
-
> "args": [
|
|
342
|
-
> "run", "--rm", "-i",
|
|
343
|
-
> "--env-file", "/ABSOLUTE/PATH/TO/.env",
|
|
344
|
-
> "ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest"
|
|
345
|
-
> ]
|
|
346
|
-
> }
|
|
347
|
-
> }
|
|
348
|
-
> }
|
|
349
|
-
> ```
|
|
350
|
-
> **Built from source (Option C)?** Replace `ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest` with `digital-ai-testing-mcp:latest`.
|
|
351
|
-
|
|
352
180
|
Restart Claude Desktop after editing — the tools appear automatically.
|
|
353
181
|
|
|
354
182
|
### Claude Code (VS Code)
|
|
355
183
|
|
|
356
184
|
1. Open the Claude Code extension panel
|
|
357
185
|
2. Go to **Settings → MCP Servers**
|
|
358
|
-
3. Add a new server with command `digital-ai-testing-mcp`, no args, and `DIGITAL_AI_BASE_URL`/`DIGITAL_AI_ACCESS_KEY`/`MCP_DEPLOYMENT_MODE=local` set in the server's environment variables
|
|
186
|
+
3. Add a new server with command `digital-ai-testing-mcp`, no args, and `DIGITAL_AI_BASE_URL`/`DIGITAL_AI_ACCESS_KEY`/`MCP_DEPLOYMENT_MODE=local` set in the server's environment variables
|
|
187
|
+
|
|
188
|
+
Using Docker? Add the Docker command from [Docker client configuration](#docker-client-configuration) instead.
|
|
359
189
|
|
|
360
190
|
### Claude Code (JetBrains / Android Studio)
|
|
361
191
|
|
|
@@ -365,46 +195,18 @@ Install the [Claude Code](https://plugins.jetbrains.com/plugin/22828-claude-code
|
|
|
365
195
|
claude mcp add digital-ai-testing --env DIGITAL_AI_BASE_URL=https://your-tenant.experitest.com --env DIGITAL_AI_ACCESS_KEY=your-access-key --env MCP_DEPLOYMENT_MODE=local -- digital-ai-testing-mcp
|
|
366
196
|
```
|
|
367
197
|
|
|
368
|
-
This stores the server configuration in `~/.claude.json
|
|
369
|
-
|
|
370
|
-
> **Using Docker instead (Option B/C)?**
|
|
371
|
-
> ```bash
|
|
372
|
-
> # macOS / Linux
|
|
373
|
-
> claude mcp add digital-ai-testing -- docker run --rm -i --env-file /absolute/path/to/.env ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest
|
|
374
|
-
>
|
|
375
|
-
> # Windows (use forward slashes — backslashes are stripped by Claude Code)
|
|
376
|
-
> claude mcp add digital-ai-testing -- docker run --rm -i --env-file C:/projects/digital-ai-testing-mcp/.env ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest
|
|
377
|
-
> ```
|
|
378
|
-
> **Windows users on Docker:** Always use forward slashes (`C:/path/to/.env`), never backslashes — regardless of whether you register via the CLI command above or through the Claude Code panel (**Settings → MCP Servers**). Backslashes are silently stripped when Claude Code writes the configuration to `~/.claude.json`, resulting in a broken path and a cryptic `-32000` reconnection error.
|
|
379
|
-
> **Built from source (Option C)?** Replace the GHCR image name with `digital-ai-testing-mcp:latest`.
|
|
198
|
+
This stores the server configuration in `~/.claude.json`, scoped to the current project. Alternatively, use the Claude Code panel: **Settings → MCP Servers** and add the same command used for [Claude Desktop](#claude-desktop).
|
|
380
199
|
|
|
381
200
|
Restart the Claude Code panel after adding the server — the tools appear automatically.
|
|
382
201
|
|
|
202
|
+
Using Docker? See [Docker client configuration](#docker-client-configuration) — and on Windows, use forward slashes in the `.env` path.
|
|
203
|
+
|
|
383
204
|
### GitHub Copilot (VS Code)
|
|
384
205
|
|
|
385
206
|
GitHub Copilot supports MCP tools in **Agent mode** only. Register the server in one of two scopes:
|
|
386
207
|
|
|
387
|
-
**User settings** — available in all workspaces. Open
|
|
388
|
-
|
|
389
|
-
```json
|
|
390
|
-
{
|
|
391
|
-
"mcp": {
|
|
392
|
-
"servers": {
|
|
393
|
-
"digital-ai-testing": {
|
|
394
|
-
"command": "digital-ai-testing-mcp",
|
|
395
|
-
"args": [],
|
|
396
|
-
"env": {
|
|
397
|
-
"DIGITAL_AI_BASE_URL": "https://your-tenant.experitest.com",
|
|
398
|
-
"DIGITAL_AI_ACCESS_KEY": "your-access-key",
|
|
399
|
-
"MCP_DEPLOYMENT_MODE": "local"
|
|
400
|
-
}
|
|
401
|
-
}
|
|
402
|
-
}
|
|
403
|
-
}
|
|
404
|
-
}
|
|
405
|
-
```
|
|
406
|
-
|
|
407
|
-
**Workspace settings** — scoped to one project. Create `.vscode/mcp.json`:
|
|
208
|
+
- **User settings** — available in all workspaces. Open the user MCP configuration (`Ctrl+Shift+P` → "MCP: Open User Configuration") and add the same `servers` entry shown for the workspace file below. (Older VS Code versions used an `"mcp": { "servers": { … } }` block in the user `settings.json` instead.)
|
|
209
|
+
- **Workspace settings** — scoped to one project. Create `.vscode/mcp.json`:
|
|
408
210
|
|
|
409
211
|
```json
|
|
410
212
|
{
|
|
@@ -424,22 +226,12 @@ GitHub Copilot supports MCP tools in **Agent mode** only. Register the server in
|
|
|
424
226
|
|
|
425
227
|
Committing `.vscode/mcp.json` to source control shares the server configuration with the entire team automatically.
|
|
426
228
|
|
|
427
|
-
> **Using Docker instead (Option B/C)?** in either settings file above, replace the `"digital-ai-testing"` entry's `command`/`args`/`env` with:
|
|
428
|
-
> ```json
|
|
429
|
-
> {
|
|
430
|
-
> "command": "docker",
|
|
431
|
-
> "args": [
|
|
432
|
-
> "run", "--rm", "-i",
|
|
433
|
-
> "--env-file", "/ABSOLUTE/PATH/TO/.env",
|
|
434
|
-
> "ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest"
|
|
435
|
-
> ]
|
|
436
|
-
> }
|
|
437
|
-
> ```
|
|
438
|
-
|
|
439
229
|
To use the tools: open Copilot Chat (`Ctrl+Alt+I`), switch the mode dropdown to **Agent**, and type your request.
|
|
440
230
|
|
|
441
231
|
> GitHub Copilot on the web (`github.com/copilot`) does not support external MCP servers. VS Code is required.
|
|
442
232
|
|
|
233
|
+
Using Docker? See [Docker client configuration](#docker-client-configuration).
|
|
234
|
+
|
|
443
235
|
### Cursor
|
|
444
236
|
|
|
445
237
|
Cursor supports MCP in **Agent mode**. Add the server in **Cursor Settings → MCP** (or `Cursor Settings → Features → MCP`):
|
|
@@ -462,28 +254,213 @@ Cursor supports MCP in **Agent mode**. Add the server in **Cursor Settings → M
|
|
|
462
254
|
|
|
463
255
|
Alternatively, create a `.cursor/mcp.json` file in your project root with the same `mcpServers` object — this scopes the server to that workspace and can be committed to share it with your team.
|
|
464
256
|
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
257
|
+
Cursor is available on macOS, Windows, and Linux. It is the recommended option for iOS developers on macOS, where Xcode is the primary IDE but does not natively support MCP.
|
|
258
|
+
|
|
259
|
+
Using Docker? See [Docker client configuration](#docker-client-configuration).
|
|
260
|
+
|
|
261
|
+
### Docker client configuration
|
|
262
|
+
|
|
263
|
+
For [Option B](#option-b--pull-from-github-container-registry) or [Option C](#option-c--build-from-source), the client starts the container instead of the `digital-ai-testing-mcp` command, and credentials come from your `.env` file. In every example:
|
|
264
|
+
|
|
265
|
+
- Replace `/ABSOLUTE/PATH/TO/.env` with the full path to your `.env` file.
|
|
266
|
+
- **Built from source (Option C)?** Replace `ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest` with `digital-ai-testing-mcp:latest`.
|
|
267
|
+
|
|
268
|
+
**Claude Desktop and Cursor** (`mcpServers` format):
|
|
269
|
+
|
|
270
|
+
```json
|
|
271
|
+
{
|
|
272
|
+
"mcpServers": {
|
|
273
|
+
"digital-ai-testing": {
|
|
274
|
+
"command": "docker",
|
|
275
|
+
"args": [
|
|
276
|
+
"run", "--rm", "-i",
|
|
277
|
+
"--env-file", "/ABSOLUTE/PATH/TO/.env",
|
|
278
|
+
"ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest"
|
|
279
|
+
]
|
|
280
|
+
}
|
|
281
|
+
}
|
|
282
|
+
}
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
**GitHub Copilot** — in either configuration file, replace the `"digital-ai-testing"` entry's `command`/`args`/`env` with:
|
|
286
|
+
|
|
287
|
+
```json
|
|
288
|
+
{
|
|
289
|
+
"command": "docker",
|
|
290
|
+
"args": [
|
|
291
|
+
"run", "--rm", "-i",
|
|
292
|
+
"--env-file", "/ABSOLUTE/PATH/TO/.env",
|
|
293
|
+
"ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest"
|
|
294
|
+
]
|
|
295
|
+
}
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
**Claude Code** (JetBrains / Android Studio, or VS Code) — from the project root, or add the same command under **Settings → MCP Servers**:
|
|
299
|
+
|
|
300
|
+
```bash
|
|
301
|
+
# macOS / Linux
|
|
302
|
+
claude mcp add digital-ai-testing -- docker run --rm -i --env-file /absolute/path/to/.env ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest
|
|
303
|
+
|
|
304
|
+
# Windows (use forward slashes — backslashes are stripped by Claude Code)
|
|
305
|
+
claude mcp add digital-ai-testing -- docker run --rm -i --env-file C:/projects/digital-ai-testing-mcp/.env ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
> **Windows users on Docker:** always use forward slashes (`C:/path/to/.env`), never backslashes — whether you register via the CLI command above or through the Claude Code panel. Claude Code silently strips backslashes when it writes the configuration to `~/.claude.json`, which breaks the path and causes a cryptic `-32000` reconnection error ([how to fix it](#failed-to-reconnect-to-digital-ai-testing--32000)).
|
|
309
|
+
|
|
310
|
+
[↑ Back to top](#table-of-contents)
|
|
311
|
+
|
|
312
|
+
---
|
|
313
|
+
|
|
314
|
+
## Access Keys
|
|
315
|
+
|
|
316
|
+
Your Digital.ai access key decides what the MCP server can do on your behalf. There are three access levels:
|
|
317
|
+
|
|
318
|
+
| Access level | What it can do |
|
|
319
|
+
|---|---|
|
|
320
|
+
| **Cloud Admin** | All tools: device management, user provisioning, project administration, infrastructure, performance data |
|
|
321
|
+
| **Project Admin** | Scoped to one project. Everything a Project User can do, plus: manage device tags, create reservations, set project notes, view project admin settings, list its own project's users. v2 API tools (agents, regions, license data, usage reports) return 403. |
|
|
322
|
+
| **Project User** | Scoped to one project. Day-to-day testing: devices, app upload/install, reserving devices for itself, test runs, reports, test views, performance data, and inspection sessions. Cannot manage tags or access admin settings; deleting reports requires the project's `allowUsersDeleteTests` setting. |
|
|
323
|
+
|
|
324
|
+
> **You can't tell the access level from the key's format.** Older keys are long `eyJ…` tokens; since Digital.ai 24.11, newly generated keys use a shorter `aut_1_…` token — for Cloud Admins too. So the server asks the API for your role (`GET /api/v1/users/my-account-info` → `Admin` / `ProjectAdmin` / `User`) and never guesses from the key's prefix. `get_server_info` shows the level detected for the active profile, and `list_environments` shows it for every profile. Per-tool role requirements are in [docs/tools.md](docs/tools.md).
|
|
325
|
+
|
|
326
|
+
If you call a Cloud Admin tool with a project-level role, the server returns a plain-language explanation — and, if a Cloud Admin profile is configured, a ready-to-use `switch_environment(...)` command.
|
|
327
|
+
|
|
328
|
+
### Finding your key
|
|
329
|
+
|
|
330
|
+
1. Log in to the Digital.ai Testing portal
|
|
331
|
+
2. Click your **name or avatar** in the top-right corner
|
|
332
|
+
3. Select **Access Key**
|
|
333
|
+
|
|
334
|
+
The key shown is tied to the **project currently selected in the portal**. If you belong to multiple projects, each has a separate key.
|
|
335
|
+
|
|
336
|
+
### Using multiple keys
|
|
337
|
+
|
|
338
|
+
Set up named profiles to switch between keys (for example, Cloud Admin and a project) without editing files. With the npm install, add the variables to the `env` block of your AI client's configuration; with Docker, add them to your `.env` file:
|
|
339
|
+
|
|
340
|
+
```
|
|
341
|
+
# Default connection (typically Cloud Admin credentials — either key format works)
|
|
342
|
+
DIGITAL_AI_BASE_URL=https://your-tenant.experitest.com
|
|
343
|
+
DIGITAL_AI_ACCESS_KEY=...your-cloud-admin-key...
|
|
344
|
+
|
|
345
|
+
# Project-scoped profiles
|
|
346
|
+
DAI_PROFILE_QA_URL=https://your-tenant.experitest.com
|
|
347
|
+
DAI_PROFILE_QA_KEY=...your-qa-key...
|
|
348
|
+
|
|
349
|
+
DAI_PROFILE_STAGING_URL=https://your-tenant.experitest.com
|
|
350
|
+
DAI_PROFILE_STAGING_KEY=...your-staging-key...
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
Then ask your AI:
|
|
354
|
+
- *"What environments are configured?"* → `list_environments`
|
|
355
|
+
- *"Switch to QA"* → `switch_environment("qa")`
|
|
356
|
+
- *"Switch back to Cloud Admin"* → `switch_environment("default")`
|
|
357
|
+
|
|
358
|
+
The switch takes effect immediately — no restart needed.
|
|
359
|
+
|
|
360
|
+
> **Security:** Never commit `.env` to source control. Each team member should use their own key, obtained from the portal under their own account.
|
|
361
|
+
|
|
362
|
+
[↑ Back to top](#table-of-contents)
|
|
363
|
+
|
|
364
|
+
---
|
|
365
|
+
|
|
366
|
+
## Configuration
|
|
367
|
+
|
|
368
|
+
Set these as environment variables: in the client's `env` block (npm install) or in your `.env` file (Docker).
|
|
369
|
+
|
|
370
|
+
| Variable | Required | Default | Description |
|
|
371
|
+
|---|---|---|---|
|
|
372
|
+
| `DIGITAL_AI_BASE_URL` | ✅ | — | Your Digital.ai tenant URL |
|
|
373
|
+
| `DIGITAL_AI_ACCESS_KEY` | ✅ | — | Access key — see [Access Keys](#access-keys) for the three access levels and key formats |
|
|
374
|
+
| `MCP_DEPLOYMENT_MODE` | Optional | `docker` | `docker` (default) or `local`. Set `local` when running via the npm package ([Option A](#option-a--install-via-npm-recommended)) so tool descriptions and upload/download behavior reflect that the server's filesystem is your own machine, not a container's |
|
|
375
|
+
| `MCP_TOOLSETS` | Optional | all | Comma-separated toolsets to load in full: `devices`, `apps`, `reporting`, `performance`, `inspection`, `authoring`, `browsers`, `admin` (or `all`, or `core` for only the always-loaded core). See [Toolsets](#toolsets-reducing-context-size) |
|
|
376
|
+
| `MCP_DEBUG_MODE` | Optional | `false` | `true` records remediation notes and a tool-call event log to help improve this MCP. See [Debug mode](#debug-mode-remediation-notes) |
|
|
377
|
+
| `MCP_REMEDIATION_DIR` | Optional | — | Debug mode only: folder to write remediation files to (overrides project detection) |
|
|
378
|
+
| `REQUEST_TIMEOUT_MS` | Optional | `30000` | API request timeout in milliseconds |
|
|
379
|
+
| `UPLOAD_TIMEOUT_MS` | Optional | `120000` | File upload timeout in milliseconds |
|
|
380
|
+
| `ACCESS_PROBE_TIMEOUT_MS` | Optional | `8000` | Timeout for the access-level detection call |
|
|
381
|
+
| `AXE_DEVTOOLS_API_KEY` | Optional | — | Deque Axe DevTools Mobile key used by `includeAxeScan` boilerplate |
|
|
382
|
+
| `TEST_ORCHESTRATOR_JAR_URL` / `TEST_ORCHESTRATOR_JAR_SHA256` | Optional | pinned | Override the Test Orchestrator agent download location; both are required |
|
|
383
|
+
| `MCP_SERVER_NAME` | Optional | `digital-ai-testing-mcp` | Server identity shown in the AI client |
|
|
384
|
+
| `MCP_SERVER_VERSION` | Optional | version from `package.json` | Override the reported server version (rarely needed) |
|
|
385
|
+
|
|
386
|
+
Additional `DAI_PROFILE_{NAME}_URL` / `DAI_PROFILE_{NAME}_KEY` pairs configure named profiles for multi-project or multi-environment use. See [Using multiple keys](#using-multiple-keys) and `.env.example` for examples.
|
|
387
|
+
|
|
388
|
+
### Toolsets (reducing context size)
|
|
389
|
+
|
|
390
|
+
All 202 tool definitions together take up roughly 70K tokens of your AI's context. If your AI client is short on context — or runs several MCP servers — set `MCP_TOOLSETS` to the areas you use, e.g. `MCP_TOOLSETS=reporting` (about 26K tokens), `MCP_TOOLSETS=inspection,authoring` (about 34K), or `MCP_TOOLSETS=core` (about 17K — only the always-loaded core in full).
|
|
391
|
+
|
|
392
|
+
Nothing becomes unavailable. Tools outside the chosen toolsets are still listed, with a one-line description. When the AI calls one, the server loads that whole toolset and replies with the tool's full description and parameter guidance, then the AI calls it again — so it always sees the detailed usage guidance before the tool runs. The AI can also load toolsets up front with `enable_toolset`. Leave `MCP_TOOLSETS` unset (the default) to load everything in full.
|
|
393
|
+
|
|
394
|
+
| Toolset | Covers |
|
|
395
|
+
|---|---|
|
|
396
|
+
| `devices` | Devices, device groups, reservations, health, agents, regions |
|
|
397
|
+
| `apps` | Application repository, file repository, provisioning profiles |
|
|
398
|
+
| `reporting` | Test reports, analytics, root-cause analysis, test views, coverage |
|
|
399
|
+
| `performance` | Performance transactions and comparisons, NV servers |
|
|
400
|
+
| `inspection` | Live device and browser inspection, Android Auto / CarPlay, remote debug |
|
|
401
|
+
| `authoring` | Test boilerplate, Espresso / XCUITest / Maestro test runs |
|
|
402
|
+
| `browsers` | Browser listing and Selenium sessions |
|
|
403
|
+
| `admin` | Users, projects, backups, usage reports, POC and project workflows |
|
|
404
|
+
|
|
405
|
+
Always loaded in full: environment and connection tools (`get_server_info`, `switch_environment`, …), `enable_toolset`, and `validate_test_script`.
|
|
406
|
+
|
|
407
|
+
### Debug mode (remediation notes)
|
|
408
|
+
|
|
409
|
+
Set `MCP_DEBUG_MODE=true` while evaluating or developing the MCP to collect feedback for improving it. With the default `false`, nothing changes.
|
|
410
|
+
|
|
411
|
+
When it is on:
|
|
412
|
+
|
|
413
|
+
- **The AI records notes.** The server asks the AI client to call `record_remediation_note` (a tool that exists only in debug mode) whenever it:
|
|
414
|
+
- hits an unexpected error;
|
|
415
|
+
- wastes calls because guidance was unclear;
|
|
416
|
+
- is corrected or stopped by you;
|
|
417
|
+
- gives up;
|
|
418
|
+
- finds a better path than the one recommended;
|
|
419
|
+
- spots a way to make things cleaner or cheaper.
|
|
420
|
+
|
|
421
|
+
It records each note once the situation is resolved or abandoned, not in the middle of your task. Expected outcomes, such as deliberate negative tests or a safety check that worked as intended, are not recorded: they show nothing to improve.
|
|
422
|
+
- **The server logs what it sees.** Every tool call is logged with its duration, response size and outcome (ok, error or guard), with flags for retries and repeated calls. This is objective data the AI can't under-report.
|
|
423
|
+
- **The AI is told up front.** The debug-mode instructions come first in what the server sends at connect time, and `get_server_info` and `check_connectivity` show "Debug mode: ON".
|
|
424
|
+
- **Errors get a reminder.** The first time a tool returns a given error or guard, the response starts with a one-line reminder to record a note if it was unexpected (in JSON responses, as a `_debugMode` field).
|
|
425
|
+
|
|
426
|
+
Debug mode costs extra tokens (the instructions and the notes). Leave it off for normal use, and don't use debug sessions as token baselines.
|
|
427
|
+
|
|
428
|
+
**Where the files go:** a `remediation/` folder in the root of **the project the AI is working in**, so each note sits next to the app and tests it's about.
|
|
429
|
+
|
|
430
|
+
| Install | Notes | Event log |
|
|
431
|
+
|---|---|---|
|
|
432
|
+
| npm (`MCP_DEPLOYMENT_MODE=local`) | The server writes `<project>/remediation/<yyyymmddhhmmss>-<id>.md` | The server writes `<project>/remediation/<yyyymmddhhmmss>-<id>.events.jsonl` |
|
|
433
|
+
| Docker | Returned to the AI, which saves them to `remediation/` in its project if it can write files (Claude Code can; Claude Desktop needs a filesystem tool) | Kept in server memory; a summary is included in every note |
|
|
434
|
+
|
|
435
|
+
With the npm install, the server finds the project in this order, and records the step it used as `location:` in each file:
|
|
436
|
+
1. `MCP_REMEDIATION_DIR`, if set (files go directly into that folder);
|
|
437
|
+
2. the workspace folder the AI client reports;
|
|
438
|
+
3. the folder the server was started in, if it looks like a project;
|
|
439
|
+
4. `~/remediation`.
|
|
440
|
+
|
|
441
|
+
The `remediation/` folder contains its own `.gitignore` (`*`), so it is never committed to the project it lives in. There is one file per session; the `<id>` suffix keeps parallel sessions apart, and each file records the machine and user, so files collected from several machines can be merged. Keys, tokens, emails and signed-URL parameters are redacted before anything is written or returned.
|
|
442
|
+
|
|
443
|
+
<details>
|
|
444
|
+
<summary><strong>Summarizing a batch of notes</strong></summary>
|
|
445
|
+
|
|
446
|
+
Collect the `remediation/` folders you want to analyse, from one machine or several, and run `npm run remediation:digest -- --dir <folder> --dir <folder>` in a clone of this repository (the npm package doesn't include it). With no `--dir`, it reads `$MCP_REMEDIATION_DIR`, or else `./remediation` and `~/remediation`. It condenses every unprocessed file into one summary, so the development agent reads a few KB instead of every raw note:
|
|
447
|
+
- notes by category, and issues that recur across sessions;
|
|
448
|
+
- per-tool errors, guards, retries, response sizes and latency;
|
|
449
|
+
- every note in a table.
|
|
450
|
+
|
|
451
|
+
Useful options:
|
|
452
|
+
- `--out <file>` writes the summary to a file; `--since YYYY-MM-DD` limits it to recent sessions; `--json` gives machine-readable output.
|
|
453
|
+
- `--archive <session…>` or `--archive-all` moves handled files into `processed/<date>/` inside the folder they came from. Nothing is deleted.
|
|
454
|
+
|
|
455
|
+
</details>
|
|
456
|
+
|
|
457
|
+
[↑ Back to top](#table-of-contents)
|
|
483
458
|
|
|
484
459
|
---
|
|
485
460
|
|
|
486
|
-
##
|
|
461
|
+
## What You Can Do
|
|
462
|
+
|
|
463
|
+
### Example prompts
|
|
487
464
|
|
|
488
465
|
Once connected, talk to the server in plain language — no tool names needed:
|
|
489
466
|
|
|
@@ -498,13 +475,45 @@ Once connected, talk to the server in plain language — no tool names needed:
|
|
|
498
475
|
|
|
499
476
|
**[→ docs/examples.md](docs/examples.md)** has 200+ prompts organized by scenario: farm health, device groups, regression sign-off, test analytics, performance, usage reports, interactive inspection, POC lifecycle, and more.
|
|
500
477
|
|
|
501
|
-
|
|
478
|
+
### Tasks at a glance
|
|
479
|
+
|
|
480
|
+
| I want to… | Where to look |
|
|
481
|
+
|---|---|
|
|
482
|
+
| Check farm health, find a device, reserve it, manage groups and tags | [Devices](docs/tools.md#devices), [Device Groups](docs/tools.md#device-groups), [Health & Diagnostics](docs/tools.md#health--diagnostics), [Reservations](docs/tools.md#reservations) |
|
|
483
|
+
| Upload, install or find the latest build of an app; manage test files and iOS profiles | [Applications](docs/tools.md#applications), [Repository](docs/tools.md#repository), [Provisioning Profiles](docs/tools.md#provisioning-profiles) |
|
|
484
|
+
| Explore an app or web page live with the AI | [Interactive inspection](#interactive-inspection) below |
|
|
485
|
+
| Create an automated test | [Creating Tests](#creating-tests) |
|
|
486
|
+
| Run Espresso, XCUITest or Maestro suites on the platform | [Test runs](#test-runs-espresso-xcuitest-maestro) below |
|
|
487
|
+
| Find out why tests failed, and track pass rates and trends | [Test Results and Root Cause Analysis](#test-results-and-root-cause-analysis) |
|
|
488
|
+
| Compare performance between versions, devices or networks | [Performance Comparison Reports](#performance-comparison-reports) |
|
|
489
|
+
| Export or summarize platform usage *(Cloud Admin)* | [Usage reports](#usage-reports) below |
|
|
490
|
+
| Set up or wind down a POC or project; manage users | [Workflow Reference](#workflow-reference), [Users](docs/tools.md#users), [Projects](docs/tools.md#projects) |
|
|
491
|
+
|
|
492
|
+
#### Interactive inspection
|
|
493
|
+
|
|
494
|
+
An inspection session opens a live connection to a real device or browser so the AI can see the screen and interact with it — useful for exploring an app and capturing real element IDs before writing a test.
|
|
495
|
+
|
|
496
|
+
- **Mobile** ([tools](docs/tools.md#mobile-inspection-sessions)) — Android and iOS, on both Appium Grid and Appium Server projects: screenshots, element discovery, the full gesture set, keys, app/device control, Android Auto / CarPlay projection and mock biometric authentication. The session shares a watch-only live view URL so you can follow along. iOS behaves differently in a few places ([limitation 29](docs/limitations.md#29-ios-inspection-sessions-differ-from-android)), and Appium Grid projects support fewer commands ([limitation 18](docs/limitations.md#18-appium-grid-projects-support-fewer-commands-than-appium-server)).
|
|
497
|
+
- **Browser** ([tools](docs/tools.md#web-inspection-sessions)) — navigate, discover elements (including inside Shadow DOM), verify CSS selectors and generate browser-neutral Selenium scripts.
|
|
498
|
+
- **Remote debug** ([tools](docs/tools.md#remote-debug)) — `get_remote_debug_command` makes a cloud device appear as if it were plugged into your machine (over ADB for Android, usbmuxd for iOS), so Android Studio, Xcode and command-line ADB can use it — for example for shell commands or file push. Install the app **before** connecting (see [Troubleshooting](#common-symptoms)).
|
|
499
|
+
|
|
500
|
+
Example prompts: [Interactive Inspection](docs/examples.md#interactive-inspection-ai-driven-test-building), [Android Auto / CarPlay](docs/examples.md#android-auto--carplay), [Web Browser Inspection](docs/examples.md#web-browser-inspection-ai-driven-browser-test-building).
|
|
502
501
|
|
|
503
|
-
|
|
502
|
+
#### Test runs (Espresso, XCUITest, Maestro)
|
|
504
503
|
|
|
505
|
-
|
|
504
|
+
The platform runs these suites itself: `execute_test_run` starts a run, `get_test_run_status` follows it, `cancel_test_run` stops it, and `get_test_run_command` gives a command to start it from your own machine. `generate_maestro_flow` builds a Maestro flow from structured steps. Maestro is Android-only. Espresso and Maestro runs are verified; XCUITest uses the same API but is not yet verified end to end. See [Test Runs](docs/tools.md#test-runs-espresso--xcuitest--maestro) and [example prompts](docs/examples.md#test-runs-espresso-xcuitest-maestro).
|
|
506
505
|
|
|
507
|
-
|
|
506
|
+
Appium and Selenium tests are different: you run them from your own Appium or Selenium client (IDE, CI scripts), and the server manages the devices, apps, reservations and results they use.
|
|
507
|
+
|
|
508
|
+
#### Usage reports
|
|
509
|
+
|
|
510
|
+
Cloud Admins can export the platform's usage-report CSVs — device reservations, users/devices/browser usage, user statistics, license usage — or have `summarize_usage_report` aggregate one in memory and return just the summary, with no file written. Usage data is kept for 2 years (platform 25.9+); earlier periods come back empty or partial. A large unfiltered export runs only after you explicitly agree (`confirmLargeExport`, see [Safety Guards](#safety-guards)). See [Usage Reports](docs/tools.md#usage-reports), [limitation 21](docs/limitations.md#21-usage-reports-2-year-retention-large-exports) and [example prompts](docs/examples.md#usage-reports-platform-administration--finops).
|
|
511
|
+
|
|
512
|
+
### All capability areas
|
|
513
|
+
|
|
514
|
+
202 tools across 27 capability areas. The complete per-tool reference — descriptions, filters, access requirements and usage notes — is in **[docs/tools.md](docs/tools.md)**.
|
|
515
|
+
|
|
516
|
+
| Area | Tools | Highlights |
|
|
508
517
|
|---|---|---|
|
|
509
518
|
| [Devices](docs/tools.md#devices) | 19 | List/query devices, find available, release, reboot, tags, Mobile Studio |
|
|
510
519
|
| [Device Groups](docs/tools.md#device-groups) | 9 | Create groups, move devices, control project access |
|
|
@@ -516,7 +525,7 @@ Once connected, talk to the server in plain language — no tool names needed:
|
|
|
516
525
|
| [Projects](docs/tools.md#projects) | 17 | Create/configure projects, settings, app assignment |
|
|
517
526
|
| [Provisioning Profiles](docs/tools.md#provisioning-profiles) | 7 | iOS signing profile lifecycle |
|
|
518
527
|
| [Backup](docs/tools.md#backup) | 1 | Trigger a system backup |
|
|
519
|
-
| [Health & Diagnostics](docs/tools.md#health--diagnostics) |
|
|
528
|
+
| [Health & Diagnostics](docs/tools.md#health--diagnostics) | 7 | Farm snapshot, iOS readiness, agent status, active sessions, reporter storage, license limits and utilization |
|
|
520
529
|
| [Coverage Analytics](docs/tools.md#coverage-analytics) | 2 | Tested-vs-available device gap analysis |
|
|
521
530
|
| [Reporting](docs/tools.md#reporting) | 23 | Search/filter test reports, root-cause analysis, failure summarization, stability, trends, logs, video download (byte ranges), share links, attachment download command, cleanup |
|
|
522
531
|
| [Test Runs](docs/tools.md#test-runs-espresso--xcuitest--maestro) | 5 | Run Espresso, XCUITest and Maestro suites on the platform; status, cancel, local command; generate Maestro flows |
|
|
@@ -525,27 +534,265 @@ Once connected, talk to the server in plain language — no tool names needed:
|
|
|
525
534
|
| [Agents](docs/tools.md#agents) | 2 | Host machine status *(Cloud Admin)* |
|
|
526
535
|
| [Regions](docs/tools.md#regions) | 2 | Region status and infrastructure topology *(Cloud Admin)* |
|
|
527
536
|
| [NV Servers](docs/tools.md#nv-servers) | 2 | Network Virtualization servers *(Cloud Admin)* |
|
|
528
|
-
| [
|
|
537
|
+
| [Server, Environments & Toolsets](docs/tools.md#server-environments--toolsets) | 6 | Server info, connectivity and workflow readiness checks, named connection profiles and runtime switching, toolset loading (plus `record_remediation_note` in debug mode) |
|
|
529
538
|
| [Workflows](docs/tools.md#workflows) | 6 | POC and project lifecycle orchestration |
|
|
530
|
-
| [Boilerplate Generation](docs/tools.md#boilerplate-generation) | 4 | Ready-to-run Appium and Selenium test
|
|
531
|
-
| [Remote Debug](docs/tools.md#remote-debug) | 1 | Connect a cloud device as
|
|
539
|
+
| [Boilerplate Generation](docs/tools.md#boilerplate-generation) | 4 | Ready-to-run Appium and Selenium test projects (Java JUnit 5, Java TestNG, Node.js, Python), Test Orchestrator agent install; validate scripts before delivery |
|
|
540
|
+
| [Remote Debug](docs/tools.md#remote-debug) | 1 | Connect a cloud Android or iOS device as if it were attached locally (ADB / usbmuxd) |
|
|
532
541
|
| [Mobile Inspection Sessions](docs/tools.md#mobile-inspection-sessions) | 24 | AI-driven live device interaction — screenshots, element discovery, full gesture set, keys, app/device control, Android Auto / CarPlay projection, mock biometric authentication |
|
|
533
542
|
| [Web Inspection Sessions](docs/tools.md#web-inspection-sessions) | 8 | AI-driven live browser interaction — navigate, Shadow DOM element discovery, CSS selector verification, browser-neutral Selenium script generation |
|
|
534
543
|
| [Performance Comparison](docs/tools.md#performance-comparison) | 4 | Two-set Speed Index comparison with confound detection, MAD outlier exclusion, and fresh-sample generation |
|
|
535
|
-
| [Usage Reports](docs/tools.md#usage-reports) | 3 | Platform usage-report CSV export and in-memory aggregation — device reservations, users/devices/browser usage, license usage; multi-section CSVs with a User Tag column,
|
|
544
|
+
| [Usage Reports](docs/tools.md#usage-reports) | 3 | Platform usage-report CSV export and in-memory aggregation — device reservations, users/devices/browser usage, user statistics, license usage; multi-section CSVs with a User Tag column, 2-year retention (platform 25.9+); large unfiltered exports run only after you explicitly agree (`confirmLargeExport`) *(Cloud Admin)* |
|
|
536
545
|
| [Resources & Prompts](docs/tools.md#resources--prompts) | — | 2 ambient resources, 7 guided prompts |
|
|
537
546
|
|
|
547
|
+
[↑ Back to top](#table-of-contents)
|
|
548
|
+
|
|
549
|
+
---
|
|
550
|
+
|
|
551
|
+
## Creating Tests
|
|
552
|
+
|
|
553
|
+
### Two ways to create a test
|
|
554
|
+
|
|
555
|
+
The server supports two ways of creating a test, for both mobile and web, and the AI is instructed to pick between them based on context:
|
|
556
|
+
|
|
557
|
+
**Autonomous** — `get_test_boilerplate` (mobile) / `get_web_test_boilerplate` (web)
|
|
558
|
+
- **When:** your intent is specific — a standard flow ("create a login test") or step-level detail — AND the selectors are known (app source in the workspace, or a prior inspection session).
|
|
559
|
+
- **Your involvement:** none required — the AI writes the test from the boilerplate, the source and best practices.
|
|
560
|
+
- **Output:** in an IDE, a local test automation project; in a chat-only client, portable project files shown inline.
|
|
561
|
+
|
|
562
|
+
**Interactive** — `start_inspection_session` / `start_browser_inspection_session`, or the `collaborative_test_creation` / `collaborative_web_test_creation` prompts
|
|
563
|
+
- **When:** the request is vague ("I want a test for app X"), the AI has no access to the app's source (chat-only, or an IDE without this app's code), or you want to watch or drive on a live device or browser.
|
|
564
|
+
- **Your involvement:** collaborative — the AI builds the test step by step with you (browser sessions relay screenshots; mobile sessions share a live device view URL).
|
|
565
|
+
- **Output:** the same, plus selectors verified against the real app or page along the way.
|
|
566
|
+
|
|
567
|
+
**A mix is often best:** even in autonomous mode, the AI can open a short inspection session — with no involvement from you — to capture or check element IDs or CSS selectors against the real build. The AI is explicitly instructed to **never make up selectors**: when in doubt it checks live, and when the mode itself is unclear it asks.
|
|
568
|
+
|
|
569
|
+
> **Stronger guarantees:** the server sends this policy to the AI automatically (in its MCP `instructions`), and both boilerplate tools refuse to produce code for a real app or URL without verified selectors. But not every client passes server `instructions` on to the AI, and no MCP server can stop an AI from writing a test file by hand — so you can add a client-side reinforcement for a guarantee that doesn't depend on the client. See **[Recommended agent guardrails](docs/recommended-agent-guardrails.md)**.
|
|
570
|
+
|
|
571
|
+
### Mobile app tests (`get_test_boilerplate`)
|
|
572
|
+
|
|
573
|
+
Generates a complete, pre-configured Appium test project. The Digital.ai server URL and access key are filled in from the active connection profile — switch profiles with `switch_environment` first if you want the script to carry a project-scoped key instead of your admin key.
|
|
574
|
+
|
|
575
|
+
**Recommended workflow** for building a new test:
|
|
576
|
+
|
|
577
|
+
1. `get_application_info` — confirm package name, launch activity, and app ID
|
|
578
|
+
2. `find_available_device` — select a healthy device and read its `region`
|
|
579
|
+
3. `install_application` — install the app on that device before starting a session
|
|
580
|
+
4. `start_inspection_session` — capture real element IDs with `get_element_tree` / `find_elements`
|
|
581
|
+
5. `get_test_boilerplate(appId, region, …)` — generates code while the inspection session is live, or with `confirmSelectorsVerified: true`
|
|
582
|
+
6. [`validate_test_script`](#check-a-test-before-you-deliver-it-validate_test_script) — before saving or presenting the test
|
|
583
|
+
7. Run the test, then `stop_inspection_session` and `release_device` — explicit cleanup
|
|
584
|
+
|
|
585
|
+
> **Inspection gate:** when you target a real app (`appId`/`packageName`/`bundleIdentifier`), this tool returns **no code** — just a structured `{ status: "blocked", reason: "no_verified_selectors" }` error — UNLESS a live inspection session exists in the MCP server, or you set `confirmSelectorsVerified: true` (for selectors already captured another way: rdb/UIAutomator, `open_mobile_studio`, or authoritative source). The built-in ExperiBank demo (no app identifiers) is never gated. This way you can never receive a placeholder scaffold that could be passed off as a finished test.
|
|
586
|
+
|
|
587
|
+
**Finding element locators:** use an inspection session first — `start_inspection_session` → `get_element_tree` / `find_elements` gives the AI the element tree, screenshots and gestures, and is the only way to capture verified selectors inside the MCP. `open_mobile_studio` (the platform's browser-based UI Inspector) is a visual alternative that needs no local tooling. For direct device access (ADB shell commands and file push on Android, Xcode on iOS), use `get_remote_debug_command` to connect the device locally.
|
|
588
|
+
|
|
589
|
+
> ⚠️ The generated script is a **finished product**, not a discovery tool — don't run it to find out what's on screen. Running it without known element selectors creates Incomplete sessions visible to the whole team. Discover selectors with an inspection session (or `open_mobile_studio`) first.
|
|
590
|
+
|
|
591
|
+
<details>
|
|
592
|
+
<summary><strong>Parameters</strong></summary>
|
|
593
|
+
|
|
594
|
+
| Parameter | Values | Default |
|
|
595
|
+
|---|---|---|
|
|
596
|
+
| `platform` | `android` \| `ios` | _(required)_ |
|
|
597
|
+
| `language` | `java-junit5` \| `java-testng` \| `nodejs` \| `python` | _(required)_ |
|
|
598
|
+
| `appId` | Numeric app ID from `list_applications` | Recommended |
|
|
599
|
+
| `deviceCategory` | `PHONE` \| `TABLET` | `PHONE` |
|
|
600
|
+
| `testName` | Any string | `"My First Mobile Test"` |
|
|
601
|
+
| `packageName` | Android package name (e.g. `com.mycompany.app`) | _(optional)_ |
|
|
602
|
+
| `mainActivity` | Android main activity (e.g. `.MainActivity`) | _(optional)_ |
|
|
603
|
+
| `bundleIdentifier` | iOS bundle ID (e.g. `com.mycompany.app`) | _(optional)_ |
|
|
604
|
+
| `region` | Region code from `find_available_device` response (e.g. `US2`, `SG1`) | _(optional, strongly recommended)_ |
|
|
605
|
+
| `projectType` | `standalone-gradle` \| `standalone-maven` \| `android-gradle-submodule` | `standalone-gradle` |
|
|
606
|
+
| `includePerformanceTransactions` | `true` \| `false` | `false` |
|
|
607
|
+
| `includeAxeScan` | `true` \| `false` | `false` |
|
|
608
|
+
| `confirmSelectorsVerified` | `true` \| `false` | _(optional)_ |
|
|
609
|
+
| `orchestration` | `auto` \| `on` \| `off` | `auto` |
|
|
610
|
+
| `orchestrationMaxRetries` | `0`–`5` | `2` (`0` for placeholder or performance tests) |
|
|
611
|
+
| `automotiveProjection` | Android Auto / CarPlay resolution | _(optional)_ |
|
|
612
|
+
| `outputFormat` | `json` \| `human` | `json` |
|
|
613
|
+
|
|
614
|
+
</details>
|
|
615
|
+
|
|
616
|
+
**Generated files.** `projectType` controls the Java output layout only (it is ignored for Node.js and Python): `standalone-gradle` produces `src/test/java/` with both `build.gradle` and `pom.xml`; `standalone-maven` produces `pom.xml` only; `android-gradle-submodule` puts the files under `e2e-tests/` (with `e2e-tests/build.gradle`) for embedding in an existing Android Studio project.
|
|
617
|
+
|
|
618
|
+
| Language | Files |
|
|
619
|
+
|---|---|
|
|
620
|
+
| `java-junit5` / `java-testng` | `src/test/java/LocalAndroidTest.java` or `LocaliOSTest.java` + build file(s) per `projectType`; orchestrated projects add `orchestrator/config.template.yml`, `orchestrator/.gitignore` and `lib/.gitignore` |
|
|
621
|
+
| `nodejs` | `package.json` + `wdio.conf.js` + `AndroidNative.js` / `iOSNative.js` + `setup-steps.sh` (WebdriverIO 7.40 for Appium Grid projects, WebdriverIO 9 for Appium Server) |
|
|
622
|
+
| `python` | `AndroidNative.py` / `iOSNative.py` + `requirements.txt` |
|
|
623
|
+
|
|
624
|
+
All boilerplate uses the ExperiBank demo app as its starting point. ExperiBank is available on most farm devices, so you can verify connectivity before switching to your own app.
|
|
625
|
+
|
|
626
|
+
**Performance transactions:** pass `includePerformanceTransactions: true` to wrap the test body in `startPerformanceTransaction` / `endPerformanceTransaction` calls. The start argument is the NV network profile (defaults to `"Monitor"` — observe without throttling); the end argument is the transaction name. The platform records CPU, memory, battery and Speed Index metrics; results appear in the reporter Transactions tab about 1 minute after `endPerformanceTransaction` and can be queried with `list_transactions`. Prerequisite: an NV server must be ONLINE and tunnel-connected in the device's region (`list_nv_servers`). A throttling profile turns on network shaping immediately; `"Monitor"` is pass-through and carries no ANR (app-not-responding) risk.
|
|
627
|
+
|
|
628
|
+
**Accessibility scanning:** pass `includeAxeScan: true` to add a Deque Axe DevTools Mobile accessibility scan (a `mobile: axeScan` executeScript call). It sets the required `appium:automationName` capability automatically (`AxeUiAutomator2` for Android, `AxeXCUITest` for iOS) and pins `appiumVersion` to 2.16.2 (the Axe driver is unavailable on Appium Server 3.3.0+; don't raise it). The key comes from `AXE_DEVTOOLS_API_KEY`; if that's unset, a placeholder is written.
|
|
629
|
+
|
|
630
|
+
Both flags can be combined — the Axe scan runs inside the performance transaction.
|
|
631
|
+
|
|
632
|
+
#### Test Orchestrator (default for Java on Appium Server)
|
|
633
|
+
|
|
634
|
+
For **Java (JUnit 5 / TestNG) on an Appium Server project**, the generated project is set up for the **Digital.ai Test Orchestrator** (the Smart Agent): failed tests are retried automatically, each attempt is reported separately, the real pass/fail is synced to Reporter, and runs are grouped by Build ID — with no change to the test code. Appium Grid, Python and Node.js projects get the classic boilerplate. Pass `orchestration: "off"` to get the classic boilerplate for Java too.
|
|
635
|
+
|
|
636
|
+
1. **`get_test_boilerplate`** adds conditional agent wiring to `build.gradle` / `pom.xml`, an `orchestrator/config.template.yml` (retries default to 2; 0 for placeholder or performance tests), plus `orchestrator/.gitignore` and `lib/.gitignore` so the rendered key and the agent JAR can never be committed. No root `.gitignore` is written (it would overwrite yours).
|
|
637
|
+
2. **`install_test_orchestrator_agent(projectDir)`** downloads the agent on demand, verifies its SHA-256 and places it at `lib/smart-agent.jar`. Under Docker it returns a download command (bash and PowerShell) that does the same and deletes the file on a checksum mismatch. If the download fails, the response gives the URL, the checksum and the target path for a manual install.
|
|
638
|
+
3. **Set `DIGITAL_AI_ACCESS_KEY` and run `gradle test` / `mvn test`.** The agent attaches only when the JAR is present **and** the key is set; the key is rendered into the git-ignored `orchestrator/rendered/` at build time, never into a tracked file, and Java 17 is required only in that case. Otherwise the project builds and runs exactly like the classic boilerplate (Gradle prints why the agent was not attached). Set `BUILD_ID` to group a CI build's results.
|
|
639
|
+
|
|
640
|
+
Requirements: a **Java 17 or 21** JDK with a current truststore, JUnit 5 or TestNG 7+, Appium java-client 8–10. See [Limitations](docs/limitations.md#17-test-orchestrator-known-issues) for known issues.
|
|
641
|
+
|
|
642
|
+
The agent JAR is **not** shipped with this server. By default it is downloaded from the [Digital.ai Test Orchestrator sample repository](https://github.com/raheekhandigitalai/Digital.ai-Testing-Test-Orchestrator), pinned to a fixed commit and verified by SHA-256 (see `resources/test-orchestrator/agent.json`). To use a different location — for example an internal mirror — set `TEST_ORCHESTRATOR_JAR_URL` **and** `TEST_ORCHESTRATOR_JAR_SHA256` (both required; a URL without a checksum is refused).
|
|
643
|
+
|
|
644
|
+
### Web tests (`get_web_test_boilerplate`)
|
|
645
|
+
|
|
646
|
+
Generates a complete, pre-configured Selenium test project for browser testing. The Digital.ai server URL and access key are filled in from the active connection profile.
|
|
647
|
+
|
|
648
|
+
**Recommended workflow** for building a new web test:
|
|
649
|
+
|
|
650
|
+
1. `list_available_browsers` — confirm which browsers and OS combinations are available in your region
|
|
651
|
+
2. `start_browser_inspection_session` — open a live browser session on the target URL
|
|
652
|
+
3. `get_page_dom` — extract the rendered DOM, including shadow trees; note element IDs and `data-testid` values
|
|
653
|
+
4. `find_web_elements` / `tap_element` / `type_into_element` — interact with elements to verify selectors work
|
|
654
|
+
5. `take_inspection_screenshot` — relay screenshots at each key step for verification
|
|
655
|
+
6. `stop_browser_inspection_session`
|
|
656
|
+
7. `get_web_test_boilerplate(confirmSelectorsVerified: true)` — generate the verified script (once the session is stopped, pass `shadowDomSupport: "always"` if `get_page_dom` reported shadow DOM)
|
|
657
|
+
|
|
658
|
+
> **Inspection gate:** when you target a real URL (`url` provided), this tool returns **no code** — just a structured `{ status: "blocked", reason: "no_verified_selectors" }` error — UNLESS a live browser inspection session exists in the MCP server, or you set `confirmSelectorsVerified: true` (for selectors captured in a prior `start_browser_inspection_session` session). This mirrors the mobile gate.
|
|
659
|
+
|
|
660
|
+
**Browser-neutral by default:** when `targetBrowser` is omitted, the generated script reads the browser name at runtime from the `BROWSER` environment variable, so the same script runs on Chrome, Firefox, Microsoft Edge, Safari and Opera without changes — just change the variable or your CI matrix. Specify `targetBrowser` only when you need browser-specific options (e.g. Safari-only session setup).
|
|
661
|
+
|
|
662
|
+
**Shadow DOM support:** with `always`, the generated script includes a `shadowQuery(host, cssSelector)` helper; with `auto` (the default), it is included only while a browser inspection session is open in the MCP server; `never` omits it. The helper reaches into shadow roots using `executeScript`. Use it for apps built with Web Components, Lit, Polymer or other shadow-DOM frameworks. `get_page_dom` detects shadow DOM automatically during inspection and reports `hasShadowDom: true` when present.
|
|
663
|
+
|
|
664
|
+
<details>
|
|
665
|
+
<summary><strong>Parameters</strong></summary>
|
|
666
|
+
|
|
667
|
+
| Parameter | Values | Default |
|
|
668
|
+
|---|---|---|
|
|
669
|
+
| `language` | `java-junit5` \| `java-testng` \| `nodejs` \| `python` | _(required)_ |
|
|
670
|
+
| `url` | URL of the page being tested | _(optional)_ |
|
|
671
|
+
| `testName` | Any string | `"WebTest"` |
|
|
672
|
+
| `targetBrowser` | `chrome` \| `firefox` \| `MicrosoftEdge` \| `safari` \| `opera` (names as `list_available_browsers` returns them; `edge` is accepted for `MicrosoftEdge`) | _(optional)_ |
|
|
673
|
+
| `shadowDomSupport` | `auto` \| `always` \| `never` | `auto` |
|
|
674
|
+
| `confirmSelectorsVerified` | `true` \| `false` | _(optional)_ |
|
|
675
|
+
| `outputFormat` | `json` \| `human` | `json` |
|
|
676
|
+
|
|
677
|
+
</details>
|
|
678
|
+
|
|
679
|
+
### Check a test before you deliver it (`validate_test_script`)
|
|
680
|
+
|
|
681
|
+
A final check for the case the boilerplate gate can't catch: a test written by hand with guessed selectors. Pass the full `scriptContent` before presenting or saving any test. It looks for unreplaced `<…>` placeholder selectors, the deliberate scaffold fail-guard, placeholder or made-up credentials, and known fabricated resource IDs. If it finds a serious problem it returns `isError` with a `fail` verdict, so a test that can't work isn't delivered as finished. A `pass` is necessary but not sufficient: it confirms the obvious placeholders are gone, not that the selectors are real — a live inspection is still the authoritative source.
|
|
682
|
+
|
|
683
|
+
| Parameter | Values | Default |
|
|
684
|
+
|---|---|---|
|
|
685
|
+
| `scriptContent` | The full test script text | _(required)_ |
|
|
686
|
+
| `fileName` | Label for the result | _(optional)_ |
|
|
687
|
+
| `outputFormat` | `json` \| `human` | `json` |
|
|
688
|
+
|
|
689
|
+
[↑ Back to top](#table-of-contents)
|
|
690
|
+
|
|
691
|
+
---
|
|
692
|
+
|
|
693
|
+
## Test Results and Root Cause Analysis
|
|
694
|
+
|
|
695
|
+
The reporting tools search test reports, summarize pass/fail results and explain failures. Field names, filter properties and ID types are in the [Test reporting schema](docs/tools.md#test-reporting-schema); more prompts are in [Root Cause Analysis](docs/examples.md#root-cause-analysis-why-tests-are-failing) and [Functional Test Analytics](docs/examples.md#functional-test-analytics).
|
|
696
|
+
|
|
697
|
+
Two things to know:
|
|
698
|
+
- **The list doesn't include failure details.** `list_test_reports` returns summary records; call `get_test_report` on a test to get its cause, error category and stack trace.
|
|
699
|
+
- **Filter by date with `startDate`/`endDate`**, not with a `start_time` filter (the platform rejects that).
|
|
700
|
+
|
|
701
|
+
**Common queries:**
|
|
702
|
+
|
|
703
|
+
```
|
|
704
|
+
# Recent failures
|
|
705
|
+
list_test_reports
|
|
706
|
+
filter: [{"property":"status","operator":"=","value":"Failed"}]
|
|
707
|
+
sort: [{"property":"start_time","descending":true}]
|
|
708
|
+
limit: 20
|
|
709
|
+
|
|
710
|
+
# Date range
|
|
711
|
+
list_test_reports
|
|
712
|
+
startDate: "2026-05-01T00:00:00Z"
|
|
713
|
+
endDate: "2026-05-15T23:59:59Z"
|
|
714
|
+
|
|
715
|
+
# Pass/fail by OS
|
|
716
|
+
get_grouped_test_reports
|
|
717
|
+
groupBy: ["device.os"]
|
|
718
|
+
pivotBy: ["status"]
|
|
719
|
+
|
|
720
|
+
# Project summary
|
|
721
|
+
get_project_test_summary
|
|
722
|
+
startDate: "2026-05-01T00:00:00Z"
|
|
723
|
+
projectName: "My Project"
|
|
724
|
+
```
|
|
725
|
+
|
|
726
|
+
**Root cause analysis of a failure:**
|
|
727
|
+
|
|
728
|
+
```
|
|
729
|
+
# 1. Find the failures
|
|
730
|
+
list_test_reports
|
|
731
|
+
filter: [{"property":"status","operator":"=","value":"Failed"}]
|
|
732
|
+
|
|
733
|
+
# 2. Diagnose — structured cause without any download
|
|
734
|
+
get_test_report(testId: 69)
|
|
735
|
+
→ cause, errorCategory, errorClassification, errorDetail (stack trace)
|
|
736
|
+
|
|
737
|
+
# 2b. Platform root-cause analysis (read-only)
|
|
738
|
+
get_root_cause_analysis(uuid: "…") # or reportUrl / testId
|
|
739
|
+
|
|
740
|
+
# 3. Full Appium log as text, when the structured fields aren't enough
|
|
741
|
+
get_test_log(uuid: "…", logType: "appium")
|
|
742
|
+
|
|
743
|
+
# 4. Pull the binary artifacts (session video .mp4, full ZIP) to YOUR machine.
|
|
744
|
+
# With the npm install, call download_test_video / download_test_attachments
|
|
745
|
+
# directly; under Docker/remote, generate a local command instead:
|
|
746
|
+
get_test_attachments_download_command(uuid: "…", localPlatform: "macos")
|
|
747
|
+
→ a curl command you run locally; the file never transits the container
|
|
748
|
+
```
|
|
749
|
+
|
|
750
|
+
Tests missing from the results? See [Troubleshooting](#common-symptoms) — a project with its own reporter needs `projectName`.
|
|
751
|
+
|
|
752
|
+
[↑ Back to top](#table-of-contents)
|
|
753
|
+
|
|
754
|
+
---
|
|
755
|
+
|
|
756
|
+
## Performance Comparison Reports
|
|
757
|
+
|
|
758
|
+
Compare performance (Speed Index, CPU, memory, battery, duration) between two sets of conditions — app v1 vs v2, device A vs device B, OS version X vs Y, region to region, two network profiles, or two automation scripts. Works with all access levels — Cloud Admin sees all projects; project-level keys (Project Admin and Project User) see only their own project's transactions.
|
|
759
|
+
|
|
760
|
+
The headline metric is reported three ways every time — **trimmed mean, median, and raw mean** — so a small or noisy sample can't hide behind a single number.
|
|
761
|
+
|
|
762
|
+
| Tool | Purpose |
|
|
763
|
+
|---|---|
|
|
764
|
+
| `compare_performance_transactions` | Two-set comparison; trimmed-mean/median/mean per side + delta and % change. MAD outliers excluded by default (≥4 samples/side). Pass `comparisonAxis` to embed a confound check. |
|
|
765
|
+
| `assess_comparison_confounds` | Verdict — **clean / caveated / confounded** — by flagging any dimension other than the declared axis (device model, OS, OS version, network profile, project, transaction name) that varies across or within the sides, plus missing telemetry and sample imbalance. |
|
|
766
|
+
| `detect_performance_outliers` | Robust median/MAD outlier flagging on a single set; returns the kept set and recommended exclusions/re-runs. |
|
|
767
|
+
| `performance_transaction_control` | Phase 2: generate fresh samples inside an inspection session — `start` (with an NV profile) → run the verified flow → `end` (names the record). Records appear in the reporter ~1 min later. |
|
|
768
|
+
|
|
769
|
+
> **Why the confound check matters:** comparing two transactions with Speed Index 1015 vs 1000 looks like a 1.5% version delta — but if they ran on *different device models, different OS versions, and different projects*, and one has no CPU/memory telemetry, `assess_comparison_confounds` returns **confounded** and names each cause, so the delta is never misread as a version regression.
|
|
770
|
+
|
|
771
|
+
The **`performance_comparison_report` prompt** runs the full workflow end to end: define the axis → scrub confounds → negotiate sample size → **require explicit plan confirmation with a time estimate** → run the series with outlier-driven re-runs → report the delta with root-cause reasoning. True host/background interference can't be observed directly; the report approximates device quietness (idle, healthy status, no concurrent reservation) and states that limit explicitly.
|
|
772
|
+
|
|
773
|
+
To record new performance samples from a generated test, see [performance transactions](#mobile-app-tests-get_test_boilerplate) in Creating Tests. More prompts: [Performance Testing Analytics](docs/examples.md#performance-testing-analytics).
|
|
774
|
+
|
|
775
|
+
[↑ Back to top](#table-of-contents)
|
|
776
|
+
|
|
538
777
|
---
|
|
539
778
|
|
|
540
779
|
## Workflow Reference
|
|
541
780
|
|
|
781
|
+
Six workflow tools set up and wind down POC and project environments. They don't make changes themselves: each returns a guided, numbered plan that the AI then carries out with the individual tools (`create_device_group`, `create_user`, …), pausing for your confirmation at the points the plan specifies. The plans use Cloud Admin tools (user, group and project administration), so run them with a Cloud Admin profile.
|
|
782
|
+
|
|
783
|
+
| Goal | POC | General project |
|
|
784
|
+
|---|---|---|
|
|
785
|
+
| Set up | [`create_poc`](#create_poc--onboard-a-new-poc-environment) | [`setup_project`](#setup_project--provision-a-new-project-environment) |
|
|
786
|
+
| Release devices and users, keep the project | [`close_poc`](#close_poc--wind-down-without-deleting) | [`close_project_resources`](#close_project_resources--release-resources-without-deleting) |
|
|
787
|
+
| Delete everything | [`delete_poc`](#delete_poc--full-teardown) | [`teardown_project`](#teardown_project--full-project-deletion) |
|
|
788
|
+
|
|
542
789
|
### POC Lifecycle
|
|
543
790
|
|
|
544
|
-
Three tools cover the full POC lifecycle. All derive the project name (`"<Customer> POC"`) and device tag (`"<customername>poc"`) from `customerName` — pass the same value
|
|
791
|
+
Three tools cover the full POC lifecycle. All derive the project name (`"<Customer> POC"`) and device tag (`"<customername>poc"`) from `customerName` — pass the same value to all three tools.
|
|
545
792
|
|
|
546
793
|
#### `create_poc` — Onboard a new POC environment
|
|
547
794
|
|
|
548
|
-
Collects all parameters
|
|
795
|
+
Collects all parameters up front, checks that the app binaries are in the repository, presents a confirmation summary to the operator (Step 0), then runs 10 steps in order:
|
|
549
796
|
|
|
550
797
|
1. Create (or reuse) a device group named `<Customer> POC`
|
|
551
798
|
2. Locate the Default device group
|
|
@@ -558,7 +805,7 @@ Collects all parameters upfront, presents a confirmation summary to the operator
|
|
|
558
805
|
9. Create users, assign to the POC project, remove from Default, and tag each account with the POC tag. Cloud Admin access is never granted through this workflow.
|
|
559
806
|
10. Assign ExperiBank (or the app identified by `androidPackageName` / `iosBundleId`) to the POC project — if a binary isn't in the repository, the summary reports it as a manual upload and the workflow continues
|
|
560
807
|
|
|
561
|
-
If any step fails, the workflow stops, reports which steps completed with all created resource IDs, and offers resume (
|
|
808
|
+
If any step fails, the workflow stops, reports which steps completed with all created resource IDs, and offers to resume (call it again — existing resources are detected and reused) or unwind (`delete_poc`). Steps 1 and 6 check for an existing group/project by name before creating, so it is safe to re-run after a partial failure.
|
|
562
809
|
|
|
563
810
|
**Required parameters:**
|
|
564
811
|
|
|
@@ -582,10 +829,6 @@ If any step fails, the workflow stops, reports which steps completed with all cr
|
|
|
582
829
|
| `androidPackageName` | `"com.experitest.ExperiBank"` |
|
|
583
830
|
| `iosBundleId` | `"com.experitest.ExperiBank"` |
|
|
584
831
|
|
|
585
|
-
Steps 1 and 6 check for an existing group/project by name before creating — safe to re-run after a partial failure.
|
|
586
|
-
|
|
587
|
-
---
|
|
588
|
-
|
|
589
832
|
#### `close_poc` — Wind down without deleting
|
|
590
833
|
|
|
591
834
|
Reverses the device and user changes from `create_poc` while leaving the project and device group intact:
|
|
@@ -598,27 +841,23 @@ Reverses the device and user changes from `create_poc` while leaving the project
|
|
|
598
841
|
|
|
599
842
|
Only `customerName` is required.
|
|
600
843
|
|
|
601
|
-
> The Digital.ai REST API
|
|
602
|
-
|
|
603
|
-
---
|
|
844
|
+
> The Digital.ai REST API has no endpoint to lock or disable a user. Deleting the account via `delete_user` is the only way the API offers to remove access for users created only for the POC.
|
|
604
845
|
|
|
605
846
|
#### `delete_poc` — Full teardown
|
|
606
847
|
|
|
607
848
|
Performs the same wind-down as `close_poc`, then permanently deletes the device group and the project. Requires `confirmDeletion: true`.
|
|
608
849
|
|
|
609
|
-
Before any destructive action, the workflow gathers a full inventory (project ID, group ID, device list, user breakdown) and presents it for explicit confirmation. Unlike `close_poc`, user processing is
|
|
850
|
+
Before any destructive action, the workflow gathers a full inventory (project ID, group ID, device list, user breakdown) and presents it for explicit confirmation. Unlike `close_poc`, user processing is confirmed once for the whole batch via this inventory rather than user by user.
|
|
610
851
|
|
|
611
852
|
**Best practice:** Run `close_poc` first — it winds down the POC while preserving the project and device group. Note that `close_poc` is not fully reversible: device moves and tag removal can be undone, but it permanently deletes POC-only user accounts (each deletion requires per-user operator confirmation). Use `delete_poc` only once the project data is confirmed no longer needed. Calling `delete_poc` without `confirmDeletion: true` returns a safe explanation and suggests `close_poc` as the alternative.
|
|
612
853
|
|
|
613
|
-
---
|
|
614
|
-
|
|
615
854
|
### Project Lifecycle
|
|
616
855
|
|
|
617
|
-
Three tools mirror the POC lifecycle for general project management. They
|
|
856
|
+
Three tools mirror the POC lifecycle for general project management. They follow the same rule for users who belong to several projects: those users are removed from this project only, never deleted from the platform.
|
|
618
857
|
|
|
619
858
|
#### `setup_project` — Provision a new project environment
|
|
620
859
|
|
|
621
|
-
|
|
860
|
+
If you call it with only a project name (no `region`, `deviceCount` or `users`), the workflow first asks whether to create a **simple** project record only, or a **full** environment (device group, device allocation, user provisioning, app assignment) — and for a full environment, collects those parameters. If any of them are supplied, it goes straight to the full setup and asks you to confirm the parameters.
|
|
622
861
|
|
|
623
862
|
Full setup steps:
|
|
624
863
|
|
|
@@ -637,12 +876,12 @@ Full setup steps:
|
|
|
637
876
|
| Parameter | Description |
|
|
638
877
|
|---|---|
|
|
639
878
|
| `projectName` | The name for the new project |
|
|
640
|
-
| `region` | Target region for device selection |
|
|
641
879
|
|
|
642
880
|
**Optional parameters:**
|
|
643
881
|
|
|
644
882
|
| Parameter | Default | Description |
|
|
645
883
|
|---|---|---|
|
|
884
|
+
| `region` | _(none)_ | Target region for device selection — needed for a full setup; without it, devices are selected from all regions |
|
|
646
885
|
| `deviceCount` | `6` | Total devices to allocate |
|
|
647
886
|
| `iosCount` | `ceil(deviceCount / 2)` | iOS device count |
|
|
648
887
|
| `androidCount` | `floor(deviceCount / 2)` | Android device count |
|
|
@@ -654,177 +893,100 @@ Full setup steps:
|
|
|
654
893
|
| `iosBundleId` | `"com.experitest.ExperiBank"` | iOS bundle ID used to find the app |
|
|
655
894
|
| `projectMemo` | _(none)_ | Notes to record on the project |
|
|
656
895
|
|
|
657
|
-
---
|
|
658
|
-
|
|
659
896
|
#### `close_project_resources` — Release resources without deleting
|
|
660
897
|
|
|
661
|
-
|
|
898
|
+
Winds down the project environment while leaving the project record intact:
|
|
662
899
|
|
|
663
900
|
1. Locate the project and its associated device group
|
|
664
901
|
2. List the devices in the project group
|
|
665
902
|
3. Remove the project-derived tag from each device
|
|
666
903
|
4. Return devices to the Default group
|
|
667
|
-
5. Process users: **delete accounts with no other project memberships; revoke project access only for multi-project users** (each action individually confirmed by the operator)
|
|
904
|
+
5. Process the users carrying the project tag (applied by `setup_project`): **delete accounts with no other project memberships; revoke project access only for multi-project users** (each action individually confirmed by the operator)
|
|
668
905
|
|
|
669
906
|
Only `projectName` is required. Active sessions and installed apps on the devices are not touched — release sessions with `release_orphaned_sessions` and remove apps with `uninstall_application` separately if needed.
|
|
670
907
|
|
|
671
|
-
---
|
|
672
|
-
|
|
673
908
|
#### `teardown_project` — Full project deletion
|
|
674
909
|
|
|
675
910
|
Performs all `close_project_resources` steps, then permanently deletes the device group and the project. Requires `confirmDeletion: true`.
|
|
676
911
|
|
|
677
|
-
The workflow presents a full inventory summary before any irreversible action
|
|
912
|
+
The workflow presents a full inventory summary before any irreversible action; as with `delete_poc`, user processing is confirmed once for the whole batch via this inventory rather than user by user. Users who belong to other projects are protected in the same way.
|
|
678
913
|
|
|
679
914
|
**Best practice:** Run `close_project_resources` first, then confirm the project is no longer needed before calling `teardown_project`.
|
|
680
915
|
|
|
681
|
-
|
|
682
|
-
|
|
683
|
-
## Boilerplate Generation
|
|
684
|
-
|
|
685
|
-
### Two ways to create a test
|
|
686
|
-
|
|
687
|
-
The server supports two test-creation modes for both mobile and web, and the agent is instructed to route between them based on context:
|
|
688
|
-
|
|
689
|
-
| | **Autonomous** (`get_test_boilerplate` / `get_web_test_boilerplate`) | **Interactive** (`start_inspection_session` / `start_browser_inspection_session` / `collaborative_test_creation` / `collaborative_web_test_creation` prompt) |
|
|
690
|
-
|---|---|---|
|
|
691
|
-
| **When** | Intent is specific — a standardized flow ("create a login test") or step-level detail — AND selectors are derivable (app source in the workspace, or a prior inspection session) | Request is vague ("I want a test for app X"), no source access (chat-only, or IDE without this app's code), or the user wants to watch/drive on a live device or browser |
|
|
692
|
-
| **User involvement** | None required — the agent writes the test from boilerplate + source knowledge + best practices | Collaborative — the agent builds the test step by step with the user (browser sessions relay screenshots; mobile sessions share a live device view URL) |
|
|
693
|
-
| **Output** | In an IDE: a local test automation project. Chat-only: portable project files presented inline | Same, plus selectors verified against the real app or page along the way |
|
|
694
|
-
|
|
695
|
-
**Hybrid is often best:** even in autonomous mode, the agent can open a short inspection session — no user involvement — to capture or verify element IDs or CSS selectors against the real build. The agent is explicitly instructed to **never fabricate selectors** — when in doubt it verifies live, and when the mode itself is ambiguous it asks.
|
|
696
|
-
|
|
697
|
-
> **Stronger guarantees:** the server ships this routing policy automatically in its MCP `instructions`, and both boilerplate tools structurally refuse to emit code for a real app or URL without verified selectors. Because not every client surfaces server `instructions` — and no MCP server can intercept a file an agent writes by hand — you can install a client-side reinforcement for a stronger, client-independent guarantee. See **[Recommended agent guardrails](docs/recommended-agent-guardrails.md)**.
|
|
698
|
-
|
|
699
|
-
### `get_test_boilerplate`
|
|
700
|
-
|
|
701
|
-
Generates a complete, pre-configured Appium test script. The Digital.ai server URL and access key are pre-filled from the active connection profile — switch profiles with `switch_environment` first to generate scripts carrying a project-scoped key instead of your admin key.
|
|
702
|
-
|
|
703
|
-
#### Test Orchestrator (default for Java on Appium Server)
|
|
704
|
-
|
|
705
|
-
For **Java (JUnit 5 / TestNG) on an Appium Server project**, the generated project is wired for the **Digital.ai Test Orchestrator** (the Smart Agent): failed tests are retried automatically, each attempt is reported separately, the real pass/fail is synced to Reporter, and runs are grouped by Build ID — with no change to the test code. Appium Grid, Python and NodeJS projects get the classic boilerplate. Pass `orchestration: "off"` to get the classic boilerplate for Java too.
|
|
706
|
-
|
|
707
|
-
| Step | What happens |
|
|
708
|
-
|---|---|
|
|
709
|
-
| 1. `get_test_boilerplate` | Adds conditional agent wiring to `build.gradle` / `pom.xml`, an `orchestrator/config.template.yml` (retries default to 2; 0 for placeholder or performance tests), plus `orchestrator/.gitignore` and `lib/.gitignore` so the rendered key and the agent JAR can never be committed. No root `.gitignore` is written (it would overwrite yours). |
|
|
710
|
-
| 2. `install_test_orchestrator_agent(projectDir)` | Downloads the agent on demand, verifies its SHA-256 and places it at `lib/smart-agent.jar`. Under Docker it returns a download command (bash and PowerShell) that does the same and deletes the file on a checksum mismatch. If the download fails, the response gives the URL, the checksum and the target path for a manual install. |
|
|
711
|
-
| 3. Set `DIGITAL_AI_ACCESS_KEY` and run `gradle test` / `mvn test` | The agent attaches only when the JAR is present **and** the key is set; the key is rendered into the git-ignored `orchestrator/rendered/` at build time, never into a tracked file, and Java 17 is required only in that case. Otherwise the project builds and runs exactly like the classic boilerplate (Gradle prints why the agent was not attached). Set `BUILD_ID` to group a CI build's results. |
|
|
712
|
-
|
|
713
|
-
Requirements: a **Java 17 or 21** JDK with a current truststore, JUnit 5 or TestNG 7+, Appium java-client 8–10. See [Limitations](docs/limitations.md#17-test-orchestrator-known-issues) for known issues.
|
|
714
|
-
|
|
715
|
-
The agent JAR is **not** shipped with this server. By default it is downloaded from the [Digital.ai Test Orchestrator sample repository](https://github.com/raheekhandigitalai/Digital.ai-Testing-Test-Orchestrator), pinned to a fixed commit and verified by SHA-256 (see `resources/test-orchestrator/agent.json`). To use a different location — for example an internal mirror — set `TEST_ORCHESTRATOR_JAR_URL` **and** `TEST_ORCHESTRATOR_JAR_SHA256` (both required; a URL without a checksum is refused).
|
|
716
|
-
|
|
717
|
-
| Parameter | Values | Default |
|
|
718
|
-
|---|---|---|
|
|
719
|
-
| `platform` | `android` \| `ios` | _(required)_ |
|
|
720
|
-
| `language` | `java-junit5` \| `java-testng` \| `nodejs` \| `python` | _(required)_ |
|
|
721
|
-
| `appId` | Numeric app ID from `list_applications` | Recommended |
|
|
722
|
-
| `deviceCategory` | `PHONE` \| `TABLET` | `PHONE` |
|
|
723
|
-
| `testName` | Any string | `"My First Mobile Test"` |
|
|
724
|
-
| `packageName` | Android package name (e.g. `com.mycompany.app`) | _(optional)_ |
|
|
725
|
-
| `mainActivity` | Android main activity (e.g. `.MainActivity`) | _(optional)_ |
|
|
726
|
-
| `bundleIdentifier` | iOS bundle ID (e.g. `com.mycompany.app`) | _(optional)_ |
|
|
727
|
-
| `region` | Region code from `find_available_device` response (e.g. `US2`, `SG1`) | _(optional, strongly recommended)_ |
|
|
728
|
-
| `projectType` | `standalone-gradle` \| `standalone-maven` \| `android-gradle-submodule` | `standalone-gradle` |
|
|
729
|
-
| `includePerformanceTransactions` | `true` \| `false` | `false` |
|
|
730
|
-
| `includeAxeScan` | `true` \| `false` | `false` |
|
|
731
|
-
| `confirmSelectorsVerified` | `true` \| `false` | _(optional)_ |
|
|
732
|
-
| `orchestration` | `auto` \| `on` \| `off` | `auto` |
|
|
733
|
-
| `orchestrationMaxRetries` | `0`–`5` | `2` (`0` for placeholder or performance tests) |
|
|
734
|
-
| `automotiveProjection` | Android Auto / CarPlay resolution | _(optional)_ |
|
|
735
|
-
| `outputFormat` | `json` \| `human` | `json` |
|
|
736
|
-
|
|
737
|
-
> **Inspection gate:** when you target a real app (`appId`/`packageName`/`bundleIdentifier`), this tool returns **no code** and a structured `{ status: "blocked", reason: "no_verified_selectors" }` error UNLESS a live inspection session exists in the MCP process, or you set `confirmSelectorsVerified: true` (escape hatch for selectors already captured via rdb/UIAutomator, `open_mobile_studio`, or authoritative source). The built-in ExperiBank demo (no app identifiers) is never gated. This makes it structurally impossible to receive a placeholder scaffold and pass it off as a finished test.
|
|
738
|
-
|
|
739
|
-
**Recommended workflow** for building a new test:
|
|
740
|
-
|
|
741
|
-
1. `get_application_info` — confirm package name, launch activity, and app ID
|
|
742
|
-
2. `find_available_device` — select a healthy device and read its `region`
|
|
743
|
-
3. `start_inspection_session` (or `open_mobile_studio`) — capture real element IDs
|
|
744
|
-
4. `get_test_boilerplate(appId, region, …)` — generates code while the inspection session is live, or with `confirmSelectorsVerified: true`
|
|
745
|
-
5. `validate_test_script` — before saving or presenting the test
|
|
746
|
-
6. Run the test, then `release_device` — explicit cleanup
|
|
747
|
-
|
|
748
|
-
**Performance transactions:** Pass `includePerformanceTransactions: true` to bracket the test body with `startPerformanceTransaction` / `endPerformanceTransaction` calls. The start arg is the NV network profile (defaults to `"Monitor"` — observe without throttling); the end arg is the transaction name. The platform records CPU, memory, battery, and Speed Index metrics; results appear in the reporter Transactions tab (~1 min after `endPerformanceTransaction`) and are queryable via `list_transactions`. Pre-requisite: an NV server must be ONLINE and tunnel-connected in the device region (`list_nv_servers`). Note: a throttling profile activates NV shaping immediately; `"Monitor"` is pass-through and carries no ANR risk.
|
|
916
|
+
[↑ Back to top](#table-of-contents)
|
|
749
917
|
|
|
750
|
-
|
|
918
|
+
---
|
|
751
919
|
|
|
752
|
-
|
|
920
|
+
## Safety Guards
|
|
753
921
|
|
|
754
|
-
|
|
922
|
+
All destructive operations require `confirmDeletion: true`. Without it:
|
|
755
923
|
|
|
756
|
-
> ⚠️
|
|
924
|
+
> ⚠️ Safety guard triggered. "Delete user 42" is a destructive operation that cannot be undone. To confirm you want to proceed, include `confirmDeletion: true` in your request. No changes were made.
|
|
757
925
|
|
|
758
|
-
|
|
926
|
+
The first call describes what will be deleted. The second call — with `confirmDeletion: true` — carries it out. This prevents accidental data loss when an AI assistant misreads your intent. `release_orphaned_sessions` works the same way: without `confirmDeletion: true` (or with `dryRun: true`) it only previews the devices it would release.
|
|
759
927
|
|
|
760
|
-
|
|
928
|
+
Two other actions have their own confirmation flags:
|
|
761
929
|
|
|
762
|
-
|
|
763
|
-
|
|
764
|
-
| `scriptContent` | The full test script text | _(required)_ |
|
|
765
|
-
| `fileName` | Label for the result | _(optional)_ |
|
|
766
|
-
| `outputFormat` | `json` \| `human` | `json` |
|
|
930
|
+
- **`confirmPublicShare`** — `share_test_report` creates a public link anyone can open without logging in, and it cannot be revoked early (see [limitation 22](docs/limitations.md#22-public-report-links-cannot-be-revoked)). Without the flag it returns a preview only.
|
|
931
|
+
- **`confirmLargeExport`** — usage-report tools stop before an unfiltered export spanning more than 31 days (it can reach hundreds of MB) and ask. Only you should approve such an export, not the AI on its own (see [limitation 21](docs/limitations.md#21-usage-reports-2-year-retention-large-exports)).
|
|
767
932
|
|
|
768
|
-
|
|
933
|
+
More checks run before any request leaves the server:
|
|
769
934
|
|
|
770
|
-
**
|
|
935
|
+
- **Permission check first:** report-deletion tools (`delete_test_reports*`, `cleanup_inspection_sessions`, `cleanup_browser_inspection_sessions`) check your detected access level first. A project-level role is refused up front when its project's `allowUsersDeleteTests` setting is readable and false, with a message telling you to have a Cloud Admin enable it (or switch to a Cloud Admin profile) — instead of an unexplained 403.
|
|
936
|
+
- **Path validation:** tools that read or write local files require an absolute path and refuse path traversal (`..`). Upload tools also refuse credential-file names (`.env*`, SSH private keys), so a misdirected request cannot publish secrets to the cloud repository.
|
|
771
937
|
|
|
772
|
-
|
|
773
|
-
|---|---|
|
|
774
|
-
| `java-junit5` / `java-testng` | `AndroidNative.java` or `iOSNative.java` + `build.gradle` + `pom.xml` |
|
|
775
|
-
| `nodejs` | `wdio.conf.js` + test file + setup shell steps (WebdriverIO 7.40 for Appium Grid projects, WebdriverIO 9 for Appium Server) |
|
|
776
|
-
| `python` | Test file + `requirements.txt` |
|
|
938
|
+
[↑ Back to top](#table-of-contents)
|
|
777
939
|
|
|
778
|
-
|
|
940
|
+
---
|
|
779
941
|
|
|
780
|
-
|
|
942
|
+
## Troubleshooting
|
|
781
943
|
|
|
782
|
-
|
|
944
|
+
### Common symptoms
|
|
783
945
|
|
|
784
|
-
|
|
|
946
|
+
| Symptom | Cause | Fix |
|
|
785
947
|
|---|---|---|
|
|
786
|
-
| `
|
|
787
|
-
| `
|
|
788
|
-
| `
|
|
789
|
-
| `
|
|
790
|
-
| `
|
|
791
|
-
| `
|
|
792
|
-
| `
|
|
793
|
-
|
|
794
|
-
|
|
795
|
-
|
|
796
|
-
**Browser-neutral by default:** when `targetBrowser` is omitted, the generated script reads the browser name at runtime from an environment variable (`BROWSER`). This means the same script runs on Chrome, Firefox, Microsoft Edge, Safari and Opera without modification — just change the env var or CI matrix. Specify `targetBrowser` only when you need browser-specific options (e.g. Safari-only session setup).
|
|
948
|
+
| `403 Forbidden` on admin tools (agents, regions, license) | Active profile has a project-level role (Project Admin or Project User) — v2 endpoints require Cloud Admin access | Ask the AI: *"switch to my Cloud Admin profile"* — it will call `list_environments` and pick the right one. See [Using multiple keys](#using-multiple-keys) |
|
|
949
|
+
| Report delete tools refuse with "Reporter deletes are disabled for project …" | The project's `allowUsersDeleteTests` setting is false, so project-level roles can't delete reports | Ask a Cloud Admin to enable it for the project, or switch to a Cloud Admin profile |
|
|
950
|
+
| `install_application` returns 400 on a device you can see | Device is reserved via an rdb (remote debug) session | Install **first**, then run `get_remote_debug_command` — not the other way around |
|
|
951
|
+
| `install_application` returns 400 (no rdb involved) | App not assigned to a project containing the target device | Call `assign_app_to_project` first |
|
|
952
|
+
| Repeatable `NoSuchElementException` while sibling tests pass | Device health, not test code — device stuck in a wrong state or offline-but-pooled | Run `get_device_health_summary`; scope the deviceQuery with `@region='<healthy-region>'` |
|
|
953
|
+
| Tests missing from `list_test_reports` results | Project has its own reporter instance — unscoped queries search the default scope | Pass `projectName` (exact name from `list_projects`) |
|
|
954
|
+
| rdb fails: `validation error / Failed to reserve device` | With a project-level role, the serial lookup returned an internal device ID instead of the real serial (the `get_remote_debug_command` response warns about this) | Ask the AI: *"switch to Cloud Admin"* → regenerate the rdb script → switch back. See [limitation 13](docs/limitations.md#13-get_remote_debug_command-requires-cloud-admin-for-reliable-serial-resolution) |
|
|
955
|
+
| Device query returns nothing for `@manufacturer` / `@tag` | These fields are silently ignored server-side | Use the `manufacturer`/`tags` parameters on `find_available_device` — they filter client-side |
|
|
956
|
+
| A downloaded file isn't on your machine, or an upload can't find your file (Docker) | Under Docker, file tools use the container's filesystem | See [Local File Transfer](#local-file-transfer-docker--remote-deployments-only) |
|
|
797
957
|
|
|
798
|
-
|
|
958
|
+
### `Failed to reconnect to digital-ai-testing: -32000`
|
|
799
959
|
|
|
800
|
-
|
|
960
|
+
On Windows with Docker, this usually means the `--env-file` path was stored with its backslashes stripped. Open `~/.claude.json`, find the `mcpServers` entry for this project, and check that the env file path looks right (e.g. `C:/projects/digital-ai-testing-mcp/.env`). If it shows something like `C:projectsdigital-ai-testing-mcp.env`, the backslashes were removed. Fix the path in the file directly (use forward slashes), then run `/mcp` in Claude Code to reconnect.
|
|
801
961
|
|
|
802
|
-
|
|
803
|
-
2. `start_browser_inspection_session` — open a live browser session on the target URL
|
|
804
|
-
3. `get_page_dom` — extract the rendered DOM, including shadow trees; note element IDs and `data-testid` values
|
|
805
|
-
4. `find_web_elements` / `tap_element` / `type_into_element` — interact with elements to verify selectors work
|
|
806
|
-
5. `take_inspection_screenshot` — relay screenshots at each key step for verification
|
|
807
|
-
6. `stop_browser_inspection_session`
|
|
808
|
-
7. `get_web_test_boilerplate(confirmSelectorsVerified: true)` — generate the verified script
|
|
962
|
+
### Workflow tools failing?
|
|
809
963
|
|
|
810
|
-
|
|
964
|
+
Run this sequence:
|
|
811
965
|
|
|
812
|
-
|
|
966
|
+
```
|
|
967
|
+
1. get_server_info — confirm tool count (expect 202; 203 with MCP_DEBUG_MODE=true) and active profile
|
|
968
|
+
2. check_workflow_readiness — which dependency tools are present or missing
|
|
969
|
+
3. check_connectivity — confirm the backend API is reachable
|
|
970
|
+
```
|
|
813
971
|
|
|
814
|
-
|
|
972
|
+
`check_workflow_readiness` returns a structured report:
|
|
815
973
|
|
|
816
|
-
|
|
974
|
+
```json
|
|
975
|
+
{
|
|
976
|
+
"allWorkflowsReady": true,
|
|
977
|
+
"registeredToolCount": 202,
|
|
978
|
+
"workflows": {
|
|
979
|
+
"create_poc": { "ready": true, "missingRead": [], "missingWrite": [] },
|
|
980
|
+
"setup_project": { "ready": true, "missingRead": [], "missingWrite": [] }
|
|
981
|
+
}
|
|
982
|
+
}
|
|
983
|
+
```
|
|
817
984
|
|
|
818
|
-
|
|
819
|
-
|---|---|
|
|
820
|
-
| `compare_performance_transactions` | Two-set comparison; trimmed-mean/median/mean per side + delta and % change. MAD outliers excluded by default (≥4 samples/side). Pass `comparisonAxis` to embed a confound check. |
|
|
821
|
-
| `assess_comparison_confounds` | Verdict — **clean / caveated / confounded** — by flagging any dimension other than the declared axis (device model, OS, OS version, network profile, project, transaction name) that varies across or within the sides, plus missing telemetry and sample imbalance. |
|
|
822
|
-
| `detect_performance_outliers` | Robust median/MAD outlier flagging on a single set; returns the kept set and recommended exclusions/re-runs. |
|
|
823
|
-
| `performance_transaction_control` | Phase 2: generate fresh samples inside an inspection session — `start` (with an NV profile) → run the verified flow → `end` (names the record). Records appear in the reporter ~1 min later. |
|
|
985
|
+
If `ready` is `false`, `missingRead` and `missingWrite` list exactly which tools are absent. The most common cause is an outdated install — see [Upgrading](#upgrading).
|
|
824
986
|
|
|
825
|
-
|
|
987
|
+
The server also logs a readiness check at startup (visible in your terminal, or `docker logs`/your AI client's MCP server logs if running in a container): `Workflow readiness: all workflows ready ✓`, or `⚠️ DEGRADED: create_poc — missing: ...` when tools are absent.
|
|
826
988
|
|
|
827
|
-
|
|
989
|
+
[↑ Back to top](#table-of-contents)
|
|
828
990
|
|
|
829
991
|
---
|
|
830
992
|
|
|
@@ -839,7 +1001,7 @@ All data-returning tools accept an `outputFormat` parameter:
|
|
|
839
1001
|
| `"json"` | ✅ Yes | Chaining tool calls — IDs and values pass directly to the next tool without parsing |
|
|
840
1002
|
| `"human"` | No | Displaying results to an operator — formatted prose with status indicators |
|
|
841
1003
|
|
|
842
|
-
The default is `"json"` because the
|
|
1004
|
+
The default is `"json"` because the main consumers are AI agents that need to pick out IDs for follow-up calls. An agent can turn JSON into readable output for you; it can't reliably pull IDs out of prose.
|
|
843
1005
|
|
|
844
1006
|
```
|
|
845
1007
|
# Example: pass device group IDs without string manipulation
|
|
@@ -852,13 +1014,13 @@ get_devices_in_group { groupId: "7" }
|
|
|
852
1014
|
add_devices_to_group { groupId: "12", deviceIds: ["abc123"] }
|
|
853
1015
|
```
|
|
854
1016
|
|
|
855
|
-
Tools that only
|
|
1017
|
+
Tools that only make changes (create, delete, install, etc.) return simple success messages and have no `outputFormat` parameter. One exception to the `json` default: `get_page_dom` defaults to `human` (a readable element list); pass `outputFormat: "json"` to chain its results.
|
|
856
1018
|
|
|
857
1019
|
### Understanding maxResults
|
|
858
1020
|
|
|
859
|
-
The Digital.ai API returns complete datasets —
|
|
1021
|
+
The Digital.ai API returns complete datasets — most endpoints have no server-side paging. The MCP caps list responses at `maxResults` (default: 50, max: 500) to keep responses a manageable size for AI assistants. `list_test_reports` uses `limit` (same default and maximum) plus `page`, and `find_elements` details at most 20 matches by default (see [limitation 19](docs/limitations.md#19-find_elements-details-only-the-first-20-matches)).
|
|
860
1022
|
|
|
861
|
-
Use filters to narrow results rather than raising `maxResults`. When results are
|
|
1023
|
+
Use filters to narrow results rather than raising `maxResults`. When results are cut off, you will see:
|
|
862
1024
|
|
|
863
1025
|
> ⚠️ Showing 50 of 312 results. Use filters to narrow results, or increase maxResults (max 500) to see more.
|
|
864
1026
|
|
|
@@ -873,110 +1035,21 @@ Use filters to narrow results rather than raising `maxResults`. When results are
|
|
|
873
1035
|
| `list_projects` | name (partial match) | name, id |
|
|
874
1036
|
| `list_project_users` | username (partial), role (exact) | username, role |
|
|
875
1037
|
| `list_applications` | nameContains, osType, packageName, bundleIdentifier, uniqueName, fileType, isForSimulator | applicationName, releaseVersion, buildVersion, createdAt (default), osType, fileType |
|
|
876
|
-
| `list_test_reports` |
|
|
877
|
-
|
|
878
|
-
All list tools accept `sortOrder: "asc" | "desc"` (default `"asc"`, except `list_applications`, which defaults to `"desc"`, newest first).
|
|
879
|
-
|
|
880
|
-
`list_test_reports` sorting is performed by the platform for every access level. If the platform ever refuses it for a credential, the server transparently falls back to a client-side scan (up to 5,000 records) so "newest first" results stay correct, just slower on large report sets.
|
|
881
|
-
|
|
882
|
-
### Test Reporting Schema
|
|
883
|
-
|
|
884
|
-
Report fields use **snake_case** to match the API response:
|
|
885
|
-
|
|
886
|
-
| Field | Type | Description |
|
|
887
|
-
|---|---|---|
|
|
888
|
-
| `uuid` | string | Unique test execution identifier |
|
|
889
|
-
| `test_id` | number | Numeric reporter ID |
|
|
890
|
-
| `name` | string | Test name |
|
|
891
|
-
| `status` | string | `Passed`, `Failed`, `Incomplete`, `Skipped`, `Error`, `Healed` |
|
|
892
|
-
| `success` | boolean | True when status is Passed |
|
|
893
|
-
| `start_time` | string | ISO 8601 execution start timestamp |
|
|
894
|
-
| `duration` | number \| null | Milliseconds (`null` for in-progress sessions) |
|
|
895
|
-
| `project_id` | number | Owning project |
|
|
896
|
-
| `has_attachment` | string | `"Y"` or `"N"` |
|
|
897
|
-
|
|
898
|
-
**Diagnostic fields — `get_test_report` (single record) only, NOT present on `list_test_reports`:**
|
|
899
|
-
|
|
900
|
-
| Field | Type | Description |
|
|
901
|
-
|---|---|---|
|
|
902
|
-
| `cause` | string | Human-readable failure cause (e.g. `"no such element - …"`) |
|
|
903
|
-
| `errorCategory` | string | Broad bucket (e.g. `assertion`) |
|
|
904
|
-
| `errorClassification` | string | Specific class (e.g. `element_not_found`) |
|
|
905
|
-
| `errorDetail` | string | Full error object including stack trace |
|
|
906
|
-
| `subTestCount` / `failedSubTestCount` | number | Sub-test totals for merged reports |
|
|
907
|
-
|
|
908
|
-
To diagnose a failure, list with `list_test_reports`, then call `get_test_report(testId)` per test — the list view does not carry the diagnostic fields (the backend omits them there).
|
|
909
|
-
|
|
910
|
-
**Confirmed working filter properties:** `status`, `name` (with `contains` for substring match), `has_attachment`, `success`, `test_id`, `project_id`, `device.os` (case-sensitive: `"Android"` / `"iOS"`), `duration`, `attachment_count`, `attachments_size`, `status_code`. Operators: `=`, `>`, `<`, `>=`, `<=`, `contains`.
|
|
911
|
-
|
|
912
|
-
**CSRF-blocked filters** — return 401 regardless of key type: `start_time`, `create_time`, `uuid`. Use the `startDate`/`endDate` parameters on `list_test_reports` for date-range filtering instead — these are applied client-side after fetching.
|
|
913
|
-
|
|
914
|
-
**Reporter ID types:**
|
|
915
|
-
|
|
916
|
-
| Identifier | Source | How to retrieve |
|
|
917
|
-
|---|---|---|
|
|
918
|
-
| `uuid` | List results, report URLs (`/reporter/video-report/<uuid>`) | `get_test_report(uuid: "...")` or `get_test_report(reportUrl: "...")` — preferred, globally unique |
|
|
919
|
-
| `test_id` | List results | `get_test_report(testId: N)` — unique only within a project; the same number can be a different test in another project |
|
|
920
|
-
| `report_api_id` | Returned by session-start tools | `get_test_by_report_id(reportApiId: "...")` — wait ~60 s after session close |
|
|
921
|
-
|
|
922
|
-
**Examples:**
|
|
923
|
-
|
|
924
|
-
```
|
|
925
|
-
# Recent failures
|
|
926
|
-
list_test_reports
|
|
927
|
-
filter: [{"property":"status","operator":"=","value":"Failed"}]
|
|
928
|
-
sort: [{"property":"start_time","descending":true}]
|
|
929
|
-
limit: 20
|
|
930
|
-
|
|
931
|
-
# Date range
|
|
932
|
-
list_test_reports
|
|
933
|
-
startDate: "2026-05-01T00:00:00Z"
|
|
934
|
-
endDate: "2026-05-15T23:59:59Z"
|
|
935
|
-
|
|
936
|
-
# Pass/fail by OS
|
|
937
|
-
get_grouped_test_reports
|
|
938
|
-
groupBy: ["device.os"]
|
|
939
|
-
pivotBy: ["status"]
|
|
940
|
-
|
|
941
|
-
# Project summary
|
|
942
|
-
get_project_test_summary
|
|
943
|
-
startDate: "2026-05-01T00:00:00Z"
|
|
944
|
-
projectName: "My Project"
|
|
945
|
-
```
|
|
946
|
-
|
|
947
|
-
#### Root cause analysis of a failure
|
|
948
|
-
|
|
949
|
-
```
|
|
950
|
-
# 1. Find the failures
|
|
951
|
-
list_test_reports
|
|
952
|
-
filter: [{"property":"status","operator":"=","value":"Failed"}]
|
|
953
|
-
|
|
954
|
-
# 2. Diagnose — structured cause without any download
|
|
955
|
-
get_test_report(testId: 69)
|
|
956
|
-
→ cause, errorCategory, errorClassification, errorDetail (stack trace)
|
|
957
|
-
|
|
958
|
-
# 2b. Platform root-cause analysis (read-only)
|
|
959
|
-
get_root_cause_analysis(uuid: "…") # or reportUrl / testId
|
|
1038
|
+
| `list_test_reports` | `filter` array (see [Test reporting schema](docs/tools.md#test-reporting-schema)), `searchValue` (name substring), startDate/endDate (date range), projectId/projectName | `sort` array, e.g. `start_time` descending |
|
|
960
1039
|
|
|
961
|
-
|
|
962
|
-
get_test_log(uuid: "…", logType: "appium")
|
|
1040
|
+
The list tools with `sortBy` also accept `sortOrder: "asc" | "desc"` (default `"asc"`, except `list_applications`, which defaults to `"desc"`, newest first).
|
|
963
1041
|
|
|
964
|
-
|
|
965
|
-
# With the npm install, call download_test_video / download_test_attachments
|
|
966
|
-
# directly; under Docker/remote, generate a local command instead:
|
|
967
|
-
get_test_attachments_download_command(uuid: "…", localPlatform: "macos")
|
|
968
|
-
→ a curl command you run locally; the file never transits the container
|
|
969
|
-
```
|
|
1042
|
+
`list_test_reports` sorting is done by the platform for every access level. If the platform ever refuses it for a credential, the server automatically falls back to scanning up to 5,000 records itself, so "newest first" results stay correct, just slower on large report sets.
|
|
970
1043
|
|
|
971
1044
|
### Local File Transfer (Docker / remote deployments only)
|
|
972
1045
|
|
|
973
|
-
**If you installed
|
|
1046
|
+
**If you installed the [npm package](#option-a--install-via-npm-recommended) (Option A) and run the server on your own machine, skip this section** — the `download_*`/`upload_*` tools already read and write your local files directly.
|
|
974
1047
|
|
|
975
|
-
The rest of this section applies only to Docker
|
|
1048
|
+
The rest of this section applies only to Docker or remote deployments (Option B or C). There, any tool that reads or writes a local file path works on the **MCP server's own filesystem** — which, in the published Docker image or a remote deployment, is **not** your machine. A `download_*` tool writes into the container; an `upload_*` tool reads from it. Neither is reachable from your local shell unless you mount a shared folder into the container.
|
|
976
1049
|
|
|
977
|
-
For those deployments, every file-transfer tool has a **command-generator
|
|
1050
|
+
For those deployments, every file-transfer tool has a **command-generator** counterpart that produces a `curl` / PowerShell command you run locally, so the file moves directly between your machine and the platform — never through the container:
|
|
978
1051
|
|
|
979
|
-
| Direction | Server-side tool (shared
|
|
1052
|
+
| Direction | Server-side tool (shared filesystem only) | Local-command counterpart (Docker/remote) |
|
|
980
1053
|
|---|---|---|
|
|
981
1054
|
| Upload app | `upload_application_file` | `get_application_upload_command` |
|
|
982
1055
|
| Upload repo file | `upload_repository_file` | `get_repository_upload_command` |
|
|
@@ -987,38 +1060,50 @@ For those deployments, every file-transfer tool has a **command-generator siblin
|
|
|
987
1060
|
| Download profile | `download_provisioning_profile` | `get_provisioning_profile_download_command` |
|
|
988
1061
|
| Download app language files | `extract_app_language_files` | `get_app_language_files_download_command` |
|
|
989
1062
|
| Download iOS app container | `download_ios_app_container` | `get_ios_app_container_download_command` |
|
|
1063
|
+
| Download usage report CSV | `download_usage_report` | `get_usage_report_download_command` (or `summarize_usage_report`, which returns an aggregated summary with no file at all) |
|
|
1064
|
+
| Install Test Orchestrator agent | `install_test_orchestrator_agent` | the same tool — under Docker it returns bash and PowerShell download commands instead of writing the file |
|
|
990
1065
|
|
|
991
|
-
For test **logs
|
|
992
|
-
|
|
993
|
-
---
|
|
1066
|
+
For test **logs**, `get_test_log` is simpler still — it returns the Appium/device log text directly in the response, with no command or download needed. The generated commands contain the active access key in plain text, so run them right away and don't save the output. Background: [limitation 15](docs/limitations.md#15-local-file-io-runs-on-the-mcp-servers-filesystem-not-yours).
|
|
994
1067
|
|
|
995
|
-
|
|
1068
|
+
[↑ Back to top](#table-of-contents)
|
|
996
1069
|
|
|
997
|
-
|
|
1070
|
+
---
|
|
998
1071
|
|
|
999
|
-
|
|
1072
|
+
## Known Limitations
|
|
1000
1073
|
|
|
1001
|
-
The
|
|
1074
|
+
The ones you're most likely to run into:
|
|
1002
1075
|
|
|
1003
|
-
|
|
1076
|
+
1. **Appium and Selenium tests aren't started by the server.** You launch them from your own Appium or Selenium client (IDE, CI scripts); the server manages the devices, apps, reservations and results. Espresso, XCUITest and Maestro suites are different — the platform runs them itself (see [Test runs](#test-runs-espresso-xcuitest-maestro)).
|
|
1077
|
+
2. **iOS inspection sessions differ from Android:** no clearing app data, no clipboard on Appium Grid devices, and back navigation uses the navigation-bar button (iOS has no Back button). See [limitation 29](docs/limitations.md#29-ios-inspection-sessions-differ-from-android).
|
|
1078
|
+
3. **Appium Grid projects support fewer commands than Appium Server** — for example no `-ios predicate string` locator, no Android Auto / CarPlay projection, no pinch/zoom. See [limitation 18](docs/limitations.md#18-appium-grid-projects-support-fewer-commands-than-appium-server).
|
|
1079
|
+
4. **Under Docker, file tools work on the container's filesystem, not yours.** Use the command-generator tools instead — see [Local File Transfer](#local-file-transfer-docker--remote-deployments-only) and [limitation 15](docs/limitations.md#15-local-file-io-runs-on-the-mcp-servers-filesystem-not-yours).
|
|
1080
|
+
5. **Usage reports go back 2 years** (platform 25.9+), and large unfiltered exports need your explicit approval. See [limitation 21](docs/limitations.md#21-usage-reports-2-year-retention-large-exports).
|
|
1081
|
+
6. **There's no way to disable or lock a user.** Removing access for a user created only for a POC means deleting the account — which is why `close_poc` asks for confirmation per user.
|
|
1082
|
+
7. **Deleting test reports with a project-level key depends on a per-project setting** (`allowUsersDeleteTests`). Cloud Admin can always delete. When the setting is off, the tools refuse up front for a Project Admin and explain how to get it enabled; a Project User cannot read the setting, so the platform returns its own 403.
|
|
1004
1083
|
|
|
1005
|
-
|
|
1006
|
-
- **Upload path validation:** tools that read local files for upload refuse relative paths, path traversal, and credential-file names (`.env*`, SSH private keys) — a misdirected request cannot publish secrets to the cloud repository.
|
|
1084
|
+
See [docs/limitations.md](docs/limitations.md) for the full list.
|
|
1007
1085
|
|
|
1008
1086
|
---
|
|
1009
1087
|
|
|
1010
1088
|
## Development
|
|
1011
1089
|
|
|
1090
|
+
To build and run the server from a clone (see [Option C](#option-c--build-from-source)), and for how to contribute, see [CONTRIBUTING.md](CONTRIBUTING.md). Security reports: [SECURITY.md](SECURITY.md).
|
|
1091
|
+
|
|
1012
1092
|
```bash
|
|
1013
1093
|
npm install
|
|
1014
1094
|
npm run build # compile TypeScript to dist/
|
|
1015
1095
|
npm run dev # tsx watch for live reload
|
|
1096
|
+
npm run typecheck
|
|
1097
|
+
npm run lint
|
|
1016
1098
|
```
|
|
1017
1099
|
|
|
1018
|
-
**Running tests** —
|
|
1100
|
+
**Running tests** — the per-domain suites (devices, users, reporting, …) need a `.env` with valid credentials and call the real Digital.ai API. `test:tools`, `test:utils` and the other unit suites run offline; `test:tools` exercises registered tool handlers (guards, access checks, path validation) through an in-memory MCP transport with no live API access.
|
|
1101
|
+
|
|
1102
|
+
<details>
|
|
1103
|
+
<summary><strong>All test scripts</strong></summary>
|
|
1019
1104
|
|
|
1020
1105
|
```bash
|
|
1021
|
-
npm run test # all tests
|
|
1106
|
+
npm run test # all tests (includes the live suites)
|
|
1022
1107
|
npm run test:tools # tool-layer guard/gate regression tests (offline)
|
|
1023
1108
|
npm run test:devices # device management
|
|
1024
1109
|
npm run test:users # user management
|
|
@@ -1032,90 +1117,23 @@ npm run test:device-groups # device groups
|
|
|
1032
1117
|
npm run test:repository # file repository
|
|
1033
1118
|
npm run test:provisioning # iOS provisioning profiles
|
|
1034
1119
|
npm run test:health # environment health
|
|
1035
|
-
npm run test:utils # utility functions
|
|
1120
|
+
npm run test:utils # utility functions (offline)
|
|
1036
1121
|
npm run test:transactions # performance transactions
|
|
1037
1122
|
npm run test:analytics # analytics tools
|
|
1038
1123
|
npm run test:infrastructure # agents / regions / NV servers / sessions
|
|
1039
1124
|
npm run test:live # live API-behavior probes (requires .env; not part of npm test)
|
|
1040
|
-
npm run test:fidelity # agent tool-choice evaluation (builds first)
|
|
1125
|
+
npm run test:fidelity # agent tool-choice evaluation (builds first; spends model tokens)
|
|
1041
1126
|
npm run remediation:digest # summarize debug-mode remediation notes
|
|
1042
1127
|
```
|
|
1043
1128
|
|
|
1044
1129
|
See `package.json` for the full list of scripts.
|
|
1045
1130
|
|
|
1046
|
-
|
|
1047
|
-
|
|
1048
|
-
## Troubleshooting
|
|
1049
|
-
|
|
1050
|
-
### Common symptoms
|
|
1051
|
-
|
|
1052
|
-
| Symptom | Cause | Fix |
|
|
1053
|
-
|---|---|---|
|
|
1054
|
-
| `403 Forbidden` on admin tools (agents, regions, license) | Active profile is a project-level key — v2 endpoints require Cloud Admin access | Ask Agent: *"switch to my Cloud Admin profile"* — it will call `list_environments` and pick the right one |
|
|
1055
|
-
| Report delete tools refuse with "Reporter deletes are disabled for project …" | The project's `allowUsersDeleteTests` setting is false, so project-level roles can't delete reports | Ask a Cloud Admin to enable it for the project, or switch to a Cloud Admin profile |
|
|
1056
|
-
| `install_application` returns 400 on a device you can see | Device is reserved via an rdb (remote debug) session | Install **first**, then run `get_remote_debug_command` — not the other way around |
|
|
1057
|
-
| `install_application` returns 400 (no rdb involved) | App not assigned to a project containing the target device | Call `assign_app_to_project` first |
|
|
1058
|
-
| Repeatable `NoSuchElementException` while sibling tests pass | Device health, not test code — device stuck in a wrong state or offline-but-pooled | Run `get_device_health_summary`; scope the deviceQuery with `@region='<healthy-region>'` |
|
|
1059
|
-
| Tests missing from `list_test_reports` results | Project has its own reporter instance — unscoped queries search the default scope | Pass `projectName` (exact name from `list_projects`) |
|
|
1060
|
-
| rdb fails: `validation error / Failed to reserve device` | Project-level key resolved an internal device ID instead of the real serial | Ask Agent: *"switch to Cloud Admin"* → regenerate the rdb script → switch back |
|
|
1061
|
-
| Device query returns nothing for `@manufacturer` / `@tag` | These fields are silently ignored server-side | Use the `manufacturer`/`tags` parameters on `find_available_device` — they filter client-side |
|
|
1062
|
-
|
|
1063
|
-
### `Failed to reconnect to digital-ai-testing: -32000`
|
|
1064
|
-
|
|
1065
|
-
On Windows this usually means the `--env-file` path was stored with backslashes stripped. Open `~/.claude.json`, find the `mcpServers` entry for this project, and check that the env file path looks correct (e.g. `C:/projects/digital-ai-testing-mcp/.env`). If it shows something like `C:projectsdigital-ai-testing-mcp.env`, the backslashes were eaten. Fix the path in the file directly (use forward slashes), then run `/mcp` in Claude Code to reconnect.
|
|
1066
|
-
|
|
1067
|
-
### Workflow tools failing?
|
|
1068
|
-
|
|
1069
|
-
Run this sequence:
|
|
1070
|
-
|
|
1071
|
-
```
|
|
1072
|
-
1. get_server_info — confirm tool count (expect 202; 203 with MCP_DEBUG_MODE=true) and active profile
|
|
1073
|
-
2. check_workflow_readiness — which dependency tools are present or missing
|
|
1074
|
-
3. check_connectivity — confirm the backend API is reachable
|
|
1075
|
-
```
|
|
1076
|
-
|
|
1077
|
-
`check_workflow_readiness` returns a structured report:
|
|
1078
|
-
|
|
1079
|
-
```json
|
|
1080
|
-
{
|
|
1081
|
-
"allWorkflowsReady": true,
|
|
1082
|
-
"registeredToolCount": 202,
|
|
1083
|
-
"workflows": {
|
|
1084
|
-
"create_poc": { "ready": true, "missingRead": [], "missingWrite": [] },
|
|
1085
|
-
"setup_project": { "ready": true, "missingRead": [], "missingWrite": [] }
|
|
1086
|
-
}
|
|
1087
|
-
}
|
|
1088
|
-
```
|
|
1089
|
-
|
|
1090
|
-
If `ready` is `false`, `missingRead` and `missingWrite` list exactly which tools are absent. The most common cause is a stale build — update it:
|
|
1091
|
-
|
|
1092
|
-
```bash
|
|
1093
|
-
# npm install (Option A)
|
|
1094
|
-
npm install -g digital-ai-testing-mcp@latest
|
|
1095
|
-
|
|
1096
|
-
# Docker (Option B/C)
|
|
1097
|
-
docker pull ghcr.io/digitalai-opensource/digital-ai-testing-mcp:latest # or: docker build -t digital-ai-testing-mcp:latest .
|
|
1098
|
-
```
|
|
1099
|
-
|
|
1100
|
-
The server also logs a readiness check at startup (visible in your terminal, or `docker logs`/your AI client's MCP server logs if running in a container): `Workflow readiness: all workflows ready ✓`, or `⚠️ DEGRADED: create_poc — missing: ...` when tools are absent.
|
|
1101
|
-
|
|
1102
|
-
---
|
|
1103
|
-
|
|
1104
|
-
## Known Limitations
|
|
1105
|
-
|
|
1106
|
-
The four most commonly encountered:
|
|
1107
|
-
|
|
1108
|
-
1. **No API to trigger Appium or Selenium test execution.** Those tests launch from Appium clients (IDE, CI scripts); the server manages devices, apps, reservations, and results. Espresso, XCUITest and Maestro suites are different: the platform runs them itself via `execute_test_run` / `get_test_run_status` / `cancel_test_run` / `get_test_run_command` (Maestro is Android-only). Espresso and Maestro runs are verified; XCUITest uses the same API but is not yet verified end to end. Interactive [mobile inspection sessions](docs/tools.md#mobile-inspection-sessions) and [browser inspection sessions](docs/tools.md#web-inspection-sessions) are also live WebDriver sessions for element discovery on Android, iOS, and desktop browsers.
|
|
1109
|
-
2. **Inspection sessions support Android and iOS** on both the legacy Appium Grid (JWP) and Appium Server (W3C/OSS) projects. iOS caveats: no clear-app-data, no clipboard on Grid devices, and back navigation uses the nav-bar button (iOS has no Back button).
|
|
1110
|
-
3. **No user disable/lock endpoint.** Removing access for a user provisioned solely for a POC means deleting the account — which is why `close_poc` requires per-user confirmation.
|
|
1111
|
-
4. **Deleting test reports with a project-level key depends on a per-project setting** (`allowUsersDeleteTests`). Cloud Admin can always delete. When the setting is off, the tools refuse up front and explain how to get it enabled.
|
|
1112
|
-
|
|
1113
|
-
See [docs/limitations.md](docs/limitations.md) for the full list.
|
|
1131
|
+
</details>
|
|
1114
1132
|
|
|
1115
1133
|
---
|
|
1116
1134
|
|
|
1117
1135
|
## License
|
|
1118
|
-
|
|
1136
|
+
|
|
1119
1137
|
This project is licensed under the Apache License, Version 2.0. See [LICENSE](./LICENSE) for the full license text.
|
|
1120
|
-
|
|
1138
|
+
|
|
1121
1139
|
Use of this software is also subject to the [Supplemental Terms of Use](./SUPPLEMENTAL_TERMS.md), which address AI client responsibilities, agentic action risks, liability limitations, and other obligations specific to the Digital.ai MCP Server for Testing. By using this software, you agree to both the Apache 2.0 License and the Supplemental Terms.
|