assuremind 1.0.1 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +5 -4
- package/LICENSE +20 -20
- package/README.md +94 -311
- package/dist/cli/index.js +1400 -303
- package/dist/cli/index.js.map +1 -1
- package/dist/index.d.mts +59 -0
- package/dist/index.d.ts +59 -0
- package/dist/index.js +43 -3
- package/dist/index.js.map +1 -1
- package/dist/index.mjs +43 -3
- package/dist/index.mjs.map +1 -1
- package/docs/CLI-REFERENCE.md +17 -0
- package/docs/GETTING-STARTED.md +7 -0
- package/docs/STUDIO.md +37 -3
- package/package.json +3 -1
- package/templates/{AUTOMIND.md → ASSUREMIND.md} +30 -2
- package/templates/docs/CLI-REFERENCE.md +18 -1
- package/templates/docs/GETTING-STARTED.md +7 -1
- package/templates/docs/STUDIO.md +43 -2
- package/ui/dist/assets/index-By2Hw5l2.css +1 -0
- package/ui/dist/assets/index-DaQ-JHje.js +819 -0
- package/ui/dist/favicon.svg +36 -36
- package/ui/dist/index.html +2 -2
- package/ui/dist/assets/index-CdtAorWT.js +0 -819
- package/ui/dist/assets/index-KjpMCzao.css +0 -1
package/CONTRIBUTING.md
CHANGED
|
@@ -38,6 +38,7 @@ assuremind/
|
|
|
38
38
|
│ ├── cli/ # CLI entry point and commands (init, run, generate, …)
|
|
39
39
|
│ ├── engine/ # Test runner, executor, self-healing, Allure reporter
|
|
40
40
|
│ ├── ai/ # Smart router, provider adapters, prompts, code cache
|
|
41
|
+
│ ├── mcp/ # Playwright MCP integration (accessibility snapshots, action mapper)
|
|
41
42
|
│ ├── storage/ # File-based JSON stores (suites, cases, results, healing)
|
|
42
43
|
│ ├── server/ # Fastify API server + WebSocket
|
|
43
44
|
│ ├── types/ # Zod schemas and TypeScript types
|
|
@@ -233,7 +234,7 @@ Suite and case file names are derived from the `name` field via `toSlug()` (lowe
|
|
|
233
234
|
|
|
234
235
|
### Self-healing cascade
|
|
235
236
|
|
|
236
|
-
When a test step fails, the engine tries up to
|
|
237
|
+
When a test step fails, the engine tries up to 5 healing levels in order:
|
|
237
238
|
|
|
238
239
|
| Level | Strategy |
|
|
239
240
|
|-------|---------|
|
|
@@ -242,7 +243,6 @@ When a test step fails, the engine tries up to 6 healing levels in order:
|
|
|
242
243
|
| 3 | Multi-Selector — try alternate selectors (ID, text, role, aria) |
|
|
243
244
|
| 4 | Visual/SoM — screenshot + AI visual analysis |
|
|
244
245
|
| 5 | Decompose — break step into smaller sub-actions |
|
|
245
|
-
| 6 | Manual — flag for human review |
|
|
246
246
|
|
|
247
247
|
Healed steps are saved as `pending` events in `results/healing/pending.json` for human review via `npx assuremind apply-healing` or the Studio Healing page.
|
|
248
248
|
|
|
@@ -250,5 +250,6 @@ Healed steps are saved as `pending` events in `results/healing/pending.json` for
|
|
|
250
250
|
|
|
251
251
|
1. **Template engine** — recognises common patterns (navigate, click, fill, etc.) and uses zero-cost templates.
|
|
252
252
|
2. **Code cache** — SHA-based cache keyed on `(normalised instruction, url pattern)`. Cache persists to `results/code-cache.json`.
|
|
253
|
-
3. **
|
|
254
|
-
4. **
|
|
253
|
+
3. **MCP enrichment** — when enabled, the SmartRouter takes a live accessibility snapshot and feeds real page elements to the AI, boosting selector accuracy to ~90-95%.
|
|
254
|
+
4. **Complexity classifier** — routes simple steps to cheap/fast models (tiered mode) and complex steps to capable models.
|
|
255
|
+
5. **Batch generation** — multiple empty steps sent in one API call.
|
package/LICENSE
CHANGED
|
@@ -1,21 +1,21 @@
|
|
|
1
|
-
MIT License
|
|
2
|
-
|
|
3
|
-
Copyright (c) 2026 Deepak Hiremath
|
|
4
|
-
|
|
5
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
-
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
-
in the Software without restriction, including without limitation the rights
|
|
8
|
-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
-
copies of the Software, and to permit persons to do so, subject to the
|
|
10
|
-
following conditions:
|
|
11
|
-
|
|
12
|
-
The above copyright notice and this permission notice shall be included in all
|
|
13
|
-
copies or substantial portions of the Software.
|
|
14
|
-
|
|
15
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
-
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
-
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Deepak Hiremath
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to do so, subject to the
|
|
10
|
+
following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
21
|
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -7,24 +7,24 @@
|
|
|
7
7
|
[](https://nodejs.org)
|
|
8
8
|
[](https://playwright.dev)
|
|
9
9
|
|
|
10
|
+
Describe tests in plain English. AI generates Playwright code. Run anywhere.
|
|
11
|
+
|
|
10
12
|
---
|
|
11
13
|
|
|
12
|
-
##
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
- **File-based storage** — plain JSON, fully Git-friendly, no database required
|
|
27
|
-
- **Cross-platform** — macOS, Linux, Windows
|
|
14
|
+
## Why Assuremind?
|
|
15
|
+
|
|
16
|
+
| Capability | What it does |
|
|
17
|
+
|-----------|-------------|
|
|
18
|
+
| **Zero coding** | Write steps in plain English — AI generates Playwright code automatically |
|
|
19
|
+
| **MCP-sighted generation** | AI sees real page elements via Playwright MCP accessibility snapshots (~90-95% accuracy) |
|
|
20
|
+
| **3 suite types** | **UI** (browser automation) · **API** (HTTP tests) · **Audit** (Playwright + Lighthouse) |
|
|
21
|
+
| **5-level self-healing** | Broken selectors are auto-fixed by AI during runs — smart retry → AI regen → multi-selector → visual → decompose |
|
|
22
|
+
| **12 AI providers** | Anthropic · OpenAI · Gemini · Groq · DeepSeek · Together · Qwen · Perplexity · Ollama · Bedrock · Azure OpenAI · Custom |
|
|
23
|
+
| **Device emulation** | iPhone, Pixel, iPad, Galaxy — full Playwright device descriptors from UI or `--device` CLI flag |
|
|
24
|
+
| **Studio UI** | Browser-based editor, run dashboard, reports, healing review, git control center — with dark mode |
|
|
25
|
+
| **Cost-optimised** | Template engine + code cache handle ~80% of steps with zero AI calls |
|
|
26
|
+
| **CI-ready** | `npx assuremind run --all --ci` — exit code 0/1, works with any pipeline |
|
|
27
|
+
| **File-based** | Plain JSON storage, fully Git-friendly, no database |
|
|
28
28
|
|
|
29
29
|
---
|
|
30
30
|
|
|
@@ -32,356 +32,139 @@
|
|
|
32
32
|
|
|
33
33
|
```bash
|
|
34
34
|
npm install assuremind
|
|
35
|
-
npx assuremind init
|
|
36
|
-
npx assuremind studio
|
|
35
|
+
npx assuremind init # folders, config, Playwright browsers
|
|
36
|
+
npx assuremind studio # opens http://localhost:4400
|
|
37
37
|
```
|
|
38
38
|
|
|
39
39
|
> **Requirements:** Node.js >= 18 · macOS / Linux / Windows
|
|
40
40
|
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
## Configuration
|
|
41
|
+
### Configure AI Provider
|
|
44
42
|
|
|
45
|
-
|
|
43
|
+
Edit `.env` — pick one provider:
|
|
46
44
|
|
|
47
45
|
```bash
|
|
48
|
-
#
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
ANTHROPIC_API_KEY=sk-ant-...
|
|
53
|
-
ANTHROPIC_MODEL=claude-sonnet-4-20250514 # optional
|
|
54
|
-
|
|
55
|
-
# OpenAI (GPT)
|
|
56
|
-
AI_PROVIDER=openai
|
|
57
|
-
OPENAI_API_KEY=sk-...
|
|
58
|
-
OPENAI_MODEL=gpt-4o # optional
|
|
59
|
-
|
|
60
|
-
# Google (Gemini)
|
|
61
|
-
AI_PROVIDER=google
|
|
62
|
-
GOOGLE_API_KEY=AIza...
|
|
63
|
-
GOOGLE_MODEL=gemini-2.5-flash # optional
|
|
64
|
-
|
|
65
|
-
# Groq (fast inference)
|
|
66
|
-
AI_PROVIDER=groq
|
|
67
|
-
GROQ_API_KEY=gsk_...
|
|
68
|
-
GROQ_MODEL=llama-3.3-70b-versatile # optional
|
|
69
|
-
|
|
70
|
-
# DeepSeek
|
|
71
|
-
AI_PROVIDER=deepseek
|
|
72
|
-
DEEPSEEK_API_KEY=sk-...
|
|
73
|
-
|
|
74
|
-
# Together AI
|
|
75
|
-
AI_PROVIDER=together
|
|
76
|
-
TOGETHER_API_KEY=...
|
|
77
|
-
|
|
78
|
-
# Perplexity
|
|
79
|
-
AI_PROVIDER=perplexity
|
|
80
|
-
PERPLEXITY_API_KEY=pplx-...
|
|
81
|
-
|
|
82
|
-
# Qwen (Alibaba)
|
|
83
|
-
AI_PROVIDER=qwen
|
|
84
|
-
QWEN_API_KEY=...
|
|
85
|
-
|
|
86
|
-
# Ollama (local)
|
|
87
|
-
AI_PROVIDER=ollama
|
|
88
|
-
OLLAMA_BASE_URL=http://localhost:11434 # optional
|
|
89
|
-
OLLAMA_MODEL=llama3.3 # optional
|
|
90
|
-
|
|
91
|
-
# AWS Bedrock
|
|
92
|
-
AI_PROVIDER=bedrock
|
|
93
|
-
AWS_ACCESS_KEY_ID=...
|
|
94
|
-
AWS_SECRET_ACCESS_KEY=...
|
|
95
|
-
AWS_REGION=us-east-1
|
|
96
|
-
|
|
97
|
-
# Azure OpenAI
|
|
98
|
-
AI_PROVIDER=azure-openai
|
|
99
|
-
AZURE_OPENAI_API_KEY=...
|
|
100
|
-
AZURE_OPENAI_ENDPOINT=https://my-resource.openai.azure.com
|
|
101
|
-
AZURE_OPENAI_DEPLOYMENT=my-deployment
|
|
102
|
-
|
|
103
|
-
# Custom OpenAI-compatible endpoint
|
|
104
|
-
AI_PROVIDER=custom
|
|
105
|
-
CUSTOM_API_KEY=...
|
|
106
|
-
CUSTOM_BASE_URL=https://my-endpoint.com/v1
|
|
107
|
-
CUSTOM_MODEL=my-model
|
|
46
|
+
# Anthropic # OpenAI # Google
|
|
47
|
+
AI_PROVIDER=anthropic AI_PROVIDER=openai AI_PROVIDER=google
|
|
48
|
+
ANTHROPIC_API_KEY=sk-ant-... OPENAI_API_KEY=sk-... GOOGLE_API_KEY=AIza...
|
|
49
|
+
ANTHROPIC_MODEL=claude-sonnet.. OPENAI_MODEL=gpt-4o GOOGLE_MODEL=gemini-2.5-pro
|
|
108
50
|
```
|
|
109
51
|
|
|
110
|
-
|
|
52
|
+
See `.env.example` for all 12 providers including Gemini, OpenAI, Anthropic, Ollama (local/free), etc.
|
|
111
53
|
|
|
112
54
|
---
|
|
113
55
|
|
|
114
|
-
## CLI
|
|
115
|
-
|
|
116
|
-
### `npx assuremind init`
|
|
117
|
-
|
|
118
|
-
Initialises a project — creates folders, `.env`, config files, and installs Playwright browsers.
|
|
119
|
-
|
|
120
|
-
```bash
|
|
121
|
-
npx assuremind init
|
|
122
|
-
npx assuremind init --skip-playwright # skip browser installation
|
|
123
|
-
```
|
|
124
|
-
|
|
125
|
-
### `npx assuremind studio`
|
|
126
|
-
|
|
127
|
-
Starts the web UI at `http://localhost:4400`.
|
|
128
|
-
|
|
129
|
-
```bash
|
|
130
|
-
npx assuremind studio
|
|
131
|
-
npx assuremind studio --port 5000
|
|
132
|
-
npx assuremind studio --no-open # don't auto-open browser
|
|
133
|
-
```
|
|
134
|
-
|
|
135
|
-
### `npx assuremind run`
|
|
136
|
-
|
|
137
|
-
Runs tests from the command line. Filters are combinable with AND logic.
|
|
56
|
+
## CLI
|
|
138
57
|
|
|
139
58
|
```bash
|
|
140
|
-
#
|
|
141
|
-
npx assuremind run --
|
|
142
|
-
|
|
143
|
-
#
|
|
144
|
-
npx assuremind
|
|
145
|
-
npx assuremind
|
|
146
|
-
npx assuremind
|
|
147
|
-
|
|
148
|
-
# Run a suite (case-insensitive partial match)
|
|
149
|
-
npx assuremind run --suite "Login Tests"
|
|
150
|
-
|
|
151
|
-
# Run by tag
|
|
152
|
-
npx assuremind run --tag smoke
|
|
153
|
-
|
|
154
|
-
# Run a single test (case-insensitive partial match)
|
|
155
|
-
npx assuremind run --test "User can log in with valid credentials"
|
|
156
|
-
|
|
157
|
-
# Combine filters
|
|
158
|
-
npx assuremind run --type audit --tag regression
|
|
159
|
-
npx assuremind run --suite "Orange HRM" --tag smoke
|
|
160
|
-
npx assuremind run --type audit --test "Login Page"
|
|
161
|
-
|
|
162
|
-
# Device emulation (mobile/tablet)
|
|
163
|
-
npx assuremind run --all --device "iPhone 15 Pro" --browser chromium
|
|
164
|
-
npx assuremind run --tag smoke --device "Pixel 7" --browser chromium
|
|
165
|
-
npx assuremind run --all --device "iPad Pro 11" --browser webkit
|
|
166
|
-
|
|
167
|
-
# With options
|
|
168
|
-
npx assuremind run --all \
|
|
169
|
-
--browser chromium firefox \
|
|
170
|
-
--parallel 4 \
|
|
171
|
-
--headed \
|
|
172
|
-
--ci \
|
|
173
|
-
--no-healing \
|
|
174
|
-
--reporter allure json
|
|
59
|
+
npx assuremind run --all # run everything
|
|
60
|
+
npx assuremind run --type ui --tag smoke # filter by type + tag
|
|
61
|
+
npx assuremind run --suite "Login" --browser chromium # run a suite
|
|
62
|
+
npx assuremind run --all --device "iPhone 15 Pro" --ci # mobile + CI mode
|
|
63
|
+
npx assuremind generate --story "User resets password" # AI generates full suite
|
|
64
|
+
npx assuremind apply-healing --yes # accept all healed selectors
|
|
65
|
+
npx assuremind validate # check config health
|
|
66
|
+
npx assuremind doctor # system diagnostics
|
|
175
67
|
```
|
|
176
68
|
|
|
177
69
|
| Flag | Description |
|
|
178
70
|
|------|-------------|
|
|
179
71
|
| `--all` | Run every suite |
|
|
180
|
-
| `--type <
|
|
181
|
-
| `--suite <name>` |
|
|
182
|
-
| `--tag <tag>` |
|
|
183
|
-
| `--
|
|
184
|
-
| `--browser <list>` |
|
|
185
|
-
| `--
|
|
186
|
-
| `--env <name>` | Variable environment: `dev` `staging` `prod` |
|
|
187
|
-
| `--parallel <n>` | Concurrent workers |
|
|
72
|
+
| `--type <ui\|api\|audit>` | Filter by suite type |
|
|
73
|
+
| `--suite <name>` | Partial name match |
|
|
74
|
+
| `--tag <tag>` | Filter by tag |
|
|
75
|
+
| `--device <name>` | Emulate device (e.g. `"iPhone 15 Pro"`, `"Pixel 7"`) |
|
|
76
|
+
| `--browser <list>` | `chromium` `firefox` `webkit` |
|
|
77
|
+
| `--ci` | CI mode — exit code reflects pass/fail |
|
|
188
78
|
| `--headed` | Show browser window |
|
|
189
|
-
| `--
|
|
190
|
-
| `--no-healing` | Disable self-healing for this run |
|
|
191
|
-
| `--reporter <list>` | `allure` `html` `json` |
|
|
192
|
-
|
|
193
|
-
### `npx assuremind generate`
|
|
194
|
-
|
|
195
|
-
Generates a test suite from a plain-English user story using AI.
|
|
196
|
-
|
|
197
|
-
```bash
|
|
198
|
-
npx assuremind generate \
|
|
199
|
-
--story "User logs in with valid credentials and sees the dashboard"
|
|
200
|
-
|
|
201
|
-
npx assuremind generate \
|
|
202
|
-
--story-file ./stories/checkout.txt \
|
|
203
|
-
--suite "Checkout Flow"
|
|
204
|
-
```
|
|
205
|
-
|
|
206
|
-
### `npx assuremind apply-healing`
|
|
207
|
-
|
|
208
|
-
Reviews and applies self-healing suggestions to test files.
|
|
209
|
-
|
|
210
|
-
```bash
|
|
211
|
-
# Interactive review
|
|
212
|
-
npx assuremind apply-healing
|
|
213
|
-
|
|
214
|
-
# Accept all pending heals without prompting
|
|
215
|
-
npx assuremind apply-healing --yes
|
|
216
|
-
|
|
217
|
-
# Load from a specific report file
|
|
218
|
-
npx assuremind apply-healing --from results/healing/healing-report-<runId>.json
|
|
219
|
-
```
|
|
220
|
-
|
|
221
|
-
### `npx assuremind validate`
|
|
222
|
-
|
|
223
|
-
Validates your config, environment variables, and test files.
|
|
79
|
+
| `--no-healing` | Disable self-healing |
|
|
224
80
|
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
Checks system requirements, AI provider connectivity, and configuration health.
|
|
81
|
+
Full reference → [docs/CLI-REFERENCE.md](docs/CLI-REFERENCE.md)
|
|
228
82
|
|
|
229
83
|
---
|
|
230
84
|
|
|
231
85
|
## Studio UI
|
|
232
86
|
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
| Page | Description |
|
|
236
|
-
|------|-------------|
|
|
237
|
-
| **Dashboard** | Run health overview, live progress, recent results, pass-rate trend sparklines |
|
|
238
|
-
| **Smart Tests** | Paste a user story or Jira link — AI creates a full test suite |
|
|
239
|
-
| **Test Editor** | 3-level editor: suites → cases → steps, with inline AI code generation |
|
|
240
|
-
| **Run Config** | Configure browsers, device emulation, parallelism and launch runs from the browser |
|
|
241
|
-
| **Reports** | Run history, pass/fail drill-down, Lighthouse score tabs (⚡ Speed, ♿ A11y, 🔍 SEO) for Audit suites, device badge per case, Allure report link |
|
|
242
|
-
| **Variables** | Manage `{{VARIABLE_NAME}}` tokens across all tests and environments |
|
|
243
|
-
| **Self-Healing** | Review and accept / reject AI-generated selector fixes |
|
|
244
|
-
| **Step Library** | Reusable step snippets shared across suites |
|
|
245
|
-
| **Docs** | Built-in documentation, CLI reference, and feature guides |
|
|
246
|
-
| **Git Control Center** | Branch management, AI commit messages, push/pull, conflict resolution |
|
|
247
|
-
| **Settings** | Environment management, browsers, healing thresholds, capture settings |
|
|
248
|
-
|
|
249
|
-
---
|
|
87
|
+
Start with `npx assuremind studio` — opens at `http://localhost:4400`.
|
|
250
88
|
|
|
251
|
-
|
|
89
|
+
**Dashboard** · **Smart Tests** · **Test Editor** · **Run Config** · **Reports** · **Variables** · **Self-Healing** · **Step Library** · **Git Control** · **Settings** · **Docs**
|
|
252
90
|
|
|
253
|
-
|
|
91
|
+
Full walkthrough → [docs/STUDIO.md](docs/STUDIO.md)
|
|
254
92
|
|
|
255
|
-
|
|
93
|
+
---
|
|
256
94
|
|
|
257
|
-
|
|
95
|
+
## MCP Integration
|
|
258
96
|
|
|
259
|
-
|
|
260
|
-
npx assuremind run --all --device "iPhone 15 Pro" --browser chromium
|
|
261
|
-
npx assuremind run --all --device "Pixel 7" --browser chromium
|
|
262
|
-
npx assuremind run --all --device "iPad Pro 11" --browser webkit
|
|
263
|
-
```
|
|
97
|
+
AI sees **real page elements** during code generation via the official `@playwright/mcp` server. Enabled by default.
|
|
264
98
|
|
|
265
|
-
|
|
99
|
+
| Mode | Accuracy | Latency | Config |
|
|
100
|
+
|------|----------|---------|--------|
|
|
101
|
+
| Blind (MCP off) | ~50-70% | Fastest | `mcp.enabled: false` |
|
|
102
|
+
| **Snapshot-driven** | ~90-95% | +2-5s first page | `mcp.enabled: true` (default) |
|
|
103
|
+
| Act-then-script | ~98-100% | +5-10s/step | `mcp.actThenScript: true` |
|
|
266
104
|
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
| iPhone 15 | 393 × 852 | 3 | Chromium |
|
|
271
|
-
| Pixel 7 | 412 × 915 | 2.625 | Chromium |
|
|
272
|
-
| Galaxy S9+ | 320 × 658 | 4.5 | Chromium |
|
|
273
|
-
| iPad Pro 11 | 834 × 1194 | 2 | WebKit |
|
|
274
|
-
| iPad Mini | 768 × 1024 | 2 | WebKit |
|
|
275
|
-
|
|
276
|
-
> **Note:** Device emulation requires Chromium for Android devices and WebKit for Apple devices. A warning is shown in the UI when an incompatible combination is selected.
|
|
105
|
+
- MCP is **only used during code generation** — test execution is never affected
|
|
106
|
+
- Silent fallback — if MCP fails, generation continues blindly without error
|
|
107
|
+
- Configure in **Settings → MCP Integration** or `autotest.config.json`
|
|
277
108
|
|
|
278
109
|
---
|
|
279
110
|
|
|
280
111
|
## Self-Healing
|
|
281
112
|
|
|
282
|
-
When
|
|
283
|
-
|
|
284
|
-
| Level | Strategy | AI Cost |
|
|
285
|
-
|-------|----------|---------|
|
|
286
|
-
| 1 | Smart Retry — wait + retry with backoff | None |
|
|
287
|
-
| 2 | AI Regeneration — AI rewrites the Playwright code | Yes |
|
|
288
|
-
| 3 | Multi-Selector — try alternate selectors (ID, text, role, aria) | Yes |
|
|
289
|
-
| 4 | Visual/SoM — screenshot + AI visual analysis | Yes |
|
|
290
|
-
| 5 | Decompose — break step into smaller sub-actions | Yes |
|
|
291
|
-
| 6 | Manual — flag for human review | None |
|
|
113
|
+
When your app's UI changes (button renamed, element moved, DOM restructured), tests break. Instead of failing immediately, Assuremind automatically attempts to fix the broken selector through a 5-level cascade — **fully automated, no manual intervention needed**:
|
|
292
114
|
|
|
293
|
-
|
|
115
|
+
| Level | What happens | Example | AI Cost |
|
|
116
|
+
|-------|-------------|---------|---------|
|
|
117
|
+
| 1 | **Smart retry** — waits for the element with exponential backoff | Element was loading slowly; retry finds it after 2s | Free |
|
|
118
|
+
| 2 | **AI regeneration** — AI rewrites the Playwright code using current page context | `#login-btn` removed → AI generates `page.getByRole('button', { name: 'Sign In' })` | 1 call |
|
|
119
|
+
| 3 | **Multi-selector** — AI generates 5 alternative selector strategies, tries each | Tries role → label → placeholder → text → CSS until one works | 1 call |
|
|
120
|
+
| 4 | **Visual analysis** — takes a screenshot, AI visually locates the element | Button has no text/role but AI sees it in the screenshot | 1 call |
|
|
121
|
+
| 5 | **Decompose** — breaks the failing step into 3-5 simpler micro-actions | "Fill login form and submit" → separate fill email + fill password + click submit | 1 call |
|
|
294
122
|
|
|
295
|
-
|
|
123
|
+
If all 5 levels fail, the step is marked **failed** and saved to the healing report for your review.
|
|
296
124
|
|
|
297
|
-
|
|
125
|
+
### How you use it
|
|
298
126
|
|
|
299
|
-
|
|
127
|
+
1. **During test runs** — healing happens automatically. If Level 2 fixes a broken `#login-btn`, your test **passes** and continues.
|
|
128
|
+
2. **After the run** — healed selectors are saved as **pending suggestions** (not auto-applied to source files).
|
|
129
|
+
3. **Review & accept** — in Studio → **Self-Healing** page, or from CLI:
|
|
130
|
+
```bash
|
|
131
|
+
npx assuremind apply-healing # interactive review: accept/reject each fix
|
|
132
|
+
npx assuremind apply-healing --yes # accept all in CI
|
|
133
|
+
```
|
|
134
|
+
4. **Accepted fixes** are written back to your `.test.json` files — next run uses the healed code permanently.
|
|
300
135
|
|
|
301
|
-
|
|
302
|
-
// variables/global.json
|
|
303
|
-
{
|
|
304
|
-
"BASE_URL": "http://localhost:3000",
|
|
305
|
-
"ADMIN_EMAIL": "admin@example.com"
|
|
306
|
-
}
|
|
307
|
-
```
|
|
308
|
-
|
|
309
|
-
Step instruction: `Navigate to {{BASE_URL}}/login and enter {{ADMIN_EMAIL}}`
|
|
310
|
-
|
|
311
|
-
Secret variables (marked with `"secret": true`) are masked in logs and reports.
|
|
136
|
+
> **CI/CD tip:** Add `npx assuremind apply-healing --yes` as a post-test step so healed selectors are committed back automatically. Enable `healing.autoPR` in Settings to auto-create a GitHub PR with the fixes.
|
|
312
137
|
|
|
313
138
|
---
|
|
314
139
|
|
|
315
|
-
## CI/CD
|
|
316
|
-
|
|
317
|
-
### GitHub Actions
|
|
140
|
+
## CI/CD
|
|
318
141
|
|
|
319
142
|
```yaml
|
|
320
|
-
|
|
143
|
+
# GitHub Actions
|
|
144
|
+
- name: Run tests
|
|
321
145
|
env:
|
|
322
|
-
AI_PROVIDER:
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
# Mobile run
|
|
328
|
-
- name: Run mobile tests
|
|
329
|
-
run: |
|
|
330
|
-
npx assuremind run --all --browser chromium --device "iPhone 15 Pro" --ci
|
|
146
|
+
AI_PROVIDER: google
|
|
147
|
+
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
|
|
148
|
+
GOOGLE_MODEL: gemini-2.5-pro
|
|
149
|
+
run: npx assuremind run --all --ci
|
|
331
150
|
```
|
|
332
151
|
|
|
333
|
-
|
|
152
|
+
Also supports **GitLab CI** and **Jenkins** — or use the built-in **CI Config Generator** in Studio (Run Config → Generate CI Config).
|
|
334
153
|
|
|
335
|
-
|
|
336
|
-
test:
|
|
337
|
-
image: node:20
|
|
338
|
-
script:
|
|
339
|
-
- npx assuremind run --all --browser chromium --parallel 2 --ci
|
|
340
|
-
variables:
|
|
341
|
-
AI_PROVIDER: anthropic
|
|
342
|
-
ANTHROPIC_API_KEY: $ANTHROPIC_API_KEY
|
|
343
|
-
```
|
|
344
|
-
|
|
345
|
-
### Jenkins
|
|
346
|
-
|
|
347
|
-
```groovy
|
|
348
|
-
stage('Test') {
|
|
349
|
-
environment {
|
|
350
|
-
AI_PROVIDER = 'anthropic'
|
|
351
|
-
ANTHROPIC_API_KEY = credentials('anthropic-api-key')
|
|
352
|
-
}
|
|
353
|
-
steps {
|
|
354
|
-
sh 'npx assuremind run --all --browser chromium --parallel 2 --ci'
|
|
355
|
-
}
|
|
356
|
-
}
|
|
357
|
-
```
|
|
358
|
-
|
|
359
|
-
> **Tip:** Use the **CI Config Generator** in Studio (**Run Config** page → *Generate CI Config*) to produce a ready-to-paste config for your platform, pre-filled with your current browser, parallel, and device settings.
|
|
360
|
-
|
|
361
|
-
Exit code `0` = all tests passed · `1` = at least one test failed.
|
|
362
|
-
|
|
363
|
-
---
|
|
364
|
-
|
|
365
|
-
## Security
|
|
366
|
-
|
|
367
|
-
- Generated Playwright code runs inside a sandboxed `new Function('page', 'context', 'expect', code)` — only `page`, `context`, and `expect` are available
|
|
368
|
-
- Secret variables are never sent to AI providers, never logged, never included in reports
|
|
369
|
-
- All generated code is validated against a forbidden-pattern list before execution
|
|
370
|
-
- Atomic file writes prevent partially-written results on crash
|
|
154
|
+
Exit code `0` = all passed · `1` = failures.
|
|
371
155
|
|
|
372
156
|
---
|
|
373
157
|
|
|
374
|
-
##
|
|
375
|
-
|
|
376
|
-
Enable tiered mode to use a cheaper/faster model for simple steps and a more capable model for complex ones:
|
|
377
|
-
|
|
378
|
-
```bash
|
|
379
|
-
AI_TIERED_ENABLED=true
|
|
380
|
-
AI_TIERED_FAST_PROVIDER=groq
|
|
381
|
-
AI_TIERED_FAST_MODEL=llama-3.1-8b-instant
|
|
382
|
-
```
|
|
158
|
+
## Documentation
|
|
383
159
|
|
|
384
|
-
|
|
160
|
+
| Resource | Location |
|
|
161
|
+
|----------|----------|
|
|
162
|
+
| Getting started | [docs/GETTING-STARTED.md](docs/GETTING-STARTED.md) |
|
|
163
|
+
| Studio walkthrough | [docs/STUDIO.md](docs/STUDIO.md) |
|
|
164
|
+
| CLI reference | [docs/CLI-REFERENCE.md](docs/CLI-REFERENCE.md) |
|
|
165
|
+
| Contributing | [CONTRIBUTING.md](CONTRIBUTING.md) |
|
|
166
|
+
| All AI providers | `.env.example` |
|
|
167
|
+
| Built-in docs | Studio → **Docs** page |
|
|
385
168
|
|
|
386
169
|
---
|
|
387
170
|
|
|
@@ -391,4 +174,4 @@ MIT — see [LICENSE](LICENSE) for details.
|
|
|
391
174
|
|
|
392
175
|
---
|
|
393
176
|
|
|
394
|
-
*Built by [Deepak Hiremath](https://
|
|
177
|
+
*Built by [Deepak Hiremath](https://www.linkedin.com/in/deepak-hiremath-0017937a/)*
|