prompttest-mobile 1.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CONTRIBUTING.md +80 -0
- package/LICENSE +42 -0
- package/LICENSES.md +81 -0
- package/README.md +526 -0
- package/SECURITY.md +56 -0
- package/TERMS.md +138 -0
- package/dist/bin/prompttest.d.ts +2 -0
- package/dist/bin/prompttest.js +1418 -0
- package/dist/bin/server.d.ts +1 -0
- package/dist/constants/commands.d.ts +125 -0
- package/dist/engine.bundle.js +1039 -0
- package/dist/index.d.ts +143 -0
- package/dist/index.js +1039 -0
- package/dist/lib/adaptive-timing.d.ts +55 -0
- package/dist/lib/adb-provisioner.d.ts +55 -0
- package/dist/lib/adb.d.ts +448 -0
- package/dist/lib/ai/heuristic-resolver.d.ts +19 -0
- package/dist/lib/ai/index.d.ts +16 -0
- package/dist/lib/ai/llm-provider.d.ts +40 -0
- package/dist/lib/ai/types.d.ts +64 -0
- package/dist/lib/baseline.d.ts +159 -0
- package/dist/lib/benchmark.d.ts +100 -0
- package/dist/lib/checkpoint.d.ts +61 -0
- package/dist/lib/ci.d.ts +49 -0
- package/dist/lib/config-loader.d.ts +87 -0
- package/dist/lib/config.d.ts +136 -0
- package/dist/lib/crawler.d.ts +335 -0
- package/dist/lib/data-loader.d.ts +53 -0
- package/dist/lib/dfs-engine.d.ts +149 -0
- package/dist/lib/dictionary.d.ts +41 -0
- package/dist/lib/doctor.d.ts +45 -0
- package/dist/lib/driver-interface.d.ts +50 -0
- package/dist/lib/enterprise.d.ts +71 -0
- package/dist/lib/errors.d.ts +68 -0
- package/dist/lib/explorer.d.ts +165 -0
- package/dist/lib/feedback.d.ts +35 -0
- package/dist/lib/form-filler.d.ts +101 -0
- package/dist/lib/ios-driver.d.ts +38 -0
- package/dist/lib/jail-guard.d.ts +59 -0
- package/dist/lib/license.d.ts +51 -0
- package/dist/lib/live-server.d.ts +52 -0
- package/dist/lib/lock.d.ts +30 -0
- package/dist/lib/logger.d.ts +68 -0
- package/dist/lib/memory.d.ts +259 -0
- package/dist/lib/patterns.d.ts +202 -0
- package/dist/lib/profiler.d.ts +75 -0
- package/dist/lib/prompt-runner.d.ts +101 -0
- package/dist/lib/quiescence.d.ts +44 -0
- package/dist/lib/recorder.d.ts +155 -0
- package/dist/lib/repl.d.ts +29 -0
- package/dist/lib/reporter.d.ts +269 -0
- package/dist/lib/runner-utils.d.ts +316 -0
- package/dist/lib/scaffold.d.ts +53 -0
- package/dist/lib/step-handlers.d.ts +390 -0
- package/dist/lib/triage.d.ts +50 -0
- package/dist/lib/wizard.d.ts +42 -0
- package/dist/lib/zip-util.d.ts +36 -0
- package/docs/ARCHITECTURE.md +154 -0
- package/docs/CLI_CONTRACT.md +131 -0
- package/docs/CLI_STUDIO_CONTRACT.md +123 -0
- package/docs/PERFORMANCE_BASELINE.md +71 -0
- package/docs/PRODUCT_STATUS.md +56 -0
- package/docs/USER_MANUAL.md +764 -0
- package/package.json +66 -0
|
@@ -0,0 +1,764 @@
|
|
|
1
|
+
# 📖 PromptTest — Comprehensive User Manual & Reference Guide
|
|
2
|
+
|
|
3
|
+
Welcome to the **PromptTest** User Manual! This guide provides detailed documentation, syntax references, real-world examples, and best practices for writing and executing autonomous mobile QA tests in plain English.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## 📑 Table of Contents
|
|
8
|
+
|
|
9
|
+
1. [Introduction & Architecture](#1-introduction--architecture)
|
|
10
|
+
2. [Installation & Setup](#2-installation--setup)
|
|
11
|
+
3. [CLI Command Reference](#3-cli-command-reference)
|
|
12
|
+
4. [PromptTest Syntax Specification](#4-prompttest-syntax-specification)
|
|
13
|
+
- [Tapping & Gestures](#tapping--gestures)
|
|
14
|
+
- [Text Entry & Form Inputs](#text-entry--form-inputs)
|
|
15
|
+
- [Assertions & Verifications](#assertions--verifications)
|
|
16
|
+
- [Navigation, Scrolling & Swiping](#navigation-scrolling--swiping)
|
|
17
|
+
- [App Lifecycle & Permissions](#app-lifecycle--permissions)
|
|
18
|
+
5. [Dynamic & Conditional Testing](#5-dynamic--conditional-testing)
|
|
19
|
+
- [Regular Expression / Pattern Matching](#regular-expression--pattern-matching)
|
|
20
|
+
- [Multi-Alternative (OR) Matching](#multi-alternative-or-matching)
|
|
21
|
+
- [Optional / Conditional Actions](#optional--conditional-actions)
|
|
22
|
+
- [Dynamic Variables & Generators](#dynamic-variables--generators)
|
|
23
|
+
6. [Autonomous Record & Replay](#6-autonomous-record--replay)
|
|
24
|
+
7. [Autonomous Exploration & State-Graph Crawler (Mode 2)](#7-autonomous-exploration--state-graph-crawler-mode-2)
|
|
25
|
+
8. [Self-Healing & Resilience Engine](#8-self-healing--resilience-engine)
|
|
26
|
+
9. [Multi-Device Matrix & CI/CD Pipelines](#9-multi-device-matrix--cicd-pipelines)
|
|
27
|
+
10. [Advanced Features & SDK](#10-advanced-features--sdk)
|
|
28
|
+
|
|
29
|
+
- [Checkpointing & Resume](#checkpointing--resume)
|
|
30
|
+
- [Visual Baselines](#visual-baselines)
|
|
31
|
+
- [Custom Reporters](#custom-reporters)
|
|
32
|
+
- [APK Auto-Install](#apk-auto-install)
|
|
33
|
+
- [Device Locking](#device-locking)
|
|
34
|
+
- [Programmatic SDK](#programmatic-sdk)
|
|
35
|
+
|
|
36
|
+
11. [Real-World Domain Recipes](#11-real-world-domain-recipes)
|
|
37
|
+
|
|
38
|
+
- [EdTech & Coaching Portals](#edtech--coaching-portals)
|
|
39
|
+
- [E-Commerce & Food Delivery](#e-commerce--food-delivery)
|
|
40
|
+
- [FinTech & Banking](#fintech--banking)
|
|
41
|
+
- [Social & Messaging](#social--messaging)
|
|
42
|
+
|
|
43
|
+
12. [Mode Expectations, Capabilities & Limitations](#12-mode-expectations-capabilities--limitations)
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## 1. Introduction & Architecture
|
|
48
|
+
|
|
49
|
+
**PromptTest** is an AI-native autonomous mobile testing framework for Android. It enables developers, QA engineers, and product teams to write executable test specs in plain English without writing fragile XPath selectors or maintaining complex Appium setups.
|
|
50
|
+
|
|
51
|
+
### Key Architecture Components:
|
|
52
|
+
|
|
53
|
+
- **`AndroidDriver` (ADB Layer):** Direct, zero-latency communication with physical Android devices and emulators over USB or Wi-Fi.
|
|
54
|
+
- **`HierarchyCrawler` (Visual UI Engine):** Extracts, cleans, and indexes on-screen Android view trees.
|
|
55
|
+
- **`PromptRunner` (Execution Brain):** Parses plain-English instructions, performs label-to-input pairing, auto-scrolls viewports, and executes resilient assertions.
|
|
56
|
+
- **`MemoryEngine` (Autonomous Knowledge Graph):** Learns screen aliases and component identifiers across test runs for instant 0ms lookups.
|
|
57
|
+
- **`AutonomousRecorder` (Interactive Record-and-Play):** Live-monitors screen interactions and auto-generates test specs.
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## 2. Installation & Setup
|
|
62
|
+
|
|
63
|
+
### Requirements
|
|
64
|
+
|
|
65
|
+
- **Node.js**: v18.0.0 or higher
|
|
66
|
+
- **Android SDK Platform Tools**: `adb` must be available in your system `PATH`.
|
|
67
|
+
- **Target Device**: Physical Android device (with USB debugging enabled) or Android Emulator.
|
|
68
|
+
|
|
69
|
+
### Running via npx (Zero Installation)
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
npx prompttest devices
|
|
73
|
+
npx prompttest run specs/my_test.txt
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
### Installing Locally in Your Repository
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
npm install --save-dev prompttest
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Add scripts to your `package.json`:
|
|
83
|
+
|
|
84
|
+
```json
|
|
85
|
+
{
|
|
86
|
+
"scripts": {
|
|
87
|
+
"test:mobile": "prompttest run specs/regression_suite.txt",
|
|
88
|
+
"test:fresh": "prompttest run specs/regression_suite.txt --fresh",
|
|
89
|
+
"test:matrix": "prompttest run specs/regression_suite.txt --all-devices"
|
|
90
|
+
}
|
|
91
|
+
}
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
## 3. CLI Command Reference
|
|
97
|
+
|
|
98
|
+
| Command | Description | Example |
|
|
99
|
+
| --------------------------------- | ----------------------------------------------- | ------------------------------------------ |
|
|
100
|
+
| `prompttest devices` | Scans and lists all connected Android devices | `npx prompttest devices` |
|
|
101
|
+
| `prompttest wifi [ip[:port]]` | Connects to a device wirelessly over Wi-Fi | `npx prompttest wifi 192.168.1.50` |
|
|
102
|
+
| `prompttest disconnect [ip]` | Disconnects wireless device | `npx prompttest disconnect` |
|
|
103
|
+
| `prompttest status` | Inspects foreground application & UI components | `npx prompttest status` |
|
|
104
|
+
| `prompttest record [spec.txt]` | Interactive record-and-play session | `npx prompttest record specs/login.txt` |
|
|
105
|
+
| `prompttest run <spec.txt>` | Executes plain-English test spec (Fail-Fast) | `npx prompttest run specs/teacher.txt` |
|
|
106
|
+
| `prompttest watch <spec.txt>` | Continuously re-runs test upon spec file save | `npx prompttest watch specs/login.txt` |
|
|
107
|
+
| `prompttest doctor` | Runs full workstation & ADB readiness diagnosis | `npx prompttest doctor` |
|
|
108
|
+
| `prompttest repl` | Launches interactive live testing shell | `npx prompttest repl` |
|
|
109
|
+
| `prompttest benchmark` | Profiles ADB hierarchy, screencap, & matching | `npx prompttest benchmark` |
|
|
110
|
+
| `prompttest merge-reports [dir]` | Merges multi-device results into Atto dashboard | `npx prompttest merge-reports output` |
|
|
111
|
+
| `prompttest explore [pkg]` | Autonomous crawler exploration (forms/tabs) | `npx prompttest explore host.exp.exponent` |
|
|
112
|
+
| `prompttest memory [pkg]` | Views learned component aliases & confidence | `npx prompttest memory host.exp.exponent` |
|
|
113
|
+
| `prompttest memory stats [pkg]` | Displays learned knowledge health statistics | `npx prompttest memory stats` |
|
|
114
|
+
| `prompttest memory export [pkg]` | Exports learned memory to a JSON file | `npx prompttest memory export host.exp` |
|
|
115
|
+
| `prompttest memory import <file>` | Imports/merges learned memory from JSON file | `npx prompttest memory import mem.json` |
|
|
116
|
+
| `prompttest memory clear [pkg]` | Clears learned knowledge cache | `npx prompttest memory clear` |
|
|
117
|
+
| `prompttest init-ci` | Generates GitHub Actions CI workflows | `npx prompttest init-ci` |
|
|
118
|
+
|
|
119
|
+
### CLI Execution Flags
|
|
120
|
+
|
|
121
|
+
#### Filtering, Flow Control & Debugging
|
|
122
|
+
|
|
123
|
+
- `--dry-run`: Validates and parses test spec syntax without device interaction.
|
|
124
|
+
- `--list-steps`: Prints a structured table of the parsed step AST to stdout and exits.
|
|
125
|
+
- `--only="<range>"`: Executes only matching step numbers or ranges (e.g. `--only="1-3,5"`).
|
|
126
|
+
- `--tags="<tag>"`: Filters execution to specs matching tagged directives (e.g. `--tags="smoke"`).
|
|
127
|
+
- `--continue` or `--continue-on-error`: Continues running remaining steps even if an upstream step fails.
|
|
128
|
+
- `--fresh` or `--cold-start`: Force-stops and restarts target app before test begins.
|
|
129
|
+
- `--resume`: Resumes test execution from the last recorded checkpoint upon failure.
|
|
130
|
+
- `--var:KEY=VALUE`: Injects inline dynamic variables (e.g. `--var:ROLE=TEACHER`).
|
|
131
|
+
|
|
132
|
+
#### Observability, Reporting & Live Monitoring
|
|
133
|
+
|
|
134
|
+
- `--json`: Outputs machine-readable execution report JSON directly to stdout.
|
|
135
|
+
- `--serve [port]`: Streams live test execution events over HTTP/SSE dashboard (default: `http://localhost:4040`).
|
|
136
|
+
- `--embed-screenshots`: Inlines base64 screenshots into standalone HTML report for zero-asset portability.
|
|
137
|
+
- `--screenshots` or `--all-screenshots`: Captures full screenshots on every step (default: on-failure only).
|
|
138
|
+
- `--video`: Records the entire test session to high-definition MP4 video via native ADB screenrecord.
|
|
139
|
+
- `--reporters=json,html,junit,md`: Generates specific output report formats.
|
|
140
|
+
- `--reporter=<path>`: Loads a custom reporter module implementing the `Reporter` interface.
|
|
141
|
+
|
|
142
|
+
#### Data-Driven & Localization
|
|
143
|
+
|
|
144
|
+
- `--data=<path.csv|json>`: Executes data-driven tests across multiple rows/parameter sets.
|
|
145
|
+
- `--iterations=N`: Repeats the test suite across N iterations or data rows.
|
|
146
|
+
- `--only-row=N`: Executes only the specified 1-indexed data row.
|
|
147
|
+
- `--locale=<code>`: Sets active taxonomy locale dictionary (e.g. `'en'`, `'es'`, `'hi'`).
|
|
148
|
+
|
|
149
|
+
#### AI, Resilience & Visual Regression
|
|
150
|
+
|
|
151
|
+
- `--heal`: Enables automatic element locator self-healing via offline synonyms and heuristic memory.
|
|
152
|
+
- `--no-ai`: Disables third-party LLM providers, forcing offline-only heuristic matching.
|
|
153
|
+
- `--expand`: Previews AI natural-language macro intent expansion without modifying the file.
|
|
154
|
+
- `--expand-apply`: Expands high-level macro steps and writes concrete actions directly to the spec file.
|
|
155
|
+
- `--save-baseline`: Saves screen layout screenshots as visual baseline.
|
|
156
|
+
- `--compare-baseline`: Compares live screens against saved baseline using perceptual diffing.
|
|
157
|
+
- `--baseline-threshold=N`: Sets perceptual diff variance threshold (0.0–1.0, default: `0.02`).
|
|
158
|
+
- `--apk=<path>`: Automatically installs or updates the target APK before test execution if absent.
|
|
159
|
+
- `--all-devices`: Runs test suite concurrently across all connected online devices.
|
|
160
|
+
- `-s <id>` or `--serial=<id>`: Targets a specific connected Android device by serial.
|
|
161
|
+
|
|
162
|
+
### Environment Diagnostic Tool (`prompttest doctor`)
|
|
163
|
+
|
|
164
|
+
Run `prompttest doctor` whenever setting up a new workstation, troubleshooting device connectivity, or configuring a CI runner:
|
|
165
|
+
|
|
166
|
+
- Validates the `adb` binary presence and version in system `PATH`.
|
|
167
|
+
- Confirms USB/Wi-Fi debugging authorization and platform-tools integrity.
|
|
168
|
+
- Verifies physical screen dimensions, orientation, and battery readiness.
|
|
169
|
+
|
|
170
|
+
### Interactive Live Sandbox (`prompttest repl`)
|
|
171
|
+
|
|
172
|
+
Use `prompttest repl` to prototype actions, debug ambiguous locators, or interactively explore screens without writing full spec files:
|
|
173
|
+
|
|
174
|
+
- `inspect`: Dumps actionable buttons, text fields, and tabs on the current screen.
|
|
175
|
+
- `tap '<label>'`: Taps element matching target text or descriptor.
|
|
176
|
+
- `type '<value>' into '<target>'`: Enters text into an input field.
|
|
177
|
+
- `screenshot [path]`: Captures screen evidence instantly.
|
|
178
|
+
|
|
179
|
+
---
|
|
180
|
+
|
|
181
|
+
## 4. PromptTest Syntax Specification
|
|
182
|
+
|
|
183
|
+
PromptTest specs are plain-text files (e.g. `specs/checkout.txt`). Steps can be numbered or written as bullet points. Lines starting with `#` are treated as comments.
|
|
184
|
+
|
|
185
|
+
### Spec Tagging & Categorization
|
|
186
|
+
|
|
187
|
+
You can annotate test specs with tags using `# @tag: <name>` directives at the top of the file:
|
|
188
|
+
|
|
189
|
+
```txt
|
|
190
|
+
# @tag: smoke
|
|
191
|
+
# @tag: regression
|
|
192
|
+
# @tag: payments
|
|
193
|
+
|
|
194
|
+
1. Tap 'Cart'
|
|
195
|
+
2. Tap 'Checkout'
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
Run only specs matching your tag filter:
|
|
199
|
+
|
|
200
|
+
```bash
|
|
201
|
+
npx prompttest run specs/checkout.txt --tags="smoke"
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
### Tapping & Gestures
|
|
205
|
+
|
|
206
|
+
```txt
|
|
207
|
+
# Basic Tap / Click
|
|
208
|
+
Tap 'Sign In'
|
|
209
|
+
Click "Add to Cart"
|
|
210
|
+
Press 'Submit'
|
|
211
|
+
|
|
212
|
+
# Ordinal Tapping (when multiple matching elements exist)
|
|
213
|
+
Tap 1st 'View Details'
|
|
214
|
+
Click 2nd 'Export Report'
|
|
215
|
+
Tap last 'Delete'
|
|
216
|
+
|
|
217
|
+
# Proximity Tapping (targets element near an anchor)
|
|
218
|
+
Tap 'Delete' next to 'Rohan Gupta'
|
|
219
|
+
Click 'Edit' near 'Batch Grade 10'
|
|
220
|
+
|
|
221
|
+
# Long Press
|
|
222
|
+
Long press 'Mid-Term Examination Schedule'
|
|
223
|
+
Press and hold 'Voice Message'
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
### Text Entry & Form Inputs
|
|
227
|
+
|
|
228
|
+
PromptTest automatically associates input fields with their preceding labels and distinguishes between password and email/text fields.
|
|
229
|
+
|
|
230
|
+
```txt
|
|
231
|
+
# Entering Credentials
|
|
232
|
+
Type 'ramesh@coachconnect.app' into 'Email Address'
|
|
233
|
+
Type 'Teacher@1234' into 'Password'
|
|
234
|
+
|
|
235
|
+
# Entering Search Queries
|
|
236
|
+
Type 'Mathematics' into 'Search by batch, subject, or teacher...'
|
|
237
|
+
|
|
238
|
+
# Clearing and Re-entering
|
|
239
|
+
Type 'New York' into 'City'
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
### Assertions & Verifications
|
|
243
|
+
|
|
244
|
+
```txt
|
|
245
|
+
# Visibility Assertions
|
|
246
|
+
Verify 'Apex Coaching Academy' is visible
|
|
247
|
+
Ensure 'Dr. Ramesh Sharma' appears
|
|
248
|
+
Assert 'Order Confirmed' exists
|
|
249
|
+
|
|
250
|
+
# Component State Assertions
|
|
251
|
+
Verify 'Submit' is disabled
|
|
252
|
+
Verify 'Pay Now' is enabled
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
### Navigation, Scrolling & Swiping
|
|
256
|
+
|
|
257
|
+
```txt
|
|
258
|
+
# System Back Navigation
|
|
259
|
+
Press back
|
|
260
|
+
Go back
|
|
261
|
+
|
|
262
|
+
# Vertical Scrolling
|
|
263
|
+
Scroll down
|
|
264
|
+
Scroll up
|
|
265
|
+
|
|
266
|
+
# Directional Swipes (for Carousels, Tabs, Bottom Sheets)
|
|
267
|
+
Swipe left
|
|
268
|
+
Swipe right
|
|
269
|
+
Swipe left on 'Featured Courses'
|
|
270
|
+
Swipe down on 'Modal Sheet'
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
### App Lifecycle & Permissions
|
|
274
|
+
|
|
275
|
+
```txt
|
|
276
|
+
# Explicit App Management
|
|
277
|
+
Launch app 'host.exp.exponent'
|
|
278
|
+
Restart app 'com.example.myapp'
|
|
279
|
+
Terminate app
|
|
280
|
+
Clear app data for 'com.example.myapp'
|
|
281
|
+
|
|
282
|
+
# System Permissions
|
|
283
|
+
Grant permission 'android.permission.CAMERA'
|
|
284
|
+
Grant permission 'android.permission.ACCESS_FINE_LOCATION' to 'com.example.myapp'
|
|
285
|
+
|
|
286
|
+
# Deep Links
|
|
287
|
+
Open deep link 'myapp://batches/grade-10'
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
---
|
|
291
|
+
|
|
292
|
+
## 5. Dynamic & Conditional Testing
|
|
293
|
+
|
|
294
|
+
### Regular Expression / Pattern Matching
|
|
295
|
+
|
|
296
|
+
When asserting dynamic numbers, timestamps, currencies, or roll numbers that vary at runtime:
|
|
297
|
+
|
|
298
|
+
```txt
|
|
299
|
+
# Percentages (e.g. 40%, 75%, 100%)
|
|
300
|
+
Verify pattern '\d+%' is visible
|
|
301
|
+
|
|
302
|
+
# Counters and Ratios (e.g. "5 Students", "5 / 30")
|
|
303
|
+
Verify pattern '\d+\s*(Students|Batches)' is visible
|
|
304
|
+
Verify pattern '\d+\s*/\s*\d+' is visible
|
|
305
|
+
|
|
306
|
+
# Currencies (e.g. "$49.99", "₹1,499")
|
|
307
|
+
Verify pattern '[$₹€]\s*\d+([.,]\d{2})?' is visible
|
|
308
|
+
|
|
309
|
+
# Formatted IDs / Roll Numbers
|
|
310
|
+
Verify pattern 'STD-\d{4}-\d{4}' is visible
|
|
311
|
+
Verify pattern 'INV-[A-Z0-9]+' is visible
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
### Multi-Alternative (OR) Matching
|
|
315
|
+
|
|
316
|
+
```txt
|
|
317
|
+
Verify '100%|90%|80%|75%' is visible
|
|
318
|
+
Verify 'Active|Present|Enrolled' is visible
|
|
319
|
+
Verify 'In Stock|Limited Stock' is visible
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
### Optional / Conditional Actions
|
|
323
|
+
|
|
324
|
+
Steps marked `(optional)` or `(if present)` will execute if the element is on screen, but will **never fail the test** if absent:
|
|
325
|
+
|
|
326
|
+
```txt
|
|
327
|
+
Tap 'Dismiss' (optional)
|
|
328
|
+
Tap 'Allow' (if present)
|
|
329
|
+
Tap 'Skip Tour' (optional)
|
|
330
|
+
```
|
|
331
|
+
|
|
332
|
+
### Dynamic Variables & Generators
|
|
333
|
+
|
|
334
|
+
PromptTest supports built-in generators and custom runtime variable injection.
|
|
335
|
+
|
|
336
|
+
#### Built-in Dynamic Generators
|
|
337
|
+
|
|
338
|
+
```txt
|
|
339
|
+
Type '$RANDOM_EMAIL' into 'Email Address' # Generates: test_492019@prompttest.io
|
|
340
|
+
Type '$RANDOM_NAME' into 'Full Name' # Generates: User_8302
|
|
341
|
+
Type '$RANDOM_PHONE' into 'Phone Number' # Generates: 9849201832
|
|
342
|
+
Type '$TIMESTAMP' into 'Reference Note' # Generates: 1788839201
|
|
343
|
+
Type '$UUID' into 'Transaction ID' # Generates: e9a2c-f901
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
#### Custom CLI Variables
|
|
347
|
+
|
|
348
|
+
In your spec file:
|
|
349
|
+
|
|
350
|
+
```txt
|
|
351
|
+
Type '$USER_EMAIL' into 'Email Address'
|
|
352
|
+
Type '$USER_PASSWORD' into 'Password'
|
|
353
|
+
Verify '$EXPECTED_ROLE' is visible
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
Execute with values:
|
|
357
|
+
|
|
358
|
+
```bash
|
|
359
|
+
npx prompttest run spec.txt --var:USER_EMAIL=ramesh@coachconnect.app --var:USER_PASSWORD=Teacher@1234 --var:EXPECTED_ROLE=TEACHER
|
|
360
|
+
```
|
|
361
|
+
|
|
362
|
+
---
|
|
363
|
+
|
|
364
|
+
## 6. Autonomous Record & Replay
|
|
365
|
+
|
|
366
|
+
Record complex end-to-end user journeys interactively on your device with enterprise-grade safety:
|
|
367
|
+
|
|
368
|
+
### Basic Recording
|
|
369
|
+
|
|
370
|
+
```bash
|
|
371
|
+
npx prompttest record specs/recorded_journey.txt
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
- **Auto-Package Tagging:** Automatically detects the foreground app and writes `# Package: <package>` to the spec header for zero-config replay.
|
|
375
|
+
- **Auto-Backup Protection:** If the file already exists, PromptTest automatically preserves a safety backup as `specs/recorded_journey.txt.bak` before saving.
|
|
376
|
+
- **Auto-Increment Defaults:** Running `npx prompttest record` without a filename will automatically generate `specs/recorded_suite_1.txt`, `specs/recorded_suite_2.txt`, etc., preventing collisions.
|
|
377
|
+
|
|
378
|
+
### Appending to Existing Flows (`--append`)
|
|
379
|
+
|
|
380
|
+
To extend an existing test suite (e.g., adding sidebar FAQ checks to an existing 25-step test):
|
|
381
|
+
|
|
382
|
+
```bash
|
|
383
|
+
npx prompttest record specs/recorded_journey.txt --append
|
|
384
|
+
```
|
|
385
|
+
|
|
386
|
+
1. PromptTest reads existing steps (e.g. 25 steps) and creates a backup.
|
|
387
|
+
2. It starts numbering new recorded steps continuously from `#26`.
|
|
388
|
+
3. It appends the new session with clear timestamped divider comments upon saving.
|
|
389
|
+
|
|
390
|
+
### Intelligent Live Filtering
|
|
391
|
+
|
|
392
|
+
- **Password Bullet Filtering:** Automatically strips security masking characters (`••••`).
|
|
393
|
+
- **Parent Label Association:** Links input fields to their visual labels rather than duplicating input contents.
|
|
394
|
+
- **Keystroke Debouncing:** Captures the final entered value instead of recording each character keystroke.
|
|
395
|
+
- **Tap Jitter Deduplication:** Filters out accidental rapid double-taps on the same UI element.
|
|
396
|
+
|
|
397
|
+
### Replay
|
|
398
|
+
|
|
399
|
+
```bash
|
|
400
|
+
npx prompttest run specs/recorded_journey.txt
|
|
401
|
+
```
|
|
402
|
+
|
|
403
|
+
- Automatically checks device app installation, launches the app if not in foreground, and executes the suite.
|
|
404
|
+
|
|
405
|
+
---
|
|
406
|
+
|
|
407
|
+
## 7. Autonomous Exploration & State-Graph Crawler (Mode 2)
|
|
408
|
+
|
|
409
|
+
PromptTest's **Autonomous Explorer v3** is a zero-script state-graph crawler designed to traverse, test, and map your mobile application automatically without writing test specs.
|
|
410
|
+
|
|
411
|
+
### Basic Autonomous Exploration
|
|
412
|
+
|
|
413
|
+
```bash
|
|
414
|
+
# Auto-detects the active foreground app and starts autonomous crawling
|
|
415
|
+
npx prompttest explore
|
|
416
|
+
|
|
417
|
+
# Target a specific installed package on a specific connected device
|
|
418
|
+
npx prompttest explore host.exp.exponent --serial=GEVKDEUWOJWC89OR
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
### The 5-Step State-Graph Exploration Pipeline
|
|
422
|
+
|
|
423
|
+
1. **Foreground Verification & Auto-Launch**: Ensures the target application is running and in the foreground. If absent, launches the app and verifies launch readiness.
|
|
424
|
+
2. **Semantic Form Detection & Auto-Filling**: Identifies username, email, password, and phone input fields, auto-populates them with realistic test data, dismisses the soft keyboard (`KEYCODE_ESCAPE`), and submits the form.
|
|
425
|
+
3. **Action Buttons & Toggle Testing**: Taps standalone action buttons, flips toggle switches and checkboxes, opens side navigation drawers, and records observable UI outcomes (`NAVIGATED`, `STATE_MUTATED`, or `NO_OP` warning).
|
|
426
|
+
4. **Master-Detail List Exploration & Vertical Scrolling**: Identifies scrollable lists and feeds (`RecyclerView`, `ScrollView`), scrolls vertically to discover off-screen elements below the fold, dives into detail views, and unwinds back via `KEYCODE_BACK`.
|
|
427
|
+
5. **Bottom Navigation Tab Traversal & Sub-Crawling**: Traverses root-level navigation tabs with bottom gesture bar tap-clamping, recursively testing sub-actions under each tab while tracking visited screens in a state graph.
|
|
428
|
+
|
|
429
|
+
---
|
|
430
|
+
|
|
431
|
+
### Safety Policy Engine (`--safety-mode`)
|
|
432
|
+
|
|
433
|
+
To protect production data, shared testing environments, and user accounts from accidental deletion or modification, PromptTest enforces a multi-tiered safety policy:
|
|
434
|
+
|
|
435
|
+
```bash
|
|
436
|
+
# Strict mode (default): Blocks auth drop, checkout/payments, and all deletions
|
|
437
|
+
npx prompttest explore host.exp.exponent --safety-mode=strict
|
|
438
|
+
|
|
439
|
+
# Moderate mode: Blocks account loss & auth, allows local in-app draft/item deletions
|
|
440
|
+
npx prompttest explore host.exp.exponent --safety-mode=moderate
|
|
441
|
+
|
|
442
|
+
# Interactive mode: Prompts user in console before executing risky actions
|
|
443
|
+
npx prompttest explore host.exp.exponent --safety-mode=interactive
|
|
444
|
+
|
|
445
|
+
# Disabled mode: Unrestricted sandbox exploration (isolated environments only)
|
|
446
|
+
npx prompttest explore host.exp.exponent --safety-mode=disabled
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
| Policy Mode | Description | Default Protections |
|
|
450
|
+
| :------------------- | :----------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------- |
|
|
451
|
+
| `strict` _(Default)_ | Maximum safety for production or shared test backends. | Blocks logout, signout, account deletion, data wipe, payments, checkout, and generic item deletions. |
|
|
452
|
+
| `moderate` | Recommended for active development and staging environments. | Shields account deletion, user wipe, and auth termination, but permits localized draft and test item deletion. |
|
|
453
|
+
| `interactive` | Hybrid human-in-the-loop testing. | Prompts `[y/N]` before touching destructive elements. Failsafe: automatically falls back to skip in headless CI. |
|
|
454
|
+
| `disabled` | Isolated sandboxes and ephemeral emulators only. | Bypasses all safety pattern checks. |
|
|
455
|
+
|
|
456
|
+
---
|
|
457
|
+
|
|
458
|
+
### Configurable Safety Blacklists (`--safety-blacklist`)
|
|
459
|
+
|
|
460
|
+
You can define custom blacklist keywords to protect domain-specific critical actions (e.g. "Publish", "Transfer", "Revoke"):
|
|
461
|
+
|
|
462
|
+
```bash
|
|
463
|
+
# Via CLI argument (comma-separated keywords)
|
|
464
|
+
npx prompttest explore host.exp.exponent --safety-blacklist="Transfer,Revoke,Publish,Notice"
|
|
465
|
+
```
|
|
466
|
+
|
|
467
|
+
Or configure permanent safety rules in your `.prompttestrc.json`:
|
|
468
|
+
|
|
469
|
+
```json
|
|
470
|
+
{
|
|
471
|
+
"safety": {
|
|
472
|
+
"mode": "strict",
|
|
473
|
+
"customBlacklist": ["Transfer Funds", "Revoke Access", "Publish Live"]
|
|
474
|
+
}
|
|
475
|
+
}
|
|
476
|
+
```
|
|
477
|
+
|
|
478
|
+
---
|
|
479
|
+
|
|
480
|
+
### External Intent Filtering & Package Jailing (`packageJailGuard`)
|
|
481
|
+
|
|
482
|
+
Mobile applications often contain links that trigger external apps (e.g. "Rate Us" opening Google Play Store, "Terms of Service" opening Chrome, or external OAuth portals).
|
|
483
|
+
|
|
484
|
+
- **Proactive Intent Filter**: The crawler automatically detects and bypasses elements matching `EXTERNAL_INTENT_PATTERNS` (`Rate Us`, `Play Store`, `Privacy Policy`, `https://`, `share via`).
|
|
485
|
+
- **Package Jail Guard**: If an unclassified action causes the device to escape the target app into Chrome or Google Play, PromptTest detects the foreign package, executes `pressBack()`, and if needed, force re-launches the target app to restore containment immediately.
|
|
486
|
+
|
|
487
|
+
---
|
|
488
|
+
|
|
489
|
+
### Development Console & Overlay Auto-Triage
|
|
490
|
+
|
|
491
|
+
In React Native and hybrid apps, runtime warnings or unhandled promise rejections often pop up overlays (LogBox, RedBox, Flutter exception banners, Android ANR alerts) that block underlying UI buttons and stall ordinary automation:
|
|
492
|
+
|
|
493
|
+
1. **Auto-Detection**: Scans the view hierarchy for LogBox overlays and ANR dialogs.
|
|
494
|
+
2. **Stack Trace Extraction**: Captures the exact error message and logs it as a critical defect in the executive report.
|
|
495
|
+
3. **Auto-Dismissal**: Taps the "Dismiss" or "Close" button so autonomous exploration can continue testing the rest of the application.
|
|
496
|
+
|
|
497
|
+
---
|
|
498
|
+
|
|
499
|
+
### Budgets & Performance Controls
|
|
500
|
+
|
|
501
|
+
```bash
|
|
502
|
+
# Deep crawl: up to 50 unique screens, 5 levels deep, 100 interaction steps
|
|
503
|
+
npx prompttest explore host.exp.exponent --max-screens=50 --max-depth=5 --step-budget=100
|
|
504
|
+
|
|
505
|
+
# Fast crawl with lazy failure-only screenshots (slashes execution time by 3-5x)
|
|
506
|
+
npx prompttest explore host.exp.exponent --screenshots=failure-only
|
|
507
|
+
```
|
|
508
|
+
|
|
509
|
+
| Flag | Default | Description |
|
|
510
|
+
| :--------------------- | :------------- | :------------------------------------------------------------------------------------------------------------------------ |
|
|
511
|
+
| `--max-screens=N` | `20` | Maximum unique screen states to index in the state graph. |
|
|
512
|
+
| `--max-depth=N` | `3` | Maximum recursion depth for nested screen diving. |
|
|
513
|
+
| `--step-budget=N` | `40` | Total interaction steps before completing exploration. |
|
|
514
|
+
| `--screenshots=<mode>` | `state-change` | Visual capture policy: `state-change` (only when screen updates), `failure-only` (only on errors), or `all` (every step). |
|
|
515
|
+
|
|
516
|
+
---
|
|
517
|
+
|
|
518
|
+
## 8. Self-Healing & Resilience Engine
|
|
519
|
+
|
|
520
|
+
PromptTest is built with multi-layered self-healing mechanisms:
|
|
521
|
+
|
|
522
|
+
1. **Pre-Authentication Fast-Forwarding:**
|
|
523
|
+
- If a test starts with a login verification (`Verify 'Welcome Back'`), but the app is already authenticated on the dashboard, PromptTest scans ahead and **fast-forwards** past the redundant login steps automatically.
|
|
524
|
+
2. **Cursor-Independent Input Flushing:**
|
|
525
|
+
- Uses `MOVE_END` + `DEL` + `MOVE_HOME` + `FORWARD_DEL` keycodes to reliably clear inputs regardless of initial cursor position or password visibility toggle icons.
|
|
526
|
+
3. **Viewport Auto-Scroll:**
|
|
527
|
+
- Detects if an asserted element is clipped above or below screen boundaries and smoothly scrolls it into view.
|
|
528
|
+
4. **Bottom Tab Bar Coordinate Normalization:**
|
|
529
|
+
- Automatically clamps vertical tap coordinates strictly within the element's actual physical bounds (`[bounds.y1 + 2, bounds.y2 - 2]`), preventing taps from landing outside the viewport or inadvertently triggering Android system gesture navigation bars.
|
|
530
|
+
|
|
531
|
+
---
|
|
532
|
+
|
|
533
|
+
## 9. Multi-Device Matrix & CI/CD Pipelines
|
|
534
|
+
|
|
535
|
+
### Concurrent Multi-Device Execution
|
|
536
|
+
|
|
537
|
+
Test your app across multiple Android models, resolutions, and OS versions concurrently with a single command:
|
|
538
|
+
|
|
539
|
+
```bash
|
|
540
|
+
npx prompttest run specs/smoke_suite.txt --all-devices
|
|
541
|
+
```
|
|
542
|
+
|
|
543
|
+
### GitHub Actions CI Workflow Setup
|
|
544
|
+
|
|
545
|
+
Run `npx prompttest init-ci` to automatically generate two production-ready GitHub Actions workflows under `.github/workflows/`:
|
|
546
|
+
|
|
547
|
+
- **`unit.yml`**: Headless Node.js 18 & 20 quality gates (Prettier check, ESLint, TypeScript typecheck, Vitest headless test suite with code coverage, and npm audit).
|
|
548
|
+
- **`device.yml`**: Android Emulator API 33 live spec execution with hardware KVM acceleration, automated JUnit XML publishing, and artifact capture.
|
|
549
|
+
|
|
550
|
+
```bash
|
|
551
|
+
npx prompttest init-ci
|
|
552
|
+
```
|
|
553
|
+
|
|
554
|
+
### Central Multi-Device Report Aggregation & Atto Quality Scoring
|
|
555
|
+
|
|
556
|
+
When running tests across multiple physical devices or emulators using `--all-devices`, individual result JSON files (`*-results.json`) are produced per device. PromptTest can merge these results into a unified matrix dashboard:
|
|
557
|
+
|
|
558
|
+
```bash
|
|
559
|
+
npx prompttest merge-reports output
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
The generated `output/aggregated-dashboard.html` and `output/aggregated-results.json` feature:
|
|
563
|
+
|
|
564
|
+
- **Atto-Inspired Release Confidence Score (0–100%)**: An enterprise quality index calculated from weighted pass rates, critical-path assertion coverage, and device diversity.
|
|
565
|
+
- **Cross-Device Comparison Matrix**: Detailed side-by-side grid comparing execution status and timing for each step across every device model.
|
|
566
|
+
- **Consolidated Defect Summary**: Highlighting visual or functional regressions that appear on specific device form factors.
|
|
567
|
+
|
|
568
|
+
---
|
|
569
|
+
|
|
570
|
+
## 10. Advanced Features & SDK
|
|
571
|
+
|
|
572
|
+
### Checkpointing & Resume
|
|
573
|
+
|
|
574
|
+
Long test suites can fail halfway due to network or environment issues. PromptTest automatically checkpoints progress after each successful step.
|
|
575
|
+
You can use the `--resume` flag to skip previously passed steps and continue execution from the exact point of failure:
|
|
576
|
+
|
|
577
|
+
```bash
|
|
578
|
+
npx prompttest run specs/regression_suite.txt --resume
|
|
579
|
+
```
|
|
580
|
+
|
|
581
|
+
### Visual Baselines
|
|
582
|
+
|
|
583
|
+
PromptTest can capture visual baselines and perform perceptual diffing to detect unintended UI changes.
|
|
584
|
+
|
|
585
|
+
- `--save-baseline`: Saves the current screen layout as a visual baseline for future comparisons.
|
|
586
|
+
- `--compare-baseline`: Compares the current screen layout against the saved baseline and fails if differences exceed the threshold.
|
|
587
|
+
- `--baseline-threshold=N`: Adjusts the diffing tolerance as a ratio from `0` to `1` (e.g., `--baseline-threshold=0.05` for 5%).
|
|
588
|
+
|
|
589
|
+
### Custom Reporters
|
|
590
|
+
|
|
591
|
+
You can integrate custom test reporters by supplying a path to your reporter module:
|
|
592
|
+
|
|
593
|
+
```bash
|
|
594
|
+
npx prompttest run specs/suite.txt --reporter=./reporters/my-custom-reporter.js
|
|
595
|
+
```
|
|
596
|
+
|
|
597
|
+
Custom reporters must implement the `Reporter` interface for pluggable report formats (e.g., Slack, DataDog, HTML).
|
|
598
|
+
|
|
599
|
+
### APK Auto-Install
|
|
600
|
+
|
|
601
|
+
For CI environments, you can automatically install a specific APK before running tests:
|
|
602
|
+
|
|
603
|
+
```bash
|
|
604
|
+
npx prompttest run specs/suite.txt --apk=./build/app-release.apk
|
|
605
|
+
```
|
|
606
|
+
|
|
607
|
+
PromptTest will verify the device architecture and seamlessly deploy the build prior to execution.
|
|
608
|
+
|
|
609
|
+
### Device Locking
|
|
610
|
+
|
|
611
|
+
PromptTest includes a cross-process, file-based mutex lock (`lib/lock.ts`) that guarantees concurrent safety. If multiple test processes try to control the same physical device or emulator simultaneously, they will wait in a queue until the lock is released.
|
|
612
|
+
|
|
613
|
+
### Programmatic SDK & Cross-Platform Drivers
|
|
614
|
+
|
|
615
|
+
PromptTest exposes a clean programmatic API using factory functions and a unified `DriverInterface`. Both Android and iOS drivers share the identical driver abstraction:
|
|
616
|
+
|
|
617
|
+
```typescript
|
|
618
|
+
import { createPromptRunner, createAndroidDriver, IosDriver } from 'prompttest';
|
|
619
|
+
|
|
620
|
+
async function runCrossPlatformTests() {
|
|
621
|
+
// 🤖 Android Execution
|
|
622
|
+
const androidDriver = createAndroidDriver('DEVICE_SERIAL');
|
|
623
|
+
const androidRunner = createPromptRunner(androidDriver);
|
|
624
|
+
const androidReport = await androidRunner.runSpec('specs/login.txt', 'com.example.app');
|
|
625
|
+
|
|
626
|
+
// 🍎 iOS Execution (Simulator or Physical Device)
|
|
627
|
+
const iosDriver = new IosDriver({
|
|
628
|
+
udid: '00008101-001234567890',
|
|
629
|
+
});
|
|
630
|
+
const iosRunner = createPromptRunner(iosDriver);
|
|
631
|
+
const iosReport = await iosRunner.runSpec('specs/login.txt', 'com.example.app');
|
|
632
|
+
|
|
633
|
+
console.log(`Android passed: ${androidReport.passed}, iOS passed: ${iosReport.passed}`);
|
|
634
|
+
}
|
|
635
|
+
|
|
636
|
+
runCrossPlatformTests();
|
|
637
|
+
```
|
|
638
|
+
|
|
639
|
+
### Cross-Platform Driver Architecture (`DriverInterface`)
|
|
640
|
+
|
|
641
|
+
PromptTest decouples test execution from OS-level automation protocols. Custom mobile environments (e.g. Flutter desktop, HarmonyOS, or cloud device farms) can implement the minimal `DriverInterface`:
|
|
642
|
+
|
|
643
|
+
- `getUiHierarchy(serial?)`: Returns the XML / accessibility view hierarchy.
|
|
644
|
+
- `tap(x, y, serial?)`: Dispatches a tap event at screen coordinates.
|
|
645
|
+
- `inputText(text, serial?)`: Inputs text into the active field.
|
|
646
|
+
- `pressKey(keyCode, serial?)`: Sends hardware key codes (e.g., BACK, ENTER).
|
|
647
|
+
- `takeScreenshot(outputPath, serial?)`: Captures screen image artifact.
|
|
648
|
+
|
|
649
|
+
---
|
|
650
|
+
|
|
651
|
+
## 11. Real-World Domain Recipes
|
|
652
|
+
|
|
653
|
+
### EdTech & Coaching Portals
|
|
654
|
+
|
|
655
|
+
```txt
|
|
656
|
+
# 1. Sign In as Faculty
|
|
657
|
+
1. Verify 'Welcome Back' is visible
|
|
658
|
+
2. Type 'ramesh@coachconnect.app' into 'Email Address'
|
|
659
|
+
3. Type 'Teacher@1234' into 'Password'
|
|
660
|
+
4. Tap 'Sign In'
|
|
661
|
+
5. Wait 2s
|
|
662
|
+
|
|
663
|
+
# 2. Verify Batches & Class Schedulers
|
|
664
|
+
6. Tap 'Batches'
|
|
665
|
+
7. Verify 'Batches & Schedules' is visible
|
|
666
|
+
8. Verify 'Grade 10 Mathematics' is visible
|
|
667
|
+
9. Verify pattern '\d+\s*/\s*30' is visible
|
|
668
|
+
|
|
669
|
+
# 3. Attendance Hub
|
|
670
|
+
10. Tap 'Attendance'
|
|
671
|
+
11. Verify 'Attendance Hub' is visible
|
|
672
|
+
12. Verify 'Rohan Gupta' is visible
|
|
673
|
+
13. Verify pattern '\d+%' is visible
|
|
674
|
+
14. Tap 'Export Report'
|
|
675
|
+
15. Verify 'Export Attendance Report' is visible
|
|
676
|
+
16. Press back
|
|
677
|
+
|
|
678
|
+
# 4. Sign Out
|
|
679
|
+
17. Tap 'Profile'
|
|
680
|
+
18. Tap 'Sign Out'
|
|
681
|
+
19. Tap 'Sign Out'
|
|
682
|
+
20. Verify 'Welcome Back' is visible
|
|
683
|
+
```
|
|
684
|
+
|
|
685
|
+
### E-Commerce & Food Delivery
|
|
686
|
+
|
|
687
|
+
```txt
|
|
688
|
+
# 1. Search & Browse
|
|
689
|
+
1. Type 'Wireless Headphones' into 'Search products...'
|
|
690
|
+
2. Tap 1st 'Add to Cart'
|
|
691
|
+
3. Tap 'Cart'
|
|
692
|
+
4. Verify 'Shopping Cart' is visible
|
|
693
|
+
5. Verify pattern 'Total:\s*₹\d+' is visible
|
|
694
|
+
|
|
695
|
+
# 2. Checkout & Address
|
|
696
|
+
6. Tap 'Proceed to Checkout'
|
|
697
|
+
7. Type '42 HSR Layout, Sector 2' into 'Delivery Address'
|
|
698
|
+
8. Type '560102' into 'Pincode'
|
|
699
|
+
9. Tap 'Cash on Delivery|UPI Payment'
|
|
700
|
+
10. Tap 'Place Order'
|
|
701
|
+
11. Verify 'Order Confirmed' is visible
|
|
702
|
+
```
|
|
703
|
+
|
|
704
|
+
### FinTech & Banking
|
|
705
|
+
|
|
706
|
+
```txt
|
|
707
|
+
# 1. Biometric / PIN Auth
|
|
708
|
+
1. Verify 'Secure Login' is visible
|
|
709
|
+
2. Type '1234' into 'Enter 4-digit PIN'
|
|
710
|
+
3. Tap 'Continue'
|
|
711
|
+
|
|
712
|
+
# 2. Balance & Transactions
|
|
713
|
+
4. Verify 'Account Overview' is visible
|
|
714
|
+
5. Verify pattern 'Balance:\s*₹[\d,]+(\.\d{2})?' is visible
|
|
715
|
+
6. Tap 'Transaction History'
|
|
716
|
+
7. Verify 'Recent Transactions' is visible
|
|
717
|
+
8. Verify pattern 'TXN-\d{8}' is visible
|
|
718
|
+
```
|
|
719
|
+
|
|
720
|
+
---
|
|
721
|
+
|
|
722
|
+
## 🎯 12. Mode Expectations, Capabilities & Limitations
|
|
723
|
+
|
|
724
|
+
> **Read first:** Full details in **[`docs/MODE_EXPECTATIONS.md`](MODE_EXPECTATIONS.md)**. This section is the honest contract for what each mode can and cannot do.
|
|
725
|
+
|
|
726
|
+
PromptTest is a **fast UI-tree automation engine over ADB**. It "sees" what `uiautomator dump` exposes — not pixels. This one fact drives every limit below.
|
|
727
|
+
|
|
728
|
+
### Which mode should I use?
|
|
729
|
+
|
|
730
|
+
| Mode | Command | Best for | Not for |
|
|
731
|
+
| ----------------------------- | -------------------------------------- | ------------------------------------------------------------ | ----------------------------------- |
|
|
732
|
+
| **Instructed (`run`)** | `prompttest run specs/foo.txt <pkg>` | Regression gates, CI, verifying a known flow with assertions | Exploratory discovery |
|
|
733
|
+
| **Autonomous (`explore`)** | `prompttest explore <pkg>` | Unattended state-graph discovery and safety-guarded sweeps | Replacing asserted regression tests |
|
|
734
|
+
| **Hybrid (`record` → `run`)** | `prompttest record` → `prompttest run` | Turning a manual walkthrough into a repeatable script | — (review the generated spec first) |
|
|
735
|
+
|
|
736
|
+
### What you should honestly expect
|
|
737
|
+
|
|
738
|
+
- ✅ **`run`**: executes plain-English steps, polls for elements, asserts visibility/state, self-heals broken locators, reuses offline memory to speed repeat runs. Deterministic per step, CI-friendly.
|
|
739
|
+
- ✅ **`explore`**: **State-Graph Crawler v3** — systematically traverses forms, toggles switches/checkboxes, inspects side navigation drawers, discovers off-screen content via vertical scrolling, dives into master-detail lists, and traverses bottom navigation tabs with sub-crawling. Protected by configurable safety blacklists, external package jailing, and dev overlay auto-dismissal.
|
|
740
|
+
- ⚠️ **Performance**: "sub-second" = per **step**, not per whole suite.
|
|
741
|
+
|
|
742
|
+
### Autonomous `explore` — capabilities and architecture
|
|
743
|
+
|
|
744
|
+
- **State-Graph Crawling**: Uses BFS/DFS graph indexing via hierarchy hashing (`computeHierarchyHash`) and offline `MemoryEngine` caching to track visited screens, detect loops, and fast-forward past redundant paths.
|
|
745
|
+
- **Safety Policy Enforcement**: Hierarchical safety modes (`strict`, `moderate`, `interactive`, `disabled`) ensure dangerous buttons (delete account, checkout, logout) are blocked and reported in the audit hit-list.
|
|
746
|
+
- **Domain Containment**: `packageJailGuard` prevents the crawler from leaving the app when tapping external intent links (Rate Us, Store URLs, Web browsers).
|
|
747
|
+
- **Overlay Triage**: Automatically extracts errors from React Native LogBox, RedBox, Flutter exception views, and Android ANRs, logging them as defects and dismissing the overlay.
|
|
748
|
+
- **Zero third-party AI required**: 100% heuristic and offline execution using the in-repo `HeuristicResolver` + `MemoryEngine`. Runs in air-gapped CI environments without network latency.
|
|
749
|
+
|
|
750
|
+
### Honest limitations (apply to every mode)
|
|
751
|
+
|
|
752
|
+
1. **Accessibility-tree dependency.** An element with no `text`, `content-desc`, `resource-id`, or clickable bounds is invisible or only tappable "blind."
|
|
753
|
+
2. **Games / canvas / fully-drawn content** may expose no hierarchy → PromptTest cannot inspect or assert it.
|
|
754
|
+
3. **OS overlays & Secure/FLAG_SECURE screens** can block dumps/capture.
|
|
755
|
+
4. **Determinism ceiling.** Timers, real services, and animations can cause flakiness; wait for a stable state in your spec.
|
|
756
|
+
5. **Zero third-party AI required.** All capabilities — including autonomous crawling — work fully offline via heuristics + memory. LLM is an optional, gated enhancement, never a prerequisite.
|
|
757
|
+
|
|
758
|
+
---
|
|
759
|
+
|
|
760
|
+
## 💡 Support & Contribution
|
|
761
|
+
|
|
762
|
+
- **Community Repository**: [github.com/shriramsingh/prompttest-community](https://github.com/shriramsingh/prompttest-community)
|
|
763
|
+
- **Issues & Requests**: [File an Issue](https://github.com/shriramsingh/prompttest-community/issues)
|
|
764
|
+
- **License**: Business Source License 1.1 (BSL 1.1)
|