prompttest 1.3.0 β 1.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +127 -290
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,66 +1,62 @@
|
|
|
1
1
|
# β‘ PromptTest
|
|
2
2
|
|
|
3
|
-
> **Ultra-fast, zero-code autonomous mobile testing & visual QA brain for Android.**
|
|
3
|
+
> **Ultra-fast, zero-code autonomous mobile testing & visual QA brain for Android.**
|
|
4
|
+
> The zero-setup, zero-instrumentation alternative to Appium & Detox for React Native, Expo, Flutter, and Native Android.
|
|
4
5
|
|
|
5
|
-
[](https://www.npmjs.com/package/prompttest)
|
|
7
|
+
[](LICENSE)
|
|
6
8
|
[](https://nodejs.org/)
|
|
7
9
|
[](https://developer.android.com/tools/adb)
|
|
8
|
-
[](https://www.typescriptlang.org/)
|
|
11
11
|
|
|
12
|
-
|
|
12
|
+
**PromptTest** enables QA engineers, mobile developers, and autonomous agents to test Android applications using **plain-English natural language scripts**. It completely bypasses heavy server frameworks (Appium, Selenium, WebDriver) in favor of high-speed native ADB streams, expectation-driven polling, and an autonomous state-graph exploration engine.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
π **[Read the Full User Manual](docs/USER_MANUAL.md)** β’ π **[System Architecture](docs/ARCHITECTURE.md)** β’ π **[License (BSL 1.1)](LICENSE)** β’ π€ **[Contributing](CONTRIBUTING.md)**
|
|
15
15
|
|
|
16
16
|
---
|
|
17
17
|
|
|
18
18
|
## π Why PromptTest?
|
|
19
19
|
|
|
20
|
-
- π **Plain-English Test Scripts**: Write tests in human language without writing
|
|
21
|
-
- β‘ **
|
|
22
|
-
-
|
|
23
|
-
-
|
|
24
|
-
- π‘ **
|
|
25
|
-
-
|
|
20
|
+
- π **Plain-English Test Scripts**: Write tests in human language without writing fragile XPath selectors or boilerplate glue code.
|
|
21
|
+
- β‘ **Zero-Setup & Zero-Instrumentation**: Connects directly over native ADB (USB or Wi-Fi). No SDKs, test dependencies, or code modifications required in your target app.
|
|
22
|
+
- π€ **Autonomous State-Graph DFS Explorer**: Crawls apps autonomously, detects bottom-tab navigation hubs, traverses nested screens, and automatically triages defects without human intervention.
|
|
23
|
+
- π‘ **Dynamic Self-Healing Locators**: Resilient heuristic matching automatically adapts to changing dynamic counts (e.g. auto-resolving `"Present (2)"` to `"Present (5)"`).
|
|
24
|
+
- π‘ **Built-In Safety Engine**: Guards production and staging apps by blocking destructive actions (e.g., `"Delete Account"`, `"Discard Changes"`) during autonomous crawls.
|
|
25
|
+
- π **Standalone HTML & JUnit Reports**: Generates executive test reports with failure-only visual screenshots and actionable error diffs.
|
|
26
|
+
- π **Built for Modern Mobile Frameworks**: Native support for **React Native**, **Expo**, **Flutter**, and Native Android (Jetpack Compose / Views).
|
|
26
27
|
|
|
27
28
|
---
|
|
28
29
|
|
|
29
|
-
## π Quick Start
|
|
30
|
+
## π 30-Second Quick Start
|
|
30
31
|
|
|
31
32
|
### Prerequisites
|
|
33
|
+
- **Node.js 18.0.0+**
|
|
34
|
+
- **Android ADB** installed and accessible in your system `PATH`
|
|
35
|
+
- An Android device or emulator with **USB Debugging** enabled
|
|
32
36
|
|
|
33
|
-
1.
|
|
34
|
-
|
|
35
|
-
3. An Android device or emulator with **USB Debugging** enabled.
|
|
36
|
-
4. _Language Support_: The default seed dictionary covers English mobile UI interactions. Applications localized in other languages work via direct label matching or custom specifications.
|
|
37
|
-
|
|
38
|
-
### Installation
|
|
39
|
-
|
|
37
|
+
### 1. Run Instant Diagnostics (Zero Install Needed)
|
|
38
|
+
Verify your environment and connected devices with zero configuration:
|
|
40
39
|
```bash
|
|
41
|
-
|
|
42
|
-
git clone https://github.com/<your-username>/prompttest.git
|
|
43
|
-
cd prompttest
|
|
44
|
-
|
|
45
|
-
# Install dependencies
|
|
46
|
-
npm install
|
|
47
|
-
|
|
48
|
-
# Build TypeScript
|
|
49
|
-
npm run build
|
|
40
|
+
npx prompttest doctor
|
|
50
41
|
```
|
|
51
42
|
|
|
52
|
-
###
|
|
43
|
+
### 2. Autonomous Exploration (0 Lines of Code)
|
|
44
|
+
Autonomously crawl, discover navigation tabs, and stress-test your app:
|
|
45
|
+
```bash
|
|
46
|
+
npx prompttest explore <your.app.package>
|
|
47
|
+
```
|
|
53
48
|
|
|
49
|
+
### 3. Install in Your Project
|
|
50
|
+
Add PromptTest as a development dependency in your mobile project:
|
|
54
51
|
```bash
|
|
55
|
-
|
|
56
|
-
npm run devices
|
|
52
|
+
npm install --save-dev prompttest
|
|
57
53
|
```
|
|
58
54
|
|
|
59
55
|
---
|
|
60
56
|
|
|
61
57
|
## βοΈ Writing Plain-English Tests
|
|
62
58
|
|
|
63
|
-
Test
|
|
59
|
+
Test specifications are simple text files containing numbered steps written in conversational English.
|
|
64
60
|
|
|
65
61
|
### Example Spec (`specs/login_and_dashboard.txt`)
|
|
66
62
|
|
|
@@ -90,313 +86,154 @@ Test specs are simple text files containing numbered steps written in conversati
|
|
|
90
86
|
16. Verify 'Welcome Back' is visible
|
|
91
87
|
```
|
|
92
88
|
|
|
93
|
-
|
|
89
|
+
### Run the Spec:
|
|
90
|
+
```bash
|
|
91
|
+
npx prompttest run specs/login_and_dashboard.txt com.yourcompany.app
|
|
92
|
+
```
|
|
94
93
|
|
|
95
|
-
|
|
94
|
+
---
|
|
96
95
|
|
|
97
|
-
|
|
96
|
+
## π Core Commands & Capabilities
|
|
98
97
|
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
| **Verify Visible** | `Verify 'Dashboard' is visible` | Polls until the text or accessibility label appears on screen. |
|
|
105
|
-
| **Verify State** | `Verify 'Submit' is disabled` | Verifies element enablement state (`enabled` / `disabled`). |
|
|
106
|
-
| **Press Back** | `Press back` | Emits Android system `KEYCODE_BACK`. |
|
|
107
|
-
| **Scroll / Swipe** | `Scroll down` or `Scroll up` | Proportional responsive swipe along the screen viewport. |
|
|
108
|
-
| **Relative Tap** | `Tap 'Delete' next to 'Rohan Gupta'` | Disambiguates duplicate elements using spatial proximity. |
|
|
109
|
-
| **Wait** | `Wait 3s` or `Wait 1500ms` | Pauses execution for custom animation delays. |
|
|
98
|
+
### 1. Spec Runner (`prompttest run`)
|
|
99
|
+
Executes deterministic plain-English test specs with fail-fast execution and self-healing locators:
|
|
100
|
+
```bash
|
|
101
|
+
# Basic run
|
|
102
|
+
npx prompttest run specs/flow.txt com.example.app
|
|
110
103
|
|
|
111
|
-
|
|
104
|
+
# Cold-restart app before testing begins
|
|
105
|
+
npx prompttest run specs/flow.txt com.example.app --fresh
|
|
112
106
|
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
| **Conditional** | `IF 'Promo' is visible ... ELSE ... ENDIF` | Executes branch based on live element presence. |
|
|
116
|
-
| **Loop** | `LOOP 3 ... ENDLOOP` | Repeats child step block N times. |
|
|
117
|
-
| **Wait Until** | `WAIT_UNTIL 'Ready' is visible (10s)` | Explicit polling wait with max timeout budget. |
|
|
118
|
-
| **Subflow Include** | `#include <specs/login.txt>` | Inlines modular specs with circular dependency guards. |
|
|
119
|
-
| **Variables** | `SET $token = 'xyz'` | Sets local variable for interpolation (`$token`). |
|
|
120
|
-
| **Sandboxed JS** | `RUN_JS return { status: 200 }` | Executes deterministic JavaScript expressions. |
|
|
107
|
+
# Capture full screenshots on every step
|
|
108
|
+
npx prompttest run specs/flow.txt com.example.app --screenshots
|
|
121
109
|
|
|
122
|
-
|
|
110
|
+
# Dry-run validation (validates grammar without touching device)
|
|
111
|
+
npx prompttest run specs/flow.txt --dry-run
|
|
123
112
|
|
|
124
|
-
|
|
113
|
+
# Display formatted table of parsed steps
|
|
114
|
+
npx prompttest run specs/flow.txt --list-steps
|
|
125
115
|
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
{"user": "Alice", "email": "$random.email"},
|
|
129
|
-
{"user": "Bob", "email": "$random.email"}
|
|
130
|
-
]
|
|
131
|
-
1. Type '$user' into 'Username'
|
|
132
|
-
2. Type '$email' into 'Email'
|
|
133
|
-
```
|
|
134
|
-
|
|
135
|
-
**Supported Generators**:
|
|
136
|
-
|
|
137
|
-
- Synthetic Data: `$random.firstName`, `$random.lastName`, `$random.email`, `$random.phone`, `$random.string(8)`
|
|
138
|
-
- Sequences & UUIDs: `$seq.id`, `$uuid`
|
|
139
|
-
- Date & Time: `$date.now`, `$date.iso`, `$date.format('YYYY-MM-DD')`
|
|
140
|
-
- Math Evaluation: `$calc($count + 1)`
|
|
116
|
+
# Run data-driven iterations
|
|
117
|
+
npx prompttest run specs/flow.txt com.example.app --iterations 5
|
|
141
118
|
|
|
142
|
-
|
|
119
|
+
# Run against a managed device pool
|
|
120
|
+
npx prompttest run specs/flow.txt com.example.app --device-pool emulator-5554,emulator-5556
|
|
143
121
|
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
2. Type '$code' into 'Verification Code'
|
|
148
|
-
3. API GET 'https://api.example.com/user/profile' EXPECT status 200
|
|
149
|
-
4. VERIFY EMAIL to 'test@domain.com' subject 'Welcome'
|
|
150
|
-
|
|
151
|
-
# Device Simulation
|
|
152
|
-
5. Set network to offline
|
|
153
|
-
6. Set battery level to 20
|
|
122
|
+
# Visual regression testing against gold baseline images
|
|
123
|
+
npx prompttest run specs/flow.txt --save-baseline
|
|
124
|
+
npx prompttest run specs/flow.txt --compare-baseline --baseline-threshold=0.02
|
|
154
125
|
```
|
|
155
126
|
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
## π₯ CLI Usage
|
|
159
|
-
|
|
160
|
-
### Run a Test Spec
|
|
161
|
-
|
|
127
|
+
### 2. Autonomous State-Graph Explorer (`prompttest explore`)
|
|
128
|
+
Autonomous crawling engine with bottom-tab detection, smart form filling, and safety policies:
|
|
162
129
|
```bash
|
|
163
|
-
#
|
|
164
|
-
npx prompttest
|
|
130
|
+
# Standard autonomous crawl (Strict safety mode)
|
|
131
|
+
npx prompttest explore com.example.app
|
|
165
132
|
|
|
166
|
-
#
|
|
167
|
-
npx prompttest
|
|
133
|
+
# Set custom screen and step interaction budgets
|
|
134
|
+
npx prompttest explore com.example.app --max-screens=30 --step-budget=60
|
|
168
135
|
|
|
169
|
-
#
|
|
170
|
-
npx prompttest
|
|
171
|
-
|
|
172
|
-
# Run against a managed device pool with concurrency leasing
|
|
173
|
-
npx prompttest run specs/suite.txt host.exp.exponent --device-pool emulator-5554,emulator-5556 --device-timeout-secs 30
|
|
136
|
+
# Add custom protected blacklist keywords
|
|
137
|
+
npx prompttest explore com.example.app --safety-blacklist="Wipe,Revoke,Transfer"
|
|
174
138
|
```
|
|
175
139
|
|
|
176
|
-
###
|
|
177
|
-
|
|
178
|
-
| Flag | Description |
|
|
179
|
-
| :--------------------------------------- | :-------------------------------------------------------------------------- |
|
|
180
|
-
| `--fresh` | Force-stops and restarts target app before test begins. |
|
|
181
|
-
| `--device-pool <d1,d2>` | Executes against the first available device from pool using atomic locking. |
|
|
182
|
-
| `--device-timeout-secs <n>` | Maximum seconds to wait for a free device lock (default: 60s). |
|
|
183
|
-
| `--iterations <n>` | Executes spec across N dynamic data iterations. |
|
|
184
|
-
| `--only-row <n>` | Filters execution to a specific 1-indexed data row. |
|
|
185
|
-
| `--continue` / `--continue-on-failure` | Continues running remaining steps after failure instead of fast-failing. |
|
|
186
|
-
| `--screenshots <all\|failure\|none>` | Controls screenshot capture policy (default: `failure`). |
|
|
187
|
-
| `--save-baseline` / `--update-baselines` | Saves gold reference baseline images for visual regression. |
|
|
188
|
-
| `--compare-baseline` | Compares execution screens against reference baseline. |
|
|
189
|
-
| `--baseline-threshold <n>` | Sets allowable perceptual diff tolerance (e.g. 0.02 = 2%). |
|
|
190
|
-
| `--format <html,json,junit,markdown>` | Comma-separated output report formats. |
|
|
191
|
-
| `-s <serial>` | Targets a specific connected device serial. |
|
|
192
|
-
| `--version` / `-v` | Prints current PromptTest version. |
|
|
193
|
-
|
|
194
|
-
### Other Commands
|
|
195
|
-
|
|
140
|
+
### 3. Interactive Record & Replay (`prompttest record`)
|
|
141
|
+
Record your real interactions on the phone and auto-generate clean test specs:
|
|
196
142
|
```bash
|
|
197
|
-
|
|
198
|
-
npx prompttest wifi 192.168.1.50
|
|
199
|
-
|
|
200
|
-
# Disconnect wireless device
|
|
201
|
-
npx prompttest disconnect
|
|
202
|
-
|
|
203
|
-
# Interactive record-and-play session
|
|
204
|
-
npx prompttest record specs/login.txt
|
|
205
|
-
|
|
206
|
-
# Inspect foreground app and active components on screen
|
|
207
|
-
npx prompttest status
|
|
208
|
-
|
|
209
|
-
# Run comprehensive environment and ADB readiness diagnostics
|
|
210
|
-
npx prompttest doctor
|
|
211
|
-
|
|
212
|
-
# Start interactive live REPL sandbox for prototyping test steps
|
|
213
|
-
npx prompttest repl
|
|
214
|
-
|
|
215
|
-
# Merge multi-device test results and generate Atto-inspired release confidence dashboard
|
|
216
|
-
npx prompttest merge-reports output
|
|
217
|
-
|
|
218
|
-
# Automatically crawl and map the app (Autonomous State-Graph Crawler v3)
|
|
219
|
-
npx prompttest explore host.exp.exponent
|
|
220
|
-
|
|
221
|
-
# Run autonomous crawl with strict safety policy and custom domain blacklist
|
|
222
|
-
npx prompttest explore host.exp.exponent --safety-mode=strict --safety-blacklist="Transfer,Revoke,Publish"
|
|
223
|
-
|
|
224
|
-
# Deep exploration with custom screen/step budgets and lazy failure-only screenshots
|
|
225
|
-
npx prompttest explore host.exp.exponent --max-screens=30 --step-budget=60 --screenshots=failure-only
|
|
226
|
-
|
|
227
|
-
# View learned component aliases & confidence
|
|
228
|
-
npx prompttest memory host.exp.exponent
|
|
229
|
-
|
|
230
|
-
# Generate GitHub Actions CI workflow
|
|
231
|
-
npx prompttest init-ci
|
|
143
|
+
npx prompttest record specs/recorded_flow.txt
|
|
232
144
|
```
|
|
233
145
|
|
|
234
|
-
###
|
|
235
|
-
|
|
236
|
-
| Flag | Default | Description | Example |
|
|
237
|
-
| :------------------------- | :------------- | :----------------------------------------------------------------------------------------------------------------- | :--------------------------------- |
|
|
238
|
-
| `--safety-mode=<mode>` | `strict` | Policy level: `strict` (all deletions blocked), `moderate` (drafts permitted), `interactive` (prompts), `disabled` | `--safety-mode=moderate` |
|
|
239
|
-
| `--safety-blacklist="..."` | `[]` | Comma-separated custom keywords to protect as destructive actions | `--safety-blacklist="Revoke,Wipe"` |
|
|
240
|
-
| `--max-screens=N` | `20` | Maximum unique screen states to discover and index | `--max-screens=50` |
|
|
241
|
-
| `--max-depth=N` | `3` | Maximum recursion depth for nested screen diving | `--max-depth=5` |
|
|
242
|
-
| `--step-budget=N` | `40` | Total interaction steps before completing exploration | `--step-budget=100` |
|
|
243
|
-
| `--screenshots=<mode>` | `state-change` | Image capture policy: `state-change`, `failure-only`, or `all` | `--screenshots=failure-only` |
|
|
244
|
-
| `-s <serial>` | auto | Target specific connected device serial | `-s GEVKDEUWOJWC89OR` |
|
|
245
|
-
|
|
246
|
-
### CLI Execution Flags (`run`)
|
|
247
|
-
|
|
248
|
-
| Flag | Description | Example |
|
|
249
|
-
| :--------------- | :----------------------------------------------------------- | :----------------------------------------------- |
|
|
250
|
-
| `--dry-run` | Validates and parses spec steps with 0 device execution | `prompttest run specs/flow.txt --dry-run` |
|
|
251
|
-
| `--list-steps` | Prints a formatted table of all parsed steps in the spec | `prompttest run specs/flow.txt --list-steps` |
|
|
252
|
-
| `--only=<range>` | Executes only specific step numbers or ranges (e.g. `2-5,8`) | `prompttest run specs/flow.txt --only="1-3,5"` |
|
|
253
|
-
| `--tags=<tags>` | Filters execution by tags declared via `# @tag: <name>` | `prompttest run specs/flow.txt --tags="smoke"` |
|
|
254
|
-
| `--json` | Outputs machine-readable run summary JSON to stdout | `prompttest run specs/flow.txt --json` |
|
|
255
|
-
| `--benchmark` | Profiles and logs execution duration for each step | `prompttest run specs/flow.txt --benchmark` |
|
|
256
|
-
| `--watch` | Watches spec file for changes and re-executes automatically | `prompttest run specs/flow.txt --watch` |
|
|
257
|
-
| `--fresh` | Force-stops and restarts target app before execution | `prompttest run specs/flow.txt --fresh` |
|
|
258
|
-
| `--all-devices` | Concurrently executes spec across all connected devices | `prompttest run specs/flow.txt --all-devices` |
|
|
259
|
-
| `--continue` | Continues executing remaining steps if a step fails | `prompttest run specs/flow.txt --continue` |
|
|
260
|
-
| `--var:K=V` | Injects runtime variable overrides into the test context | `prompttest run specs/flow.txt --var:USER=Alice` |
|
|
261
|
-
|
|
262
|
-
---
|
|
263
|
-
|
|
264
|
-
## π³ Docker Container Deployment
|
|
265
|
-
|
|
266
|
-
PromptTest is fully containerized with Android platform-tools and headless runtime support:
|
|
267
|
-
|
|
146
|
+
### 4. Device Management & Wi-Fi Debugging
|
|
147
|
+
Connect to physical phones wirelessly without keeping USB cables attached:
|
|
268
148
|
```bash
|
|
269
|
-
#
|
|
270
|
-
|
|
149
|
+
# List connected devices and serials
|
|
150
|
+
npx prompttest devices
|
|
271
151
|
|
|
272
|
-
#
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
152
|
+
# Connect to device wirelessly over Wi-Fi
|
|
153
|
+
npx prompttest wifi 192.168.1.50
|
|
154
|
+
|
|
155
|
+
# Inspect active screen hierarchy and detected components
|
|
156
|
+
npx prompttest status
|
|
277
157
|
```
|
|
278
158
|
|
|
279
159
|
---
|
|
280
160
|
|
|
281
|
-
##
|
|
282
|
-
|
|
283
|
-
```mermaid
|
|
284
|
-
flowchart TD
|
|
285
|
-
A[Plain-English Spec .txt] --> B[DSL Parser & Tokenizer]
|
|
286
|
-
B --> C[PromptRunner Engine]
|
|
287
|
-
C --> D{Expectation Poller}
|
|
288
|
-
D -->|Match Found| E[ADB Fast Native Driver]
|
|
289
|
-
D -->|Not Found| F[Pre-Fail Triage / Auto-Scroll]
|
|
290
|
-
F -->|Recovered| E
|
|
291
|
-
F -->|Failed| G[Diagnostic Blackbox Snapshot]
|
|
292
|
-
E --> H[Screen Execution & Settle]
|
|
293
|
-
H --> I[Autonomous Knowledge Memory Engine]
|
|
294
|
-
I --> J[Markdown & JSON QA Reports]
|
|
295
|
-
```
|
|
161
|
+
## π Plain-English Syntax Reference
|
|
296
162
|
|
|
297
|
-
|
|
163
|
+
| Interaction | Example Syntax | Description |
|
|
164
|
+
| :--- | :--- | :--- |
|
|
165
|
+
| **Tap / Click** | `Tap 'Sign In'` | Clicks buttons, icons, tabs, or links by text or accessibility label. |
|
|
166
|
+
| **Text Input** | `Type 'test@example.com' into 'Email'` | Focuses input, clears existing text, and types value safely. |
|
|
167
|
+
| **Assertions** | `Verify 'Dashboard' is visible` | Polls until the element appears on screen. |
|
|
168
|
+
| **Negative Check** | `Verify 'Loading...' is not visible` | Confirms an element or spinner has dismissed. |
|
|
169
|
+
| **Relative Tap** | `Tap 'Delete' next to 'Order #12'` | Disambiguates duplicate elements using spatial proximity. |
|
|
170
|
+
| **Gestures** | `Scroll down`, `Scroll up`, `Swipe left` | Responsive proportional swipe gestures across the viewport. |
|
|
171
|
+
| **System Keys** | `Press back`, `Press home` | Emits hardware key events (`KEYCODE_BACK`, etc.). |
|
|
172
|
+
| **Pauses** | `Wait 2s` or `Wait 1500ms` | Pauses execution for custom animation delays. |
|
|
173
|
+
| **Conditionals** | `Tap 'Dismiss' (if present)` | Executes step only if target element is visible without failing. |
|
|
298
174
|
|
|
299
|
-
|
|
300
|
-
2. **Dynamic Expectation Poller (`lib/prompt-runner.ts`)**: Instant 0ms cache pass when screens match, dynamically polling for state transitions without rigid sleep constants.
|
|
301
|
-
3. **Autonomous Knowledge Engine (`lib/memory.ts`)**: Persists successful UI mappings into `memory/<package>.json` to accelerate future runs.
|
|
302
|
-
4. **Visual QA Sentinel (`lib/crawler.ts`)**: Scans for layout overflow, overlapping views, and unlabelled accessibility nodes.
|
|
175
|
+
*For advanced syntax (loops, regex assertions, data-driven iterations, subflow `#include`), check the **[Comprehensive User Manual](docs/USER_MANUAL.md)**.*
|
|
303
176
|
|
|
304
177
|
---
|
|
305
178
|
|
|
306
|
-
##
|
|
179
|
+
## π» Programmatic TypeScript SDK
|
|
307
180
|
|
|
308
|
-
|
|
181
|
+
PromptTest exports a full programmatic SDK for embedding into Node.js test runners or custom CI scripts:
|
|
309
182
|
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
| **Teacher Role Suite** | 36 | 36 | 0 | **100%** | 64.0s |
|
|
313
|
-
| **Student Role Suite** | 24 | 24 | 0 | **100%** | 44.6s |
|
|
314
|
-
| **Total Benchmark** | **60** | **60** | **0** | **100%** | **108.6s** |
|
|
183
|
+
```typescript
|
|
184
|
+
import { createPromptRunner, createAndroidDriver } from 'prompttest';
|
|
315
185
|
|
|
316
|
-
|
|
186
|
+
// Initialize native ADB driver for device
|
|
187
|
+
const driver = createAndroidDriver('DEVICE_SERIAL');
|
|
188
|
+
const runner = createPromptRunner(driver);
|
|
317
189
|
|
|
318
|
-
|
|
190
|
+
// Execute spec and retrieve structured execution report
|
|
191
|
+
const report = await runner.runSpec('specs/login.txt', 'com.example.app', {
|
|
192
|
+
fresh: true,
|
|
193
|
+
screenshots: 'failure-only',
|
|
194
|
+
});
|
|
319
195
|
|
|
320
|
-
|
|
321
|
-
prompttest/
|
|
322
|
-
βββ bin/
|
|
323
|
-
β βββ prompttest.ts # CLI Entrypoint & Argument Router
|
|
324
|
-
βββ constants/
|
|
325
|
-
β βββ commands.ts # ADB Commands & Keycode Constants
|
|
326
|
-
βββ docs/
|
|
327
|
-
β βββ ACTION_ITEMS.md # Working fix backlog (Phase 23 findings)
|
|
328
|
-
β βββ MODE_EXPECTATIONS.md # Realistic capabilities, limits & boundaries per mode
|
|
329
|
-
β βββ ARCHITECTURE.md # System Architecture & Design
|
|
330
|
-
β βββ USER_MANUAL.md # Comprehensive User Guide
|
|
331
|
-
β βββ adr/ # Architecture Decision Records
|
|
332
|
-
β βββ IMPLEMENTATION_PLAN.md
|
|
333
|
-
βββ lib/
|
|
334
|
-
β βββ adb.ts # ADB Interface & Streaming Driver
|
|
335
|
-
β βββ baseline.ts # Visual Baseline Diffing & Masking
|
|
336
|
-
β βββ checkpoint.ts # Test Progress Checkpointing
|
|
337
|
-
β βββ config.ts # Configuration Constants
|
|
338
|
-
β βββ crawler.ts # Visual Defect Detection & Screen Crawling
|
|
339
|
-
β βββ data-loader.ts # Data-Driven Iterations & Dynamic Generators
|
|
340
|
-
β βββ dictionary.ts # UI Taxonomy & Intent Dictionary
|
|
341
|
-
β βββ explorer.ts # Autonomous Exploration Graph
|
|
342
|
-
β βββ form-filler.ts # Smart Form Filling Engine
|
|
343
|
-
β βββ live-monitor.ts # Real-Time Event Streaming Server
|
|
344
|
-
β βββ lock.ts # File-Locking Device Leasing Mutex
|
|
345
|
-
β βββ logger.ts # Structured Logging
|
|
346
|
-
β βββ memory.ts # Persistent Knowledge & Fast-Forward Engine
|
|
347
|
-
β βββ patterns.ts # Plain-English Grammar Parser
|
|
348
|
-
β βββ prompt-runner.ts # Orchestration Engine (<1000 lines)
|
|
349
|
-
β βββ recorder.ts # Autonomous Record & Replay
|
|
350
|
-
β βββ reporter.ts # Multi-Format Report Generator (HTML/JSON/JUnit)
|
|
351
|
-
β βββ runner-utils.ts # AST Parsing, Modular Includes, Diagnostic Errors
|
|
352
|
-
β βββ step-handlers.ts # Decoupled Step Execution Handlers
|
|
353
|
-
βββ memory/ # Autonomous Learned Knowledge Mappings
|
|
354
|
-
βββ output/ # Generated Test Reports & Artifacts
|
|
355
|
-
βββ specs/ # Plain-English Test Specification Files
|
|
356
|
-
βββ tests/ # Vitest Automated Test Suites
|
|
357
|
-
βββ index.ts # Programmatic API Entrypoint
|
|
358
|
-
βββ package.json
|
|
359
|
-
βββ tsconfig.json
|
|
360
|
-
βββ CONTRIBUTING.md
|
|
361
|
-
βββ LICENSE
|
|
362
|
-
βββ README.md
|
|
196
|
+
console.log(`Passed: ${report.passedSteps}/${report.totalSteps}`);
|
|
363
197
|
```
|
|
364
198
|
|
|
365
199
|
---
|
|
366
200
|
|
|
367
|
-
##
|
|
368
|
-
|
|
369
|
-
You can use PromptTest programmatically within your own Node.js scripts using the unified `DriverInterface`:
|
|
370
|
-
|
|
371
|
-
```typescript
|
|
372
|
-
import { createPromptRunner, createAndroidDriver, createIosDriver } from 'prompttest';
|
|
201
|
+
## π Proven Real-Device Benchmarks
|
|
373
202
|
|
|
374
|
-
|
|
375
|
-
const androidDriver = createAndroidDriver('DEVICE_SERIAL');
|
|
376
|
-
const androidRunner = createPromptRunner(androidDriver);
|
|
377
|
-
const androidReport = await androidRunner.runSpec('specs/login.txt', 'com.example.app');
|
|
203
|
+
Tested and verified against live physical Android devices running complex multi-role enterprise apps:
|
|
378
204
|
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
205
|
+
| Test Suite | Total Steps | Passed | Failed | Pass Rate | Execution Time |
|
|
206
|
+
| :--- | :---: | :---: | :---: | :---: | :---: |
|
|
207
|
+
| **CoachConnect Enterprise Suite** | 52 | 52 | 0 | **100%** | 47.8s |
|
|
208
|
+
| **Govindam Multilingual Suite** | 8 | 8 | 0 | **100%** | 35.1s |
|
|
209
|
+
| **Calculator Operations Suite** | 5 | 5 | 0 | **100%** | 19.6s |
|
|
210
|
+
| **Automated Vitest Test Matrix** | 294 | 294 | 0 | **100%** | 14.8s |
|
|
384
211
|
|
|
385
|
-
|
|
212
|
+
## π³ Docker Container Deployment
|
|
386
213
|
|
|
387
|
-
|
|
214
|
+
PromptTest is containerized with Android platform-tools and headless runtime support:
|
|
388
215
|
|
|
389
|
-
|
|
390
|
-
|
|
216
|
+
```bash
|
|
217
|
+
docker build -t prompttest .
|
|
218
|
+
docker run --rm --net=host -v $(pwd)/specs:/app/specs prompttest run specs/flow.txt com.example.app
|
|
219
|
+
```
|
|
391
220
|
|
|
392
221
|
---
|
|
393
222
|
|
|
394
|
-
##
|
|
223
|
+
## π Packaged Artifacts & Documentation
|
|
395
224
|
|
|
396
|
-
|
|
225
|
+
- **[docs/USER_MANUAL.md](docs/USER_MANUAL.md)**: Exhaustive reference manual covering all keywords, configurations, edge-case triage, and real-world recipes.
|
|
226
|
+
- **[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)**: Deep dive into the ADB streaming driver, state-graph crawler, and memory indexing engine.
|
|
227
|
+
- **[LICENSE](LICENSE)**: Legal terms under the Business Source License (BSL 1.1).
|
|
228
|
+
- **[CONTRIBUTING.md](CONTRIBUTING.md)**: Guidelines for contributing bug fixes and enhancements.
|
|
229
|
+
- **[SECURITY.md](SECURITY.md)**: Security vulnerability disclosure policy.
|
|
397
230
|
|
|
398
231
|
---
|
|
399
232
|
|
|
400
233
|
## π License
|
|
401
234
|
|
|
402
|
-
|
|
235
|
+
PromptTest is licensed under the **Business Source License 1.1 (BSL 1.1)**:
|
|
236
|
+
|
|
237
|
+
- **Free for Community Use**: 100% free for educational use, individual developers, open-source projects, and startups/businesses with under $100,000 in annual revenue.
|
|
238
|
+
- **Commercial License**: Required for organizations exceeding the annual revenue threshold or companies wrapping PromptTest into hosted cloud services.
|
|
239
|
+
- See the full terms in **[LICENSE](LICENSE)**. For commercial licensing inquiries, contact `jairam.singh9@gmail.com`.
|