@qate/cli 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +449 -449
- package/dist/AuthFlowExecutor.d.ts +87 -0
- package/dist/AuthFlowExecutor.d.ts.map +1 -0
- package/dist/AuthFlowExecutor.js +351 -0
- package/dist/AuthFlowExecutor.js.map +1 -0
- package/dist/AxiosGenerator.js +28 -28
- package/dist/JunitXmlGenerator.d.ts +12 -0
- package/dist/JunitXmlGenerator.d.ts.map +1 -0
- package/dist/JunitXmlGenerator.js +114 -0
- package/dist/JunitXmlGenerator.js.map +1 -0
- package/dist/PlaywrightGenerator.d.ts +1 -1
- package/dist/PlaywrightGenerator.d.ts.map +1 -1
- package/dist/PlaywrightGenerator.js +796 -758
- package/dist/PlaywrightGenerator.js.map +1 -1
- package/dist/RestApiExecutor.d.ts +3 -4
- package/dist/RestApiExecutor.d.ts.map +1 -1
- package/dist/RestApiExecutor.js +63 -3
- package/dist/RestApiExecutor.js.map +1 -1
- package/dist/cli.js +536 -148
- package/dist/cli.js.map +1 -1
- package/dist/client.d.ts +78 -5
- package/dist/client.d.ts.map +1 -1
- package/dist/client.js +47 -6
- package/dist/client.js.map +1 -1
- package/package.json +71 -71
package/README.md
CHANGED
|
@@ -1,449 +1,449 @@
|
|
|
1
|
-
# @qate/cli
|
|
2
|
-
|
|
3
|
-
Qate CLI for CI/CD pipeline integration. Run test sets and test sequences from your CI pipelines.
|
|
4
|
-
|
|
5
|
-
## Installation
|
|
6
|
-
|
|
7
|
-
```bash
|
|
8
|
-
npm install -g @qate/cli
|
|
9
|
-
```
|
|
10
|
-
|
|
11
|
-
## Authentication
|
|
12
|
-
|
|
13
|
-
Generate an API key from Settings > CI/CD API Keys in the Qate dashboard.
|
|
14
|
-
|
|
15
|
-
Set the API key as an environment variable:
|
|
16
|
-
```bash
|
|
17
|
-
export QATE_API_KEY=qate_xxxxxxxxxxxx
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
Or pass it with each command:
|
|
21
|
-
```bash
|
|
22
|
-
qate run -n "My Test Set" --api-key qate_xxxxxxxxxxxx
|
|
23
|
-
```
|
|
24
|
-
|
|
25
|
-
## Commands
|
|
26
|
-
|
|
27
|
-
### Test Sets (Parallel Execution)
|
|
28
|
-
|
|
29
|
-
Test sets run all tests in parallel for maximum speed.
|
|
30
|
-
|
|
31
|
-
```bash
|
|
32
|
-
# List all test sets
|
|
33
|
-
qate list:testsets
|
|
34
|
-
qate list # alias
|
|
35
|
-
|
|
36
|
-
# Run a test set
|
|
37
|
-
qate run:testset -n "Regression Suite"
|
|
38
|
-
qate run -n "Regression Suite" # alias
|
|
39
|
-
|
|
40
|
-
# Run and wait for completion
|
|
41
|
-
qate run -n "Regression Suite" --wait
|
|
42
|
-
|
|
43
|
-
# Run with timeout (default: 600s)
|
|
44
|
-
qate run -n "Regression Suite" --wait --timeout 300
|
|
45
|
-
|
|
46
|
-
# Check execution status
|
|
47
|
-
qate status:testset -e ci_xxx_xxx
|
|
48
|
-
|
|
49
|
-
# Export test set definition
|
|
50
|
-
qate export:testset -n "Regression Suite" -o tests.json
|
|
51
|
-
```
|
|
52
|
-
|
|
53
|
-
### Test Sequences (Sequential Execution)
|
|
54
|
-
|
|
55
|
-
Test sequences run tests in order, optionally sharing browser state between tests.
|
|
56
|
-
|
|
57
|
-
```bash
|
|
58
|
-
# List all test sequences
|
|
59
|
-
qate list:sequences
|
|
60
|
-
|
|
61
|
-
# Run a test sequence
|
|
62
|
-
qate run:sequence -n "User Flow"
|
|
63
|
-
|
|
64
|
-
# Run and wait for completion
|
|
65
|
-
qate run:sequence -n "User Flow" --wait
|
|
66
|
-
|
|
67
|
-
# Check execution status
|
|
68
|
-
qate status:sequence -e seq_xxx_xxx
|
|
69
|
-
|
|
70
|
-
# Export test sequence definition
|
|
71
|
-
qate export:sequence -n "User Flow" -o sequence.json
|
|
72
|
-
```
|
|
73
|
-
|
|
74
|
-
### Generic Status Command
|
|
75
|
-
|
|
76
|
-
Auto-detects whether the execution is a test set or sequence based on the ID prefix:
|
|
77
|
-
|
|
78
|
-
```bash
|
|
79
|
-
qate status -e ci_xxx_xxx # Test set
|
|
80
|
-
qate status -e seq_xxx_xxx # Test sequence
|
|
81
|
-
```
|
|
82
|
-
|
|
83
|
-
### Generate Playwright Tests
|
|
84
|
-
|
|
85
|
-
Generate standalone Playwright test files from your Qate tests. These can be run locally or in your CI pipeline.
|
|
86
|
-
|
|
87
|
-
```bash
|
|
88
|
-
# Generate from a test set
|
|
89
|
-
qate generate -n "Regression Suite" -o ./e2e-tests
|
|
90
|
-
qate generate:testset -n "Regression Suite" -o ./e2e-tests # explicit
|
|
91
|
-
|
|
92
|
-
# Generate from a test sequence
|
|
93
|
-
qate generate:sequence -n "User Flow" -o ./e2e-tests
|
|
94
|
-
|
|
95
|
-
# Specify provider (for cloud testing)
|
|
96
|
-
qate generate -n "Regression Suite" -o ./e2e-tests --provider browserstack
|
|
97
|
-
qate generate -n "Regression Suite" -o ./e2e-tests --provider saucelabs
|
|
98
|
-
qate generate -n "Regression Suite" -o ./e2e-tests --provider lambdatest
|
|
99
|
-
|
|
100
|
-
# Specify browsers
|
|
101
|
-
qate generate -n "Regression Suite" -o ./e2e-tests --browsers "chromium,firefox,webkit"
|
|
102
|
-
```
|
|
103
|
-
|
|
104
|
-
After generating, run the tests:
|
|
105
|
-
|
|
106
|
-
```bash
|
|
107
|
-
cd ./e2e-tests
|
|
108
|
-
npm install
|
|
109
|
-
npx playwright install
|
|
110
|
-
npx playwright test
|
|
111
|
-
```
|
|
112
|
-
|
|
113
|
-
## CI/CD Examples
|
|
114
|
-
|
|
115
|
-
### GitHub Actions (Local Playwright Execution)
|
|
116
|
-
|
|
117
|
-
Run tests locally in your CI using generated Playwright tests:
|
|
118
|
-
|
|
119
|
-
```yaml
|
|
120
|
-
name: E2E Tests
|
|
121
|
-
on: [push, pull_request]
|
|
122
|
-
|
|
123
|
-
jobs:
|
|
124
|
-
test:
|
|
125
|
-
runs-on: ubuntu-latest
|
|
126
|
-
steps:
|
|
127
|
-
- uses: actions/checkout@v4
|
|
128
|
-
|
|
129
|
-
- name: Setup Node.js
|
|
130
|
-
uses: actions/setup-node@v4
|
|
131
|
-
with:
|
|
132
|
-
node-version: '20'
|
|
133
|
-
|
|
134
|
-
- name: Install Qate CLI
|
|
135
|
-
run: npm install -g @qate/cli
|
|
136
|
-
|
|
137
|
-
- name: Generate Playwright Tests
|
|
138
|
-
env:
|
|
139
|
-
QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
|
|
140
|
-
run: qate generate -n "Regression Suite" -o ./e2e
|
|
141
|
-
|
|
142
|
-
- name: Install Dependencies
|
|
143
|
-
working-directory: ./e2e
|
|
144
|
-
run: npm ci
|
|
145
|
-
|
|
146
|
-
- name: Install Playwright Browsers
|
|
147
|
-
working-directory: ./e2e
|
|
148
|
-
run: npx playwright install --with-deps chromium
|
|
149
|
-
|
|
150
|
-
- name: Run Tests
|
|
151
|
-
working-directory: ./e2e
|
|
152
|
-
run: npx playwright test
|
|
153
|
-
|
|
154
|
-
- name: Upload Test Results
|
|
155
|
-
uses: actions/upload-artifact@v4
|
|
156
|
-
if: always()
|
|
157
|
-
with:
|
|
158
|
-
name: playwright-report
|
|
159
|
-
path: e2e/playwright-report/
|
|
160
|
-
```
|
|
161
|
-
|
|
162
|
-
### GitHub Actions (BrowserStack)
|
|
163
|
-
|
|
164
|
-
Run tests on BrowserStack's cloud infrastructure:
|
|
165
|
-
|
|
166
|
-
```yaml
|
|
167
|
-
name: E2E Tests (BrowserStack)
|
|
168
|
-
on: [push, pull_request]
|
|
169
|
-
|
|
170
|
-
jobs:
|
|
171
|
-
test:
|
|
172
|
-
runs-on: ubuntu-latest
|
|
173
|
-
steps:
|
|
174
|
-
- uses: actions/checkout@v4
|
|
175
|
-
|
|
176
|
-
- name: Setup Node.js
|
|
177
|
-
uses: actions/setup-node@v4
|
|
178
|
-
with:
|
|
179
|
-
node-version: '20'
|
|
180
|
-
|
|
181
|
-
- name: Install Qate CLI
|
|
182
|
-
run: npm install -g @qate/cli
|
|
183
|
-
|
|
184
|
-
- name: Generate Playwright Tests
|
|
185
|
-
env:
|
|
186
|
-
QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
|
|
187
|
-
run: qate generate -n "Regression Suite" -o ./e2e --provider browserstack
|
|
188
|
-
|
|
189
|
-
- name: Install Dependencies
|
|
190
|
-
working-directory: ./e2e
|
|
191
|
-
run: npm ci
|
|
192
|
-
|
|
193
|
-
- name: Run Tests on BrowserStack
|
|
194
|
-
working-directory: ./e2e
|
|
195
|
-
env:
|
|
196
|
-
BROWSERSTACK_USERNAME: ${{ secrets.BROWSERSTACK_USERNAME }}
|
|
197
|
-
BROWSERSTACK_ACCESS_KEY: ${{ secrets.BROWSERSTACK_ACCESS_KEY }}
|
|
198
|
-
run: npx playwright test
|
|
199
|
-
```
|
|
200
|
-
|
|
201
|
-
### GitHub Actions (LambdaTest)
|
|
202
|
-
|
|
203
|
-
Run tests on LambdaTest's cloud infrastructure:
|
|
204
|
-
|
|
205
|
-
```yaml
|
|
206
|
-
name: E2E Tests (LambdaTest)
|
|
207
|
-
on: [push, pull_request]
|
|
208
|
-
|
|
209
|
-
jobs:
|
|
210
|
-
test:
|
|
211
|
-
runs-on: ubuntu-latest
|
|
212
|
-
steps:
|
|
213
|
-
- uses: actions/checkout@v4
|
|
214
|
-
|
|
215
|
-
- name: Setup Node.js
|
|
216
|
-
uses: actions/setup-node@v4
|
|
217
|
-
with:
|
|
218
|
-
node-version: '20'
|
|
219
|
-
|
|
220
|
-
- name: Install Qate CLI
|
|
221
|
-
run: npm install -g @qate/cli
|
|
222
|
-
|
|
223
|
-
- name: Generate Playwright Tests
|
|
224
|
-
env:
|
|
225
|
-
QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
|
|
226
|
-
run: qate generate -n "Regression Suite" -o ./e2e --provider lambdatest
|
|
227
|
-
|
|
228
|
-
- name: Install Dependencies
|
|
229
|
-
working-directory: ./e2e
|
|
230
|
-
run: npm ci
|
|
231
|
-
|
|
232
|
-
- name: Run Tests on LambdaTest
|
|
233
|
-
working-directory: ./e2e
|
|
234
|
-
env:
|
|
235
|
-
LT_USERNAME: ${{ secrets.LT_USERNAME }}
|
|
236
|
-
LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
|
|
237
|
-
run: npx playwright test
|
|
238
|
-
```
|
|
239
|
-
|
|
240
|
-
### GitHub Actions (Sauce Labs)
|
|
241
|
-
|
|
242
|
-
Run tests on Sauce Labs using saucectl:
|
|
243
|
-
|
|
244
|
-
```yaml
|
|
245
|
-
name: E2E Tests (Sauce Labs)
|
|
246
|
-
on: [push, pull_request]
|
|
247
|
-
|
|
248
|
-
jobs:
|
|
249
|
-
test:
|
|
250
|
-
runs-on: ubuntu-latest
|
|
251
|
-
steps:
|
|
252
|
-
- uses: actions/checkout@v4
|
|
253
|
-
|
|
254
|
-
- name: Setup Node.js
|
|
255
|
-
uses: actions/setup-node@v4
|
|
256
|
-
with:
|
|
257
|
-
node-version: '20'
|
|
258
|
-
|
|
259
|
-
- name: Install Qate CLI
|
|
260
|
-
run: npm install -g @qate/cli
|
|
261
|
-
|
|
262
|
-
- name: Generate Playwright Tests
|
|
263
|
-
env:
|
|
264
|
-
QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
|
|
265
|
-
run: qate generate -n "Regression Suite" -o ./e2e --provider saucelabs
|
|
266
|
-
|
|
267
|
-
- name: Install Dependencies
|
|
268
|
-
working-directory: ./e2e
|
|
269
|
-
run: npm ci
|
|
270
|
-
|
|
271
|
-
- name: Install saucectl
|
|
272
|
-
run: npm install -g saucectl
|
|
273
|
-
|
|
274
|
-
- name: Run Tests on Sauce Labs
|
|
275
|
-
working-directory: ./e2e
|
|
276
|
-
env:
|
|
277
|
-
SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
|
|
278
|
-
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
|
|
279
|
-
run: saucectl run
|
|
280
|
-
```
|
|
281
|
-
|
|
282
|
-
> **Note:** Sauce Labs uses `saucectl` CLI instead of direct Playwright execution. The generated `.sauce/config.yml` file contains the test configuration.
|
|
283
|
-
|
|
284
|
-
### GitLab CI
|
|
285
|
-
|
|
286
|
-
```yaml
|
|
287
|
-
e2e-tests:
|
|
288
|
-
stage: test
|
|
289
|
-
image: mcr.microsoft.com/playwright:v1.40.0-jammy
|
|
290
|
-
script:
|
|
291
|
-
- npm install -g @qate/cli
|
|
292
|
-
- qate generate -n "Regression Suite" -o ./e2e
|
|
293
|
-
- cd e2e && npm ci
|
|
294
|
-
- npx playwright test
|
|
295
|
-
variables:
|
|
296
|
-
QATE_API_KEY: $QATE_API_KEY
|
|
297
|
-
artifacts:
|
|
298
|
-
when: always
|
|
299
|
-
paths:
|
|
300
|
-
- e2e/playwright-report/
|
|
301
|
-
expire_in: 1 week
|
|
302
|
-
```
|
|
303
|
-
|
|
304
|
-
### Azure DevOps
|
|
305
|
-
|
|
306
|
-
```yaml
|
|
307
|
-
- task: NodeTool@0
|
|
308
|
-
inputs:
|
|
309
|
-
versionSpec: '20.x'
|
|
310
|
-
|
|
311
|
-
- script: npm install -g @qate/cli
|
|
312
|
-
displayName: 'Install Qate CLI'
|
|
313
|
-
|
|
314
|
-
- script: qate generate -n "Regression Suite" -o ./e2e
|
|
315
|
-
displayName: 'Generate Playwright Tests'
|
|
316
|
-
env:
|
|
317
|
-
QATE_API_KEY: $(QATE_API_KEY)
|
|
318
|
-
|
|
319
|
-
- script: |
|
|
320
|
-
cd e2e
|
|
321
|
-
npm ci
|
|
322
|
-
npx playwright install --with-deps chromium
|
|
323
|
-
npx playwright test
|
|
324
|
-
displayName: 'Run E2E Tests'
|
|
325
|
-
|
|
326
|
-
- task: PublishTestResults@2
|
|
327
|
-
inputs:
|
|
328
|
-
testResultsFiles: 'e2e/test-results.json'
|
|
329
|
-
condition: always()
|
|
330
|
-
```
|
|
331
|
-
|
|
332
|
-
### Jenkins
|
|
333
|
-
|
|
334
|
-
```groovy
|
|
335
|
-
pipeline {
|
|
336
|
-
agent any
|
|
337
|
-
environment {
|
|
338
|
-
QATE_API_KEY = credentials('qate-api-key')
|
|
339
|
-
}
|
|
340
|
-
stages {
|
|
341
|
-
stage('Generate Tests') {
|
|
342
|
-
steps {
|
|
343
|
-
sh 'npm install -g @qate/cli'
|
|
344
|
-
sh 'qate generate -n "Regression Suite" -o ./e2e'
|
|
345
|
-
}
|
|
346
|
-
}
|
|
347
|
-
stage('Run E2E Tests') {
|
|
348
|
-
steps {
|
|
349
|
-
dir('e2e') {
|
|
350
|
-
sh 'npm ci'
|
|
351
|
-
sh 'npx playwright install --with-deps chromium'
|
|
352
|
-
sh 'npx playwright test'
|
|
353
|
-
}
|
|
354
|
-
}
|
|
355
|
-
}
|
|
356
|
-
}
|
|
357
|
-
post {
|
|
358
|
-
always {
|
|
359
|
-
archiveArtifacts artifacts: 'e2e/playwright-report/**', allowEmptyArchive: true
|
|
360
|
-
}
|
|
361
|
-
}
|
|
362
|
-
}
|
|
363
|
-
```
|
|
364
|
-
|
|
365
|
-
## Options
|
|
366
|
-
|
|
367
|
-
### Global Options
|
|
368
|
-
|
|
369
|
-
| Option | Description |
|
|
370
|
-
|--------|-------------|
|
|
371
|
-
| `-a, --api-key <key>` | API key (or use QATE_API_KEY env var) |
|
|
372
|
-
| `-u, --url <url>` | Qate API URL (default: https://api.qate.ai) |
|
|
373
|
-
| `--json` | Output results as JSON |
|
|
374
|
-
|
|
375
|
-
### Run Options
|
|
376
|
-
|
|
377
|
-
| Option | Description |
|
|
378
|
-
|--------|-------------|
|
|
379
|
-
| `-n, --name <name>` | Name of the test set/sequence to run |
|
|
380
|
-
| `--app <id>` | Application ID (if name is not unique) |
|
|
381
|
-
| `-w, --wait` | Wait for execution to complete |
|
|
382
|
-
| `--timeout <seconds>` | Timeout for waiting (default: 600) |
|
|
383
|
-
|
|
384
|
-
### Export Options
|
|
385
|
-
|
|
386
|
-
| Option | Description |
|
|
387
|
-
|--------|-------------|
|
|
388
|
-
| `-n, --name <name>` | Name of the test set/sequence to export |
|
|
389
|
-
| `-o, --output <file>` | Output file path (default: stdout) |
|
|
390
|
-
| `--app <id>` | Application ID (if name is not unique) |
|
|
391
|
-
|
|
392
|
-
### Generate Options
|
|
393
|
-
|
|
394
|
-
| Option | Description |
|
|
395
|
-
|--------|-------------|
|
|
396
|
-
| `-n, --name <name>` | Name of the test set/sequence |
|
|
397
|
-
| `-o, --output <dir>` | Output directory for generated files |
|
|
398
|
-
| `--app <id>` | Application ID (if name is not unique) |
|
|
399
|
-
| `--provider <provider>` | Test provider: local, browserstack, saucelabs, lambdatest (default: local) |
|
|
400
|
-
| `--browsers <browsers>` | Comma-separated list of browsers (default: chromium) |
|
|
401
|
-
|
|
402
|
-
## Exit Codes
|
|
403
|
-
|
|
404
|
-
| Code | Meaning |
|
|
405
|
-
|------|---------|
|
|
406
|
-
| 0 | Success (tests passed or still running) |
|
|
407
|
-
| 1 | Failure (tests failed or error occurred) |
|
|
408
|
-
|
|
409
|
-
## Test Sets vs Test Sequences
|
|
410
|
-
|
|
411
|
-
| Feature | Test Set | Test Sequence |
|
|
412
|
-
|---------|----------|---------------|
|
|
413
|
-
| Execution | Parallel | Sequential |
|
|
414
|
-
| Speed | Faster | Slower |
|
|
415
|
-
| State sharing | Independent | Optional |
|
|
416
|
-
| Use case | Regression testing | User flows, E2E journeys |
|
|
417
|
-
| Stop on failure | No | Configurable |
|
|
418
|
-
|
|
419
|
-
## Windows Desktop Applications
|
|
420
|
-
|
|
421
|
-
For Windows desktop applications, tests are executed on a connected desktop agent. The CLI triggers server-side execution and polls for results.
|
|
422
|
-
|
|
423
|
-
```bash
|
|
424
|
-
# Run tests on a connected desktop agent
|
|
425
|
-
qate run -n "Desktop App Tests"
|
|
426
|
-
|
|
427
|
-
# Run a sequence on a connected desktop agent
|
|
428
|
-
qate run:sequence -n "Desktop Flow"
|
|
429
|
-
|
|
430
|
-
# Install a build on the agent machine before running tests
|
|
431
|
-
qate install --app <applicationId> --path "C:\builds\MyApp-Setup.exe" --args "/silent"
|
|
432
|
-
|
|
433
|
-
# Full CI pipeline: install build, then run tests
|
|
434
|
-
qate install --app 64a... --path "C:\builds\setup.exe" --args "/S"
|
|
435
|
-
qate run -n "Smoke Tests" --app 64a...
|
|
436
|
-
```
|
|
437
|
-
|
|
438
|
-
### Install Command Options
|
|
439
|
-
|
|
440
|
-
| Option | Description |
|
|
441
|
-
|--------|-------------|
|
|
442
|
-
| `--app <id>` | Application ID (required) |
|
|
443
|
-
| `--path <path>` | Path to installer on the agent machine (required) |
|
|
444
|
-
| `--args <arguments>` | Installer arguments (e.g., `/silent`, `/S`) |
|
|
445
|
-
| `--timeout <seconds>` | Timeout in seconds (default: 300) |
|
|
446
|
-
|
|
447
|
-
## License
|
|
448
|
-
|
|
449
|
-
MIT
|
|
1
|
+
# @qate/cli
|
|
2
|
+
|
|
3
|
+
Qate CLI for CI/CD pipeline integration. Run test sets and test sequences from your CI pipelines.
|
|
4
|
+
|
|
5
|
+
## Installation
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
npm install -g @qate/cli
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
## Authentication
|
|
12
|
+
|
|
13
|
+
Generate an API key from Settings > CI/CD API Keys in the Qate dashboard.
|
|
14
|
+
|
|
15
|
+
Set the API key as an environment variable:
|
|
16
|
+
```bash
|
|
17
|
+
export QATE_API_KEY=qate_xxxxxxxxxxxx
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
Or pass it with each command:
|
|
21
|
+
```bash
|
|
22
|
+
qate run -n "My Test Set" --api-key qate_xxxxxxxxxxxx
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Commands
|
|
26
|
+
|
|
27
|
+
### Test Sets (Parallel Execution)
|
|
28
|
+
|
|
29
|
+
Test sets run all tests in parallel for maximum speed.
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
# List all test sets
|
|
33
|
+
qate list:testsets
|
|
34
|
+
qate list # alias
|
|
35
|
+
|
|
36
|
+
# Run a test set
|
|
37
|
+
qate run:testset -n "Regression Suite"
|
|
38
|
+
qate run -n "Regression Suite" # alias
|
|
39
|
+
|
|
40
|
+
# Run and wait for completion
|
|
41
|
+
qate run -n "Regression Suite" --wait
|
|
42
|
+
|
|
43
|
+
# Run with timeout (default: 600s)
|
|
44
|
+
qate run -n "Regression Suite" --wait --timeout 300
|
|
45
|
+
|
|
46
|
+
# Check execution status
|
|
47
|
+
qate status:testset -e ci_xxx_xxx
|
|
48
|
+
|
|
49
|
+
# Export test set definition
|
|
50
|
+
qate export:testset -n "Regression Suite" -o tests.json
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
### Test Sequences (Sequential Execution)
|
|
54
|
+
|
|
55
|
+
Test sequences run tests in order, optionally sharing browser state between tests.
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
# List all test sequences
|
|
59
|
+
qate list:sequences
|
|
60
|
+
|
|
61
|
+
# Run a test sequence
|
|
62
|
+
qate run:sequence -n "User Flow"
|
|
63
|
+
|
|
64
|
+
# Run and wait for completion
|
|
65
|
+
qate run:sequence -n "User Flow" --wait
|
|
66
|
+
|
|
67
|
+
# Check execution status
|
|
68
|
+
qate status:sequence -e seq_xxx_xxx
|
|
69
|
+
|
|
70
|
+
# Export test sequence definition
|
|
71
|
+
qate export:sequence -n "User Flow" -o sequence.json
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
### Generic Status Command
|
|
75
|
+
|
|
76
|
+
Auto-detects whether the execution is a test set or sequence based on the ID prefix:
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
qate status -e ci_xxx_xxx # Test set
|
|
80
|
+
qate status -e seq_xxx_xxx # Test sequence
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
### Generate Playwright Tests
|
|
84
|
+
|
|
85
|
+
Generate standalone Playwright test files from your Qate tests. These can be run locally or in your CI pipeline.
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
# Generate from a test set
|
|
89
|
+
qate generate -n "Regression Suite" -o ./e2e-tests
|
|
90
|
+
qate generate:testset -n "Regression Suite" -o ./e2e-tests # explicit
|
|
91
|
+
|
|
92
|
+
# Generate from a test sequence
|
|
93
|
+
qate generate:sequence -n "User Flow" -o ./e2e-tests
|
|
94
|
+
|
|
95
|
+
# Specify provider (for cloud testing)
|
|
96
|
+
qate generate -n "Regression Suite" -o ./e2e-tests --provider browserstack
|
|
97
|
+
qate generate -n "Regression Suite" -o ./e2e-tests --provider saucelabs
|
|
98
|
+
qate generate -n "Regression Suite" -o ./e2e-tests --provider lambdatest
|
|
99
|
+
|
|
100
|
+
# Specify browsers
|
|
101
|
+
qate generate -n "Regression Suite" -o ./e2e-tests --browsers "chromium,firefox,webkit"
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
After generating, run the tests:
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
cd ./e2e-tests
|
|
108
|
+
npm install
|
|
109
|
+
npx playwright install
|
|
110
|
+
npx playwright test
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
## CI/CD Examples
|
|
114
|
+
|
|
115
|
+
### GitHub Actions (Local Playwright Execution)
|
|
116
|
+
|
|
117
|
+
Run tests locally in your CI using generated Playwright tests:
|
|
118
|
+
|
|
119
|
+
```yaml
|
|
120
|
+
name: E2E Tests
|
|
121
|
+
on: [push, pull_request]
|
|
122
|
+
|
|
123
|
+
jobs:
|
|
124
|
+
test:
|
|
125
|
+
runs-on: ubuntu-latest
|
|
126
|
+
steps:
|
|
127
|
+
- uses: actions/checkout@v4
|
|
128
|
+
|
|
129
|
+
- name: Setup Node.js
|
|
130
|
+
uses: actions/setup-node@v4
|
|
131
|
+
with:
|
|
132
|
+
node-version: '20'
|
|
133
|
+
|
|
134
|
+
- name: Install Qate CLI
|
|
135
|
+
run: npm install -g @qate/cli
|
|
136
|
+
|
|
137
|
+
- name: Generate Playwright Tests
|
|
138
|
+
env:
|
|
139
|
+
QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
|
|
140
|
+
run: qate generate -n "Regression Suite" -o ./e2e
|
|
141
|
+
|
|
142
|
+
- name: Install Dependencies
|
|
143
|
+
working-directory: ./e2e
|
|
144
|
+
run: npm ci
|
|
145
|
+
|
|
146
|
+
- name: Install Playwright Browsers
|
|
147
|
+
working-directory: ./e2e
|
|
148
|
+
run: npx playwright install --with-deps chromium
|
|
149
|
+
|
|
150
|
+
- name: Run Tests
|
|
151
|
+
working-directory: ./e2e
|
|
152
|
+
run: npx playwright test
|
|
153
|
+
|
|
154
|
+
- name: Upload Test Results
|
|
155
|
+
uses: actions/upload-artifact@v4
|
|
156
|
+
if: always()
|
|
157
|
+
with:
|
|
158
|
+
name: playwright-report
|
|
159
|
+
path: e2e/playwright-report/
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
### GitHub Actions (BrowserStack)
|
|
163
|
+
|
|
164
|
+
Run tests on BrowserStack's cloud infrastructure:
|
|
165
|
+
|
|
166
|
+
```yaml
|
|
167
|
+
name: E2E Tests (BrowserStack)
|
|
168
|
+
on: [push, pull_request]
|
|
169
|
+
|
|
170
|
+
jobs:
|
|
171
|
+
test:
|
|
172
|
+
runs-on: ubuntu-latest
|
|
173
|
+
steps:
|
|
174
|
+
- uses: actions/checkout@v4
|
|
175
|
+
|
|
176
|
+
- name: Setup Node.js
|
|
177
|
+
uses: actions/setup-node@v4
|
|
178
|
+
with:
|
|
179
|
+
node-version: '20'
|
|
180
|
+
|
|
181
|
+
- name: Install Qate CLI
|
|
182
|
+
run: npm install -g @qate/cli
|
|
183
|
+
|
|
184
|
+
- name: Generate Playwright Tests
|
|
185
|
+
env:
|
|
186
|
+
QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
|
|
187
|
+
run: qate generate -n "Regression Suite" -o ./e2e --provider browserstack
|
|
188
|
+
|
|
189
|
+
- name: Install Dependencies
|
|
190
|
+
working-directory: ./e2e
|
|
191
|
+
run: npm ci
|
|
192
|
+
|
|
193
|
+
- name: Run Tests on BrowserStack
|
|
194
|
+
working-directory: ./e2e
|
|
195
|
+
env:
|
|
196
|
+
BROWSERSTACK_USERNAME: ${{ secrets.BROWSERSTACK_USERNAME }}
|
|
197
|
+
BROWSERSTACK_ACCESS_KEY: ${{ secrets.BROWSERSTACK_ACCESS_KEY }}
|
|
198
|
+
run: npx playwright test
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
### GitHub Actions (LambdaTest)
|
|
202
|
+
|
|
203
|
+
Run tests on LambdaTest's cloud infrastructure:
|
|
204
|
+
|
|
205
|
+
```yaml
|
|
206
|
+
name: E2E Tests (LambdaTest)
|
|
207
|
+
on: [push, pull_request]
|
|
208
|
+
|
|
209
|
+
jobs:
|
|
210
|
+
test:
|
|
211
|
+
runs-on: ubuntu-latest
|
|
212
|
+
steps:
|
|
213
|
+
- uses: actions/checkout@v4
|
|
214
|
+
|
|
215
|
+
- name: Setup Node.js
|
|
216
|
+
uses: actions/setup-node@v4
|
|
217
|
+
with:
|
|
218
|
+
node-version: '20'
|
|
219
|
+
|
|
220
|
+
- name: Install Qate CLI
|
|
221
|
+
run: npm install -g @qate/cli
|
|
222
|
+
|
|
223
|
+
- name: Generate Playwright Tests
|
|
224
|
+
env:
|
|
225
|
+
QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
|
|
226
|
+
run: qate generate -n "Regression Suite" -o ./e2e --provider lambdatest
|
|
227
|
+
|
|
228
|
+
- name: Install Dependencies
|
|
229
|
+
working-directory: ./e2e
|
|
230
|
+
run: npm ci
|
|
231
|
+
|
|
232
|
+
- name: Run Tests on LambdaTest
|
|
233
|
+
working-directory: ./e2e
|
|
234
|
+
env:
|
|
235
|
+
LT_USERNAME: ${{ secrets.LT_USERNAME }}
|
|
236
|
+
LT_ACCESS_KEY: ${{ secrets.LT_ACCESS_KEY }}
|
|
237
|
+
run: npx playwright test
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
### GitHub Actions (Sauce Labs)
|
|
241
|
+
|
|
242
|
+
Run tests on Sauce Labs using saucectl:
|
|
243
|
+
|
|
244
|
+
```yaml
|
|
245
|
+
name: E2E Tests (Sauce Labs)
|
|
246
|
+
on: [push, pull_request]
|
|
247
|
+
|
|
248
|
+
jobs:
|
|
249
|
+
test:
|
|
250
|
+
runs-on: ubuntu-latest
|
|
251
|
+
steps:
|
|
252
|
+
- uses: actions/checkout@v4
|
|
253
|
+
|
|
254
|
+
- name: Setup Node.js
|
|
255
|
+
uses: actions/setup-node@v4
|
|
256
|
+
with:
|
|
257
|
+
node-version: '20'
|
|
258
|
+
|
|
259
|
+
- name: Install Qate CLI
|
|
260
|
+
run: npm install -g @qate/cli
|
|
261
|
+
|
|
262
|
+
- name: Generate Playwright Tests
|
|
263
|
+
env:
|
|
264
|
+
QATE_API_KEY: ${{ secrets.QATE_API_KEY }}
|
|
265
|
+
run: qate generate -n "Regression Suite" -o ./e2e --provider saucelabs
|
|
266
|
+
|
|
267
|
+
- name: Install Dependencies
|
|
268
|
+
working-directory: ./e2e
|
|
269
|
+
run: npm ci
|
|
270
|
+
|
|
271
|
+
- name: Install saucectl
|
|
272
|
+
run: npm install -g saucectl
|
|
273
|
+
|
|
274
|
+
- name: Run Tests on Sauce Labs
|
|
275
|
+
working-directory: ./e2e
|
|
276
|
+
env:
|
|
277
|
+
SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }}
|
|
278
|
+
SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }}
|
|
279
|
+
run: saucectl run
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
> **Note:** Sauce Labs uses `saucectl` CLI instead of direct Playwright execution. The generated `.sauce/config.yml` file contains the test configuration.
|
|
283
|
+
|
|
284
|
+
### GitLab CI
|
|
285
|
+
|
|
286
|
+
```yaml
|
|
287
|
+
e2e-tests:
|
|
288
|
+
stage: test
|
|
289
|
+
image: mcr.microsoft.com/playwright:v1.40.0-jammy
|
|
290
|
+
script:
|
|
291
|
+
- npm install -g @qate/cli
|
|
292
|
+
- qate generate -n "Regression Suite" -o ./e2e
|
|
293
|
+
- cd e2e && npm ci
|
|
294
|
+
- npx playwright test
|
|
295
|
+
variables:
|
|
296
|
+
QATE_API_KEY: $QATE_API_KEY
|
|
297
|
+
artifacts:
|
|
298
|
+
when: always
|
|
299
|
+
paths:
|
|
300
|
+
- e2e/playwright-report/
|
|
301
|
+
expire_in: 1 week
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
### Azure DevOps
|
|
305
|
+
|
|
306
|
+
```yaml
|
|
307
|
+
- task: NodeTool@0
|
|
308
|
+
inputs:
|
|
309
|
+
versionSpec: '20.x'
|
|
310
|
+
|
|
311
|
+
- script: npm install -g @qate/cli
|
|
312
|
+
displayName: 'Install Qate CLI'
|
|
313
|
+
|
|
314
|
+
- script: qate generate -n "Regression Suite" -o ./e2e
|
|
315
|
+
displayName: 'Generate Playwright Tests'
|
|
316
|
+
env:
|
|
317
|
+
QATE_API_KEY: $(QATE_API_KEY)
|
|
318
|
+
|
|
319
|
+
- script: |
|
|
320
|
+
cd e2e
|
|
321
|
+
npm ci
|
|
322
|
+
npx playwright install --with-deps chromium
|
|
323
|
+
npx playwright test
|
|
324
|
+
displayName: 'Run E2E Tests'
|
|
325
|
+
|
|
326
|
+
- task: PublishTestResults@2
|
|
327
|
+
inputs:
|
|
328
|
+
testResultsFiles: 'e2e/test-results.json'
|
|
329
|
+
condition: always()
|
|
330
|
+
```
|
|
331
|
+
|
|
332
|
+
### Jenkins
|
|
333
|
+
|
|
334
|
+
```groovy
|
|
335
|
+
pipeline {
|
|
336
|
+
agent any
|
|
337
|
+
environment {
|
|
338
|
+
QATE_API_KEY = credentials('qate-api-key')
|
|
339
|
+
}
|
|
340
|
+
stages {
|
|
341
|
+
stage('Generate Tests') {
|
|
342
|
+
steps {
|
|
343
|
+
sh 'npm install -g @qate/cli'
|
|
344
|
+
sh 'qate generate -n "Regression Suite" -o ./e2e'
|
|
345
|
+
}
|
|
346
|
+
}
|
|
347
|
+
stage('Run E2E Tests') {
|
|
348
|
+
steps {
|
|
349
|
+
dir('e2e') {
|
|
350
|
+
sh 'npm ci'
|
|
351
|
+
sh 'npx playwright install --with-deps chromium'
|
|
352
|
+
sh 'npx playwright test'
|
|
353
|
+
}
|
|
354
|
+
}
|
|
355
|
+
}
|
|
356
|
+
}
|
|
357
|
+
post {
|
|
358
|
+
always {
|
|
359
|
+
archiveArtifacts artifacts: 'e2e/playwright-report/**', allowEmptyArchive: true
|
|
360
|
+
}
|
|
361
|
+
}
|
|
362
|
+
}
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
## Options
|
|
366
|
+
|
|
367
|
+
### Global Options
|
|
368
|
+
|
|
369
|
+
| Option | Description |
|
|
370
|
+
|--------|-------------|
|
|
371
|
+
| `-a, --api-key <key>` | API key (or use QATE_API_KEY env var) |
|
|
372
|
+
| `-u, --url <url>` | Qate API URL (default: https://api.qate.ai) |
|
|
373
|
+
| `--json` | Output results as JSON |
|
|
374
|
+
|
|
375
|
+
### Run Options
|
|
376
|
+
|
|
377
|
+
| Option | Description |
|
|
378
|
+
|--------|-------------|
|
|
379
|
+
| `-n, --name <name>` | Name of the test set/sequence to run |
|
|
380
|
+
| `--app <id>` | Application ID (if name is not unique) |
|
|
381
|
+
| `-w, --wait` | Wait for execution to complete |
|
|
382
|
+
| `--timeout <seconds>` | Timeout for waiting (default: 600) |
|
|
383
|
+
|
|
384
|
+
### Export Options
|
|
385
|
+
|
|
386
|
+
| Option | Description |
|
|
387
|
+
|--------|-------------|
|
|
388
|
+
| `-n, --name <name>` | Name of the test set/sequence to export |
|
|
389
|
+
| `-o, --output <file>` | Output file path (default: stdout) |
|
|
390
|
+
| `--app <id>` | Application ID (if name is not unique) |
|
|
391
|
+
|
|
392
|
+
### Generate Options
|
|
393
|
+
|
|
394
|
+
| Option | Description |
|
|
395
|
+
|--------|-------------|
|
|
396
|
+
| `-n, --name <name>` | Name of the test set/sequence |
|
|
397
|
+
| `-o, --output <dir>` | Output directory for generated files |
|
|
398
|
+
| `--app <id>` | Application ID (if name is not unique) |
|
|
399
|
+
| `--provider <provider>` | Test provider: local, browserstack, saucelabs, lambdatest (default: local) |
|
|
400
|
+
| `--browsers <browsers>` | Comma-separated list of browsers (default: chromium) |
|
|
401
|
+
|
|
402
|
+
## Exit Codes
|
|
403
|
+
|
|
404
|
+
| Code | Meaning |
|
|
405
|
+
|------|---------|
|
|
406
|
+
| 0 | Success (tests passed or still running) |
|
|
407
|
+
| 1 | Failure (tests failed or error occurred) |
|
|
408
|
+
|
|
409
|
+
## Test Sets vs Test Sequences
|
|
410
|
+
|
|
411
|
+
| Feature | Test Set | Test Sequence |
|
|
412
|
+
|---------|----------|---------------|
|
|
413
|
+
| Execution | Parallel | Sequential |
|
|
414
|
+
| Speed | Faster | Slower |
|
|
415
|
+
| State sharing | Independent | Optional |
|
|
416
|
+
| Use case | Regression testing | User flows, E2E journeys |
|
|
417
|
+
| Stop on failure | No | Configurable |
|
|
418
|
+
|
|
419
|
+
## Windows Desktop Applications
|
|
420
|
+
|
|
421
|
+
For Windows desktop applications, tests are executed on a connected desktop agent. The CLI triggers server-side execution and polls for results.
|
|
422
|
+
|
|
423
|
+
```bash
|
|
424
|
+
# Run tests on a connected desktop agent
|
|
425
|
+
qate run -n "Desktop App Tests"
|
|
426
|
+
|
|
427
|
+
# Run a sequence on a connected desktop agent
|
|
428
|
+
qate run:sequence -n "Desktop Flow"
|
|
429
|
+
|
|
430
|
+
# Install a build on the agent machine before running tests
|
|
431
|
+
qate install --app <applicationId> --path "C:\builds\MyApp-Setup.exe" --args "/silent"
|
|
432
|
+
|
|
433
|
+
# Full CI pipeline: install build, then run tests
|
|
434
|
+
qate install --app 64a... --path "C:\builds\setup.exe" --args "/S"
|
|
435
|
+
qate run -n "Smoke Tests" --app 64a...
|
|
436
|
+
```
|
|
437
|
+
|
|
438
|
+
### Install Command Options
|
|
439
|
+
|
|
440
|
+
| Option | Description |
|
|
441
|
+
|--------|-------------|
|
|
442
|
+
| `--app <id>` | Application ID (required) |
|
|
443
|
+
| `--path <path>` | Path to installer on the agent machine (required) |
|
|
444
|
+
| `--args <arguments>` | Installer arguments (e.g., `/silent`, `/S`) |
|
|
445
|
+
| `--timeout <seconds>` | Timeout in seconds (default: 300) |
|
|
446
|
+
|
|
447
|
+
## License
|
|
448
|
+
|
|
449
|
+
MIT
|