@qaily/playwright-html-reporter 1.0.1-beta.3 → 1.0.1-beta.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +442 -82
- package/dist/cli.mjs +432 -346
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,133 +1,496 @@
|
|
|
1
1
|
# @qaily/playwright-html-reporter
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
**Qaily** is an ecosystem of QA tools for AI agents and QA engineers.
|
|
4
|
+
|
|
5
|
+
`@qaily/playwright-html-reporter` is the web interface for
|
|
6
|
+
[`@qaily/playwright-jsonl-reporter`](https://github.com/qaily/qaily-playwright-jsonl-reporter). The JSONL reporter
|
|
7
|
+
writes every Playwright run in a compact, AI-friendly format. An AI agent can read the runs and their history without
|
|
8
|
+
filling its context window or burning tokens, and find patterns in failures and their root causes.
|
|
9
|
+
|
|
10
|
+
This package adds the report for people, and is built to keep a year of history:
|
|
11
|
+
|
|
12
|
+
- **Monitoring** of hourly or daily runs: the latest run of each lane (an environment, a suite, a branch), refreshing
|
|
13
|
+
by itself.
|
|
14
|
+
- **Weekly and monthly statistics:** pass rate, flakiness and duration trends.
|
|
15
|
+
- **Issues:** failing tests grouped by cause, across runs.
|
|
16
|
+
- **Tests:** every attempt with its steps, errors and stack traces, screenshots, logs and Playwright traces.
|
|
17
|
+
- **Coverage heat map:** which parts of the product the tests exercised ([docs/heatmap.md](docs/heatmap.md)).
|
|
18
|
+
- **Retention built in:** heavy artifacts (screenshots, traces, videos) are deleted after a few days, while the light
|
|
19
|
+
data behind the dashboards is kept.
|
|
20
|
+
- **Static:** one page on any static host, such as GitHub Pages. Data is pre-aggregated into small Parquet and JSON
|
|
21
|
+
files and queried in the browser with DuckDB-WASM. Artifacts can stay in your bucket (AWS S3, Google Cloud Storage,
|
|
22
|
+
MinIO, Dell ECS), read through short-lived signed links.
|
|
23
|
+
- **Optional AI:** analyse failures and suggest fixes through a local agent host ([docs/ai.md](docs/ai.md)).
|
|
5
24
|
|
|
6
25
|
**[Live demo](https://qaily.github.io/qaily-playwright-html-reporter/)**, built from synthetic data.
|
|
7
26
|
|
|
8
|
-
|
|
9
|
-
(every attempt with steps, errors, screenshots and traces) and a coverage Heat Map.
|
|
10
|
-
- **Static:** a single-page app on any static host, such as GitHub Pages. Data is pre-aggregated into small Parquet
|
|
11
|
-
and JSON files and queried in the browser with DuckDB-WASM, so months of runs load in seconds.
|
|
12
|
-
- **Large artifacts stay in your bucket:** traces, videos and screenshots are read straight from S3, Google Cloud
|
|
13
|
-
Storage or MinIO through short-lived signed links, so they never go into the site.
|
|
14
|
-
- **Optional AI:** analyse failures and suggest fixes through a local agent host (see [docs/ai.md](docs/ai.md)).
|
|
15
|
-
- **Themes:** Chronoscope (default, 3D dials and carousels), Classic (light and dark) and Dark Ops, switched in Settings.
|
|
27
|
+
## Contents
|
|
16
28
|
|
|
17
|
-
|
|
29
|
+
1. [Install](#1-install)
|
|
30
|
+
2. [Record runs](#2-record-runs)
|
|
31
|
+
3. [View the report on your machine](#3-view-the-report-on-your-machine)
|
|
32
|
+
4. [Host the report](#4-host-the-report)
|
|
33
|
+
5. [Storage and credentials](#5-storage-and-credentials)
|
|
34
|
+
6. [Settings](#6-settings)
|
|
35
|
+
7. [Retention](#7-retention)
|
|
36
|
+
8. [Commands](#8-commands)
|
|
37
|
+
9. [Develop and release](#9-develop-and-release)
|
|
38
|
+
|
|
39
|
+
## 1. Install
|
|
18
40
|
|
|
19
41
|
```bash
|
|
20
|
-
npm install --save-dev @qaily/playwright-html-reporter
|
|
42
|
+
npm install --save-dev @qaily/playwright-html-reporter
|
|
21
43
|
```
|
|
22
44
|
|
|
23
|
-
|
|
45
|
+
See [CHANGELOG.md](CHANGELOG.md) for versions and release channels.
|
|
46
|
+
|
|
47
|
+
You need:
|
|
24
48
|
|
|
25
|
-
|
|
49
|
+
- Node 20 or newer, and `@playwright/test`. Coverage labels on steps need Playwright 1.58 or newer.
|
|
50
|
+
- The [AWS CLI v2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), only for a bucket.
|
|
51
|
+
`qaily-report` uses it, and its login, for every provider.
|
|
26
52
|
|
|
27
|
-
|
|
28
|
-
The commands that talk to a bucket (`sync`, `pull`, `publish`, and `serve` when it signs links) use the
|
|
29
|
-
[AWS CLI v2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and its login.
|
|
53
|
+
## 2. Record runs
|
|
30
54
|
|
|
31
|
-
|
|
55
|
+
Add the reporter to `playwright.config.ts`. It writes the JSONL files (through `@qaily/playwright-jsonl-reporter`),
|
|
56
|
+
so you don't add that reporter separately.
|
|
32
57
|
|
|
33
58
|
```ts
|
|
34
|
-
// playwright.config.ts
|
|
35
59
|
import { defineConfig } from '@playwright/test';
|
|
36
60
|
|
|
37
61
|
export default defineConfig({
|
|
38
|
-
reporter: [
|
|
62
|
+
reporter: [
|
|
63
|
+
['list'],
|
|
64
|
+
['@qaily/playwright-html-reporter', {
|
|
65
|
+
outputDir: 'reports/data',
|
|
66
|
+
environment: process.env.TEST_ENV ?? 'local',
|
|
67
|
+
category: 'ui:web',
|
|
68
|
+
tag: '@regression',
|
|
69
|
+
}],
|
|
70
|
+
],
|
|
71
|
+
use: {
|
|
72
|
+
screenshot: 'only-on-failure',
|
|
73
|
+
trace: 'retain-on-failure',
|
|
74
|
+
},
|
|
39
75
|
});
|
|
40
76
|
```
|
|
41
77
|
|
|
42
|
-
|
|
78
|
+
| Option | Default | What it does |
|
|
79
|
+
|---|---|---|
|
|
80
|
+
| `outputDir` | `'./reports/data'` | Where run folders are written |
|
|
81
|
+
| `environment` | none | The environment tested, e.g. `'staging'`. Lanes and filters use it |
|
|
82
|
+
| `category` | none | What kind of run, e.g. `'ui:web'` or `'api'`. Lanes use it |
|
|
83
|
+
| `tag` | none | The suite's name, shown on the run, e.g. `'@regression'` |
|
|
84
|
+
| `coverage.levels` | `['Component', 'Test Name', 'Feature', 'Area']` | Step annotation types used as the heat map's four levels |
|
|
85
|
+
| `archived` | `true` | Write `.jsonl.gz` (`true`) or plain `.jsonl` (`false`) |
|
|
86
|
+
| `writePlainText` | `true` | With `archived`, also write plain `.jsonl` copies |
|
|
87
|
+
| `compressLevel` | `6` | Gzip level, 1 to 9 |
|
|
88
|
+
| `includeAttachmentChecksums` | `false` | Record the SHA-256 of each attachment |
|
|
89
|
+
| `maxErrorMessageLength` | `500` | Error messages are cut at this length |
|
|
90
|
+
| `afterJsonl` | none | An async function called after the run's files are written |
|
|
91
|
+
|
|
92
|
+
The commit and branch are taken from git. Each run becomes one folder:
|
|
43
93
|
|
|
44
94
|
```
|
|
45
|
-
reports/data
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
95
|
+
reports/data/
|
|
96
|
+
└── <time>_<commit>_<n>/ one folder per run
|
|
97
|
+
├── manifest.jsonl.gz run start and end: commit, branch, environment, totals
|
|
98
|
+
├── test-summary.jsonl.gz one line per test: status, duration, error summary
|
|
99
|
+
├── test_details.jsonl.gz one line per attempt: steps and annotations
|
|
100
|
+
├── errors.jsonl.gz errors and stack traces
|
|
101
|
+
├── attachments-index.jsonl.gz list of attachments
|
|
102
|
+
├── step-coverage.jsonl.gz heat map labels (only when steps are labelled)
|
|
103
|
+
└── attachments/ screenshots, traces, videos, logs
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Screenshots and traces become attachments when Playwright makes them, so set `screenshot` and `trace` in `use` as you
|
|
107
|
+
need them. Traces are the heaviest part: [Retention](#7-retention) deletes them after a few days.
|
|
108
|
+
|
|
109
|
+
## 3. View the report on your machine
|
|
110
|
+
|
|
111
|
+
After a test run:
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
npx qaily-report show
|
|
49
115
|
```
|
|
50
116
|
|
|
51
|
-
|
|
117
|
+
This builds the report from `reports/data` into `report-data` and serves it at <http://127.0.0.1:4173>. Run it again
|
|
118
|
+
after the next test run; it only rebuilds what changed. `--in`, `--out` and `--port` change the folders and the port.
|
|
119
|
+
|
|
120
|
+
**Runs in a bucket** (set up in [section 4](#4-host-the-report)), with the bucket in `report-data/settings.json` and an
|
|
121
|
+
AWS CLI login ([section 5](#5-storage-and-credentials)). Use one of these, depending on where the report data lives:
|
|
122
|
+
|
|
123
|
+
| Your setup | Command | What it does |
|
|
52
124
|
|---|---|---|
|
|
53
|
-
|
|
|
54
|
-
|
|
|
55
|
-
| `environment` | none | Environment name, e.g. `'staging'`; lanes can match it |
|
|
56
|
-
| `category` | none | What kind of run this is, e.g. `'ui:web'`; lanes can match it |
|
|
57
|
-
| `tag` | none | Suite name shown on the run, e.g. `'@regression'` |
|
|
58
|
-
| `coverage.levels` | `['Component', 'Test Name', 'Feature', 'Area']` | Step annotation types used as the heat map's four levels |
|
|
59
|
-
| `includeAttachmentChecksums` | `false` | SHA-256 of each attachment |
|
|
60
|
-
| `maxErrorMessageLength` | `500` | Error messages are cut at this length |
|
|
61
|
-
| `afterJsonl` | none | Called after the run's files are written |
|
|
125
|
+
| Mode 2: Pages + bucket for artifacts | `show --sync` | Downloads the run folders (no attachments), builds the report on your machine, serves it |
|
|
126
|
+
| Mode 3: all data in the bucket | `show --pull` | Downloads the report data CI published, serves it. No build |
|
|
62
127
|
|
|
63
|
-
|
|
128
|
+
```bash
|
|
129
|
+
npx qaily-report show --sync --aws-profile qaily
|
|
130
|
+
npx qaily-report show --pull --aws-profile qaily
|
|
131
|
+
```
|
|
64
132
|
|
|
65
|
-
|
|
66
|
-
|
|
133
|
+
- **`--sync`** works in both modes. It shows only runs whose folders are still in the bucket: runs whose folders were
|
|
134
|
+
deleted by retention are missing. In mode 3, use it to see runs CI hasn't published yet.
|
|
135
|
+
- **`--pull`** is faster and includes runs whose folders have expired, but it's only as fresh as CI's last `publish`.
|
|
136
|
+
In mode 2 there's nothing to pull: CI deploys the data with the site.
|
|
137
|
+
- **Not both:** pull, then sync, is what CI does in mode 3 to keep history and add the newest runs. To look at the
|
|
138
|
+
report, one is enough, so `show` refuses both together.
|
|
67
139
|
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
140
|
+
Screenshots and traces stay in the bucket: the local server signs links to them with your login, which never reaches
|
|
141
|
+
the browser. The bucket needs no CORS rules for this.
|
|
142
|
+
|
|
143
|
+
**Without tests:** `npx qaily-report demo && npx qaily-report serve` shows 120 synthetic runs.
|
|
144
|
+
|
|
145
|
+
**Settings:** open Settings in the report. **For me** changes apply to your browser only. Under **For the team**,
|
|
146
|
+
**Save to settings.json** writes `report-data/settings.json`. Commit that file: CI uses it ([section 6](#6-settings)).
|
|
147
|
+
Add the rest of `report-data/` to `.gitignore`:
|
|
148
|
+
|
|
149
|
+
```gitignore
|
|
150
|
+
report-data/*
|
|
151
|
+
!report-data/settings.json
|
|
152
|
+
!report-data/coverage-catalog.json
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
## 4. Host the report
|
|
156
|
+
|
|
157
|
+
Three ways to share the report, here with GitHub Actions and GitHub Pages. Every workflow is in
|
|
158
|
+
[docs/workflows/](docs/workflows) ready to copy, and the steps are plain `qaily-report` commands, so other CI systems and
|
|
159
|
+
static hosts work the same way.
|
|
160
|
+
|
|
161
|
+
| | 1. Pages only | 2. Pages + bucket for artifacts | 3. All data in the bucket |
|
|
162
|
+
|---|---|---|---|
|
|
163
|
+
| On the site | App, report data, the latest runs' attachments | App and report data | App and `settings.json` only |
|
|
164
|
+
| In the bucket | Nothing | Run folders with attachments | Run folders and the report data |
|
|
165
|
+
| Site rebuilt | Every run | Every run | Once, and when the version or settings change |
|
|
166
|
+
| History kept in | A `report-data` branch | The bucket's run folders | The bucket |
|
|
167
|
+
| Dashboards need credentials | No | No | Yes, every viewer |
|
|
168
|
+
| Attachments need credentials | No | Yes | Yes |
|
|
169
|
+
| Who can read test names and errors | Anyone who can open the site | Anyone who can open the site | Only people with bucket access |
|
|
170
|
+
| Workflow | [pages-only.yml](docs/workflows/pages-only.yml) | [pages-s3-attachments.yml](docs/workflows/pages-s3-attachments.yml) | [s3-data.yml](docs/workflows/s3-data.yml) + [s3-data-site.yml](docs/workflows/s3-data-site.yml) |
|
|
171
|
+
|
|
172
|
+
Choose **1** to start, or when the site's readers may see everything. Choose **2** to keep large artifacts out of the
|
|
173
|
+
site. Choose **3** when the data must stay private and access to the site can't be limited, or to publish new runs without
|
|
174
|
+
redeploying. For all three, set Settings → Pages → Source to **GitHub Actions**.
|
|
175
|
+
|
|
176
|
+
### Mode 1: GitHub Pages only
|
|
177
|
+
|
|
178
|
+
After the tests, the job adds the run to the report data and deploys the site. The report data is small (rollups and
|
|
179
|
+
JSONL), and is kept in a `report-data` branch between runs; `build` keeps the earlier runs from it. Attachments are
|
|
180
|
+
published only for the runs in the latest deploy.
|
|
181
|
+
|
|
182
|
+
```yaml
|
|
183
|
+
- run: npx playwright test
|
|
184
|
+
- run: npx qaily-report build --in reports/data --out report-history # report-history: the report-data branch
|
|
185
|
+
- run: |
|
|
186
|
+
cp -r node_modules/@qaily/playwright-html-reporter/app site
|
|
187
|
+
cp -r report-history site/data
|
|
188
|
+
npx qaily-report trace-viewer --out site/trace
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
[pages-only.yml](docs/workflows/pages-only.yml) has the full job, including the branch and the attachments.
|
|
192
|
+
|
|
193
|
+
### Mode 2: GitHub Pages + bucket for artifacts
|
|
194
|
+
|
|
195
|
+
The test job uploads the run folder, attachments included. The report job rebuilds the report from every run in the
|
|
196
|
+
bucket, without downloading attachments, and deploys it. Viewers connect credentials to open screenshots and traces.
|
|
197
|
+
|
|
198
|
+
```yaml
|
|
199
|
+
# test job
|
|
200
|
+
- run: npx playwright test
|
|
201
|
+
- run: npx qaily-report upload # reports/data → bucket, .sync-complete marker last
|
|
202
|
+
# report job
|
|
203
|
+
- run: npx qaily-report sync # run folders from the bucket, no attachments, then build
|
|
204
|
+
- run: |
|
|
205
|
+
cp -r node_modules/@qaily/playwright-html-reporter/app site
|
|
206
|
+
cp -r report-data site/data
|
|
207
|
+
npx qaily-report trace-viewer --out site/trace
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
`report-data/settings.json` names the bucket ([settings for each mode](#settings-for-each-mode)). CI needs a key that
|
|
211
|
+
can upload and list the run folders. If run folders are deleted after a while, add `pull` before `sync` and `publish`
|
|
212
|
+
after it, so the bucket keeps the report data between builds.
|
|
213
|
+
|
|
214
|
+
### Mode 3: all data in the bucket
|
|
215
|
+
|
|
216
|
+
The site is deployed once, with the app and `settings.json`. After each test run, CI publishes the report data to the
|
|
217
|
+
bucket, and viewers see the new run on their next load.
|
|
218
|
+
|
|
219
|
+
```yaml
|
|
220
|
+
# after the tests, every run (s3-data.yml)
|
|
221
|
+
- run: npx qaily-report upload
|
|
222
|
+
- run: npx qaily-report pull # what was published last time
|
|
223
|
+
- run: npx qaily-report sync # run folders → rebuild the months that changed
|
|
224
|
+
- run: npx qaily-report publish # report data → bucket, catalog.json last
|
|
225
|
+
# when the version or settings change (s3-data-site.yml)
|
|
226
|
+
- run: |
|
|
227
|
+
cp -r node_modules/@qaily/playwright-html-reporter/app site
|
|
228
|
+
mkdir -p site/data && cp report-data/settings.json site/data/
|
|
229
|
+
npx qaily-report trace-viewer --out site/trace
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
Viewers paste credentials in Settings → Tokens. Team settings (lanes, thresholds, heat map rules) can then be saved from
|
|
233
|
+
the report to the bucket, with no redeploy ([docs/storage.md](docs/storage.md#7-team-settings-in-the-bucket)). Run folders
|
|
234
|
+
can expire while the dashboards keep every run.
|
|
235
|
+
|
|
236
|
+
### Reading the bucket from the browser: CORS or the storage bridge
|
|
237
|
+
|
|
238
|
+
In modes 2 and 3 the page reads the bucket from the viewer's browser, which needs the bucket's permission (CORS):
|
|
239
|
+
|
|
240
|
+
- **You can set CORS rules:** allow the site's origin, e.g. `https://<org>.github.io`, for `GET` and `HEAD` (and `PUT` to
|
|
241
|
+
save team settings). Examples for each provider are in [docs/storage.md](docs/storage.md#3-cors).
|
|
242
|
+
- **You can't** (Dell ECS, or no admin rights): use the **storage bridge**. Upload a small page to the bucket once, and set
|
|
243
|
+
Settings → Connections → **Storage access** to **Bridge**:
|
|
244
|
+
|
|
245
|
+
```bash
|
|
246
|
+
npx qaily-report bridge --origin https://<org>.github.io --aws-profile qaily
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
The report then reads the bucket through that page, with no CORS rules. Each viewer can switch between Direct and Bridge
|
|
250
|
+
under **For me**. How it works and what it allows: [docs/storage.md](docs/storage.md#9-storage-without-cors-the-storage-bridge).
|
|
251
|
+
|
|
252
|
+
Without CORS or the bridge, mode 2 still shows dashboards, screenshots and downloads (on most storage), but not text
|
|
253
|
+
attachments; mode 3 shows nothing. `qaily-report show` needs neither.
|
|
254
|
+
|
|
255
|
+
**Traces** open in the trace viewer copied next to the report (`qaily-report trace-viewer`, with
|
|
256
|
+
`"traceViewer": "trace/"`). The report hands it the trace file, so the viewer needs no CORS of its own
|
|
257
|
+
([docs/storage.md](docs/storage.md#10-self-hosted-trace-viewer)).
|
|
258
|
+
|
|
259
|
+
## 5. Storage and credentials
|
|
260
|
+
|
|
261
|
+
| Storage | `endpoint` | `region` | `pathStyle` | Browser access |
|
|
262
|
+
|---|---|---|---|---|
|
|
263
|
+
| AWS S3 (tested) | empty | the bucket's region | `false` | CORS |
|
|
264
|
+
| Google Cloud Storage (tested) | `https://storage.googleapis.com` | `auto` | `false` | CORS |
|
|
265
|
+
| MinIO | the server's URL | usually `us-east-1` | `true` | CORS (server-wide) |
|
|
266
|
+
| Dell ECS | the server's URL | any, e.g. `us-east-1` | `true` | Storage bridge |
|
|
267
|
+
|
|
268
|
+
The commands log in through the AWS CLI v2, for every provider:
|
|
269
|
+
|
|
270
|
+
```bash
|
|
271
|
+
aws configure --profile qaily # access key and secret (HMAC key for Google Cloud Storage)
|
|
272
|
+
aws configure sso --profile qaily # or AWS IAM Identity Center; then: aws sso login --profile qaily
|
|
273
|
+
npx qaily-report show --sync --aws-profile qaily # or AWS_PROFILE=qaily, or the AWS_ACCESS_KEY_ID variables
|
|
72
274
|
```
|
|
73
275
|
|
|
74
|
-
|
|
276
|
+
For the browser (Settings → Tokens), paste one of:
|
|
75
277
|
|
|
76
278
|
```bash
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
279
|
+
aws configure export-credentials --profile qaily --format process
|
|
280
|
+
aws sts get-session-token --profile qaily --duration-seconds 43200 --query Credentials --output json # 12-hour keys
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
[docs/storage.md](docs/storage.md) has the rest:
|
|
284
|
+
- provider notes;
|
|
285
|
+
- CORS rules;
|
|
286
|
+
- IAM policies for viewers, CI and the retention job;
|
|
287
|
+
- certificates for internal storage;
|
|
288
|
+
- what the page stores and for how long.
|
|
289
|
+
|
|
290
|
+
## 6. Settings
|
|
291
|
+
|
|
292
|
+
Settings come in three layers, the later ones winning:
|
|
293
|
+
|
|
294
|
+
1. **Site:** `settings.json` next to the report data. In this README it's `report-data/settings.json`, committed and
|
|
295
|
+
deployed by CI.
|
|
296
|
+
2. **Team settings in the bucket** (mode 3): `team-settings.json` in the data folder. **Save to bucket** writes it, and it
|
|
297
|
+
never contains the bucket connection.
|
|
298
|
+
3. **Personal:** Settings → **For me**, kept in each viewer's browser.
|
|
299
|
+
|
|
300
|
+
Edit them in the report: Settings shows where each value comes from, and **For the team** saves to the right place
|
|
301
|
+
(**Save to settings.json**, **Save to bucket** or **Commit to GitHub**). Or edit the JSON. Unset fields keep their
|
|
302
|
+
defaults.
|
|
303
|
+
|
|
304
|
+
### Settings for each mode
|
|
305
|
+
|
|
306
|
+
Mode 1, no bucket:
|
|
307
|
+
|
|
308
|
+
```json
|
|
309
|
+
{
|
|
310
|
+
"lanes": [
|
|
311
|
+
{ "id": "main", "label": "Main branch", "match": { "branch": "^main$" } },
|
|
312
|
+
{ "id": "other", "label": "Other branches", "match": {} }
|
|
313
|
+
],
|
|
314
|
+
"connections": { "traceViewer": "trace/" }
|
|
315
|
+
}
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
Mode 2: the same, plus the bucket:
|
|
319
|
+
|
|
320
|
+
```json
|
|
321
|
+
{
|
|
322
|
+
"connections": {
|
|
323
|
+
"traceViewer": "trace/",
|
|
324
|
+
"s3": { "bucket": "test-reports", "region": "eu-north-1", "prefix": "jsonl-report" }
|
|
325
|
+
}
|
|
326
|
+
}
|
|
327
|
+
```
|
|
328
|
+
|
|
329
|
+
Mode 3: the data source and the data folder too. Add `"transport": "bridge"` for the storage bridge:
|
|
330
|
+
|
|
331
|
+
```json
|
|
332
|
+
{
|
|
333
|
+
"connections": {
|
|
334
|
+
"traceViewer": "trace/",
|
|
335
|
+
"s3": {
|
|
336
|
+
"bucket": "test-reports",
|
|
337
|
+
"region": "eu-north-1",
|
|
338
|
+
"prefix": "jsonl-report",
|
|
339
|
+
"dataSource": "bucket",
|
|
340
|
+
"dataPrefix": "qaily-data"
|
|
341
|
+
}
|
|
342
|
+
}
|
|
343
|
+
}
|
|
80
344
|
```
|
|
81
345
|
|
|
82
|
-
|
|
83
|
-
|
|
346
|
+
For Google Cloud Storage, MinIO or ECS, add `endpoint` and `pathStyle` from [section 5](#5-storage-and-credentials).
|
|
347
|
+
`prefix` is where run folders go; `dataPrefix` must be outside it.
|
|
348
|
+
|
|
349
|
+
### Lanes
|
|
350
|
+
|
|
351
|
+
A lane is a line of runs on the Monitoring page and a filter elsewhere. Its `match` fields are all optional: `category`
|
|
352
|
+
and `branch` are regular expressions, and `environment` must match exactly. Each run joins the first lane that matches,
|
|
353
|
+
so list specific lanes first.
|
|
354
|
+
|
|
355
|
+
```json
|
|
356
|
+
{ "id": "web-staging", "label": "Web · Staging", "match": { "category": "^ui:web$", "environment": "staging" } }
|
|
357
|
+
```
|
|
84
358
|
|
|
85
|
-
|
|
86
|
-
Under **For the team**, the save button says where it writes. **Save to settings.json** writes `report-data/settings.json` when the report is opened through `serve` (on GitHub Pages it
|
|
87
|
-
can commit the file instead, see [docs/ai.md](docs/ai.md#drafts-and-publishing)). A report that reads its data from a
|
|
88
|
-
bucket keeps team settings there instead (**Save to bucket**), as `team-settings.json` next to the data; the bucket
|
|
89
|
-
connection itself always comes from the site's `settings.json` (see
|
|
90
|
-
[docs/s3-artifacts.md](docs/s3-artifacts.md#team-settings-in-the-bucket)).
|
|
359
|
+
### Coverage heat map
|
|
91
360
|
|
|
92
|
-
|
|
361
|
+
The heat map places steps on four levels: Component, Scenario, Feature and Area. Steps get there in two ways:
|
|
93
362
|
|
|
94
|
-
|
|
95
|
-
|
|
363
|
+
- **Mapping rules** in Settings → Coverage heat map, with no test changes:
|
|
364
|
+
```json
|
|
365
|
+
{ "coverage": { "rules": [ { "step": "^Archive board", "component": "Permissions", "feature": "Boards", "area": "Archive" } ] } }
|
|
366
|
+
```
|
|
367
|
+
- **Labels on steps**, with Playwright 1.58+:
|
|
368
|
+
```ts
|
|
369
|
+
import { coverage } from '@qaily/playwright-html-reporter';
|
|
370
|
+
|
|
371
|
+
await test.step('Archive board', async (step) => {
|
|
372
|
+
coverage(step, { component: 'Permissions', scenario: 'Workspace owners', feature: 'Boards', area: 'Archive' });
|
|
373
|
+
});
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
A planned matrix shows the gaps: `npx qaily-report catalog --csv coverage/matrix.csv` writes
|
|
377
|
+
`report-data/coverage-catalog.json`. Details are in [docs/heatmap.md](docs/heatmap.md).
|
|
378
|
+
|
|
379
|
+
## 7. Retention
|
|
380
|
+
|
|
381
|
+
Attachments are most of the storage: a run with traces can be 1 GB or more, while its JSONL is a few hundred KB. The
|
|
382
|
+
report is built to keep a year of light data and drop heavy artifacts after a few days. Example: 20 runs a day of
|
|
383
|
+
1.5 GB is 900 GB a month with everything kept, and about 90 GB with attachments kept for 3 days.
|
|
384
|
+
|
|
385
|
+
| What | How long | How |
|
|
386
|
+
|---|---|---|
|
|
387
|
+
| Attachments | Days | `qaily-report expire-attachments`, on a schedule |
|
|
388
|
+
| Run folders | Weeks, or forever | A lifecycle rule on the run-folder prefix, or keep them |
|
|
389
|
+
| Report data (dashboards) | A year or more | Kept by `build`; `--prune-before YYYY-MM` drops older months |
|
|
390
|
+
|
|
391
|
+
### Delete old attachments: `expire-attachments`
|
|
96
392
|
|
|
97
393
|
```bash
|
|
98
|
-
npx qaily-report
|
|
394
|
+
npx qaily-report expire-attachments --older-than 3 --dry-run # what would be deleted
|
|
395
|
+
npx qaily-report expire-attachments --older-than 3 # delete it
|
|
99
396
|
```
|
|
100
397
|
|
|
101
|
-
|
|
398
|
+
It deletes `<prefix>/<run>/attachments/` of runs whose attachments are all older than the given days, and keeps the rest
|
|
399
|
+
of each run folder. The dashboards, and other tools that read the run folders, keep working. The age lives in your
|
|
400
|
+
workflow, not in the bucket, so it works where you can't change the bucket's settings. Run it daily:
|
|
401
|
+
[expire-attachments.yml](docs/workflows/expire-attachments.yml).
|
|
402
|
+
|
|
403
|
+
A lifecycle rule can't do this, because its filter matches the start of a key and `attachments/` is in the middle. In a
|
|
404
|
+
versioned bucket, a delete only hides a file, so also add a rule that deletes old versions, or pass `--all-versions`.
|
|
405
|
+
|
|
406
|
+
The report then says, for an old run, that its attachments are no longer in storage. The test's error, steps and history
|
|
407
|
+
are still shown.
|
|
408
|
+
|
|
409
|
+
### Delete old run folders: lifecycle rules
|
|
410
|
+
|
|
411
|
+
To delete whole run folders after, say, 30 days, add a lifecycle rule on the run-folder prefix. Dashboards keep those runs
|
|
412
|
+
as long as CI keeps the report data: mode 3 does this already; in mode 2, add `pull` and `publish`
|
|
413
|
+
([docs/storage.md](docs/storage.md#11-when-run-folders-expire-too)).
|
|
414
|
+
|
|
415
|
+
AWS S3 (`aws s3api put-bucket-lifecycle-configuration --bucket <bucket> --lifecycle-configuration file://lifecycle.json`).
|
|
416
|
+
This replaces the bucket's existing rules, so include any you have:
|
|
102
417
|
|
|
103
418
|
```json
|
|
104
|
-
{
|
|
419
|
+
{
|
|
420
|
+
"Rules": [
|
|
421
|
+
{ "ID": "run-folders", "Status": "Enabled", "Filter": { "Prefix": "jsonl-report/" }, "Expiration": { "Days": 30 } },
|
|
422
|
+
{ "ID": "old-versions", "Status": "Enabled", "Filter": {}, "NoncurrentVersionExpiration": { "NoncurrentDays": 1 } }
|
|
423
|
+
]
|
|
424
|
+
}
|
|
105
425
|
```
|
|
106
426
|
|
|
107
|
-
|
|
108
|
-
[docs/report-site.example.yml](docs/report-site.example.yml) does this on GitHub Pages every 30 minutes.
|
|
427
|
+
Google Cloud Storage (`gcloud storage buckets update gs://<bucket> --lifecycle-file=lifecycle.json`):
|
|
109
428
|
|
|
110
|
-
|
|
111
|
-
|
|
429
|
+
```json
|
|
430
|
+
{ "rule": [ { "action": { "type": "Delete" }, "condition": { "age": 30, "matchesPrefix": ["jsonl-report/"] } } ] }
|
|
431
|
+
```
|
|
432
|
+
|
|
433
|
+
MinIO: `mc ilm rule add --expire-days 30 --prefix "jsonl-report/" <alias>/<bucket>`.
|
|
434
|
+
|
|
435
|
+
Dell ECS accepts the AWS rule through its S3 API (`--endpoint-url`) when the bucket's owner allows it. Otherwise use
|
|
436
|
+
`expire-attachments` only.
|
|
437
|
+
|
|
438
|
+
Rules count days from upload, and providers usually apply them once a day.
|
|
439
|
+
|
|
440
|
+
## 8. Commands
|
|
112
441
|
|
|
113
|
-
|
|
442
|
+
Run them as `npx qaily-report <command>`; `npx qaily-report help` lists them. The defaults match the setup above: run
|
|
443
|
+
folders in `reports/data`, report data in `report-data`, and the bucket from `report-data/settings.json`.
|
|
114
444
|
|
|
115
445
|
| Command | What it does |
|
|
116
446
|
|---|---|
|
|
117
|
-
| `
|
|
118
|
-
| `serve
|
|
119
|
-
| `
|
|
120
|
-
| `
|
|
121
|
-
| `
|
|
122
|
-
| `
|
|
123
|
-
| `
|
|
124
|
-
| `
|
|
125
|
-
| `
|
|
447
|
+
| `show` | Build from local run folders and serve |
|
|
448
|
+
| `serve` | Serve report data that's already built |
|
|
449
|
+
| `build` | Run folders → report data; only changed months are rebuilt |
|
|
450
|
+
| `upload` | Run folders → bucket, attachments included, marker last |
|
|
451
|
+
| `sync` | Bucket run folders → local folder (no attachments), then build |
|
|
452
|
+
| `pull` | Published report data → local folder |
|
|
453
|
+
| `publish` | Report data → bucket, `catalog.json` last |
|
|
454
|
+
| `expire-attachments` | Delete the attachments of old runs, keep the rest |
|
|
455
|
+
| `bridge` | Upload the storage bridge page |
|
|
456
|
+
| `trace-viewer` | Copy Playwright's trace viewer from the test project |
|
|
457
|
+
| `catalog` | Planned coverage matrix for the heat map |
|
|
458
|
+
| `demo` | Synthetic runs, to try the report |
|
|
459
|
+
|
|
460
|
+
Options, with their defaults:
|
|
461
|
+
|
|
462
|
+
```text
|
|
463
|
+
show --in reports/data --out report-data --port 4173
|
|
464
|
+
--sync run folders from the bucket instead
|
|
465
|
+
--pull published report data from the bucket instead
|
|
466
|
+
serve --data report-data --attachments <run folders>
|
|
467
|
+
--port 4173 --host 127.0.0.1 --app <built app>
|
|
468
|
+
build --in reports/data --out report-data
|
|
469
|
+
--force rebuild every month
|
|
470
|
+
--prune-before YYYY-MM drop older months for good
|
|
471
|
+
upload --in reports/data --dry-run
|
|
472
|
+
sync --to .qaily/runs --out report-data --with-attachments
|
|
473
|
+
pull --out report-data
|
|
474
|
+
publish --out report-data --allow-shrink
|
|
475
|
+
--team-settings seed|replace|skip team-settings.json
|
|
476
|
+
expire-attachments --older-than <days> (required)
|
|
477
|
+
--dry-run --all-versions
|
|
478
|
+
bridge --origin <site origin> (required) --dry-run
|
|
479
|
+
trace-viewer --out <site>/trace (required) --from <test project>
|
|
480
|
+
catalog --csv <matrix.csv> (required)
|
|
481
|
+
--out report-data/coverage-catalog.json
|
|
482
|
+
demo --out report-data --runs 120 --days 30
|
|
483
|
+
```
|
|
484
|
+
|
|
485
|
+
Bucket options, for every command that uses a bucket:
|
|
126
486
|
|
|
127
|
-
|
|
128
|
-
|
|
487
|
+
```text
|
|
488
|
+
--bucket <name> --region <region> --endpoint <url> --path-style
|
|
489
|
+
--prefix <run folders> --data-prefix <report data>
|
|
490
|
+
--aws-profile <name>
|
|
491
|
+
```
|
|
129
492
|
|
|
130
|
-
## Develop
|
|
493
|
+
## 9. Develop and release
|
|
131
494
|
|
|
132
495
|
```bash
|
|
133
496
|
npm ci
|
|
@@ -137,11 +500,8 @@ npm run app:test && npm run app:typecheck
|
|
|
137
500
|
npm run pkg:build # dist/ and app/, as published to npm
|
|
138
501
|
```
|
|
139
502
|
|
|
140
|
-
The workspace layout is
|
|
141
|
-
|
|
142
|
-
## Releasing
|
|
143
|
-
|
|
144
|
-
Actions → **Release** → Run workflow (bump and channel), as for `@qaily/playwright-jsonl-reporter`. See [docs/releasing.md](docs/releasing.md).
|
|
503
|
+
The workspace layout is in [packages/README.md](packages/README.md). Releases: Actions → **Release** → Run workflow,
|
|
504
|
+
see [docs/releasing.md](docs/releasing.md).
|
|
145
505
|
|
|
146
506
|
## License
|
|
147
507
|
|