@qaily/playwright-html-reporter 1.0.1-beta.4 → 1.0.1-beta.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +442 -82
  2. package/dist/cli.mjs +432 -346
  3. package/package.json +1 -1
package/README.md CHANGED
@@ -1,133 +1,496 @@
1
1
  # @qaily/playwright-html-reporter
2
2
 
3
- A Playwright reporter with a static, fast report built for test history: what failed today, how this week and
4
- month went, which issues keep coming back, and which parts of the product the tests actually cover.
3
+ **Qaily** is an ecosystem of QA tools for AI agents and QA engineers.
4
+
5
+ `@qaily/playwright-html-reporter` is the web interface for
6
+ [`@qaily/playwright-jsonl-reporter`](https://github.com/qaily/qaily-playwright-jsonl-reporter). The JSONL reporter
7
+ writes every Playwright run in a compact, AI-friendly format. An AI agent can read the runs and their history without
8
+ filling its context window or burning tokens, and find patterns in failures and their root causes.
9
+
10
+ This package adds the report for people, and is built to keep a year of history:
11
+
12
+ - **Monitoring** of hourly or daily runs: the latest run of each lane (an environment, a suite, a branch), refreshing
13
+ by itself.
14
+ - **Weekly and monthly statistics:** pass rate, flakiness and duration trends.
15
+ - **Issues:** failing tests grouped by cause, across runs.
16
+ - **Tests:** every attempt with its steps, errors and stack traces, screenshots, logs and Playwright traces.
17
+ - **Coverage heat map:** which parts of the product the tests exercised ([docs/heatmap.md](docs/heatmap.md)).
18
+ - **Retention built in:** heavy artifacts (screenshots, traces, videos) are deleted after a few days, while the light
19
+ data behind the dashboards is kept.
20
+ - **Static:** one page on any static host, such as GitHub Pages. Data is pre-aggregated into small Parquet and JSON
21
+ files and queried in the browser with DuckDB-WASM. Artifacts can stay in your bucket (AWS S3, Google Cloud Storage,
22
+ MinIO, Dell ECS), read through short-lived signed links.
23
+ - **Optional AI:** analyse failures and suggest fixes through a local agent host ([docs/ai.md](docs/ai.md)).
5
24
 
6
25
  **[Live demo](https://qaily.github.io/qaily-playwright-html-reporter/)**, built from synthetic data.
7
26
 
8
- - **Dashboards:** Monitoring (latest run per lane), Weekly, Monthly, Issues (failures grouped by cause), Tests
9
- (every attempt with steps, errors, screenshots and traces) and a coverage Heat Map.
10
- - **Static:** a single-page app on any static host, such as GitHub Pages. Data is pre-aggregated into small Parquet
11
- and JSON files and queried in the browser with DuckDB-WASM, so months of runs load in seconds.
12
- - **Large artifacts stay in your bucket:** traces, videos and screenshots are read straight from S3, Google Cloud
13
- Storage or MinIO through short-lived signed links, so they never go into the site.
14
- - **Optional AI:** analyse failures and suggest fixes through a local agent host (see [docs/ai.md](docs/ai.md)).
15
- - **Themes:** Chronoscope (default, 3D dials and carousels), Classic (light and dark) and Dark Ops, switched in Settings.
27
+ ## Contents
16
28
 
17
- ## Install
29
+ 1. [Install](#1-install)
30
+ 2. [Record runs](#2-record-runs)
31
+ 3. [View the report on your machine](#3-view-the-report-on-your-machine)
32
+ 4. [Host the report](#4-host-the-report)
33
+ 5. [Storage and credentials](#5-storage-and-credentials)
34
+ 6. [Settings](#6-settings)
35
+ 7. [Retention](#7-retention)
36
+ 8. [Commands](#8-commands)
37
+ 9. [Develop and release](#9-develop-and-release)
38
+
39
+ ## 1. Install
18
40
 
19
41
  ```bash
20
- npm install --save-dev @qaily/playwright-html-reporter@beta
42
+ npm install --save-dev @qaily/playwright-html-reporter
21
43
  ```
22
44
 
23
- Versions are still betas (`1.0.1-beta.N`), published under the `beta` tag; see [CHANGELOG.md](CHANGELOG.md).
45
+ See [CHANGELOG.md](CHANGELOG.md) for versions and release channels.
46
+
47
+ You need:
24
48
 
25
- Or straight from GitHub (builds on install, about 20 s): `npm install --save-dev github:qaily/qaily-playwright-html-reporter`.
49
+ - Node 20 or newer, and `@playwright/test`. Coverage labels on steps need Playwright 1.58 or newer.
50
+ - The [AWS CLI v2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), only for a bucket.
51
+ `qaily-report` uses it, and its login, for every provider.
26
52
 
27
- Node 20 or newer. The reporter needs `@playwright/test`; step coverage labels need Playwright 1.58+.
28
- The commands that talk to a bucket (`sync`, `pull`, `publish`, and `serve` when it signs links) use the
29
- [AWS CLI v2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and its login.
53
+ ## 2. Record runs
30
54
 
31
- ## 1. Record runs
55
+ Add the reporter to `playwright.config.ts`. It writes the JSONL files (through `@qaily/playwright-jsonl-reporter`),
56
+ so you don't add that reporter separately.
32
57
 
33
58
  ```ts
34
- // playwright.config.ts
35
59
  import { defineConfig } from '@playwright/test';
36
60
 
37
61
  export default defineConfig({
38
- reporter: [['@qaily/playwright-html-reporter', { outputDir: 'reports/data' }]],
62
+ reporter: [
63
+ ['list'],
64
+ ['@qaily/playwright-html-reporter', {
65
+ outputDir: 'reports/data',
66
+ environment: process.env.TEST_ENV ?? 'local',
67
+ category: 'ui:web',
68
+ tag: '@regression',
69
+ }],
70
+ ],
71
+ use: {
72
+ screenshot: 'only-on-failure',
73
+ trace: 'retain-on-failure',
74
+ },
39
75
  });
40
76
  ```
41
77
 
42
- Each run becomes a folder under `outputDir`:
78
+ | Option | Default | What it does |
79
+ |---|---|---|
80
+ | `outputDir` | `'./reports/data'` | Where run folders are written |
81
+ | `environment` | none | The environment tested, e.g. `'staging'`. Lanes and filters use it |
82
+ | `category` | none | What kind of run, e.g. `'ui:web'` or `'api'`. Lanes use it |
83
+ | `tag` | none | The suite's name, shown on the run, e.g. `'@regression'` |
84
+ | `coverage.levels` | `['Component', 'Test Name', 'Feature', 'Area']` | Step annotation types used as the heat map's four levels |
85
+ | `archived` | `true` | Write `.jsonl.gz` (`true`) or plain `.jsonl` (`false`) |
86
+ | `writePlainText` | `true` | With `archived`, also write plain `.jsonl` copies |
87
+ | `compressLevel` | `6` | Gzip level, 1 to 9 |
88
+ | `includeAttachmentChecksums` | `false` | Record the SHA-256 of each attachment |
89
+ | `maxErrorMessageLength` | `500` | Error messages are cut at this length |
90
+ | `afterJsonl` | none | An async function called after the run's files are written |
91
+
92
+ The commit and branch are taken from git. Each run becomes one folder:
43
93
 
44
94
  ```
45
- reports/data/<ISO time>_<commit>_<n>/
46
- manifest.jsonl.gz test-summary.jsonl.gz test_details.jsonl.gz errors.jsonl.gz attachments-index.jsonl.gz
47
- step-coverage.jsonl.gz (only when steps are labelled)
48
- attachments/ (screenshots, traces, logs)
95
+ reports/data/
96
+ └── <time>_<commit>_<n>/ one folder per run
97
+ ├── manifest.jsonl.gz run start and end: commit, branch, environment, totals
98
+ ├── test-summary.jsonl.gz one line per test: status, duration, error summary
99
+ ├── test_details.jsonl.gz one line per attempt: steps and annotations
100
+ ├── errors.jsonl.gz errors and stack traces
101
+ ├── attachments-index.jsonl.gz list of attachments
102
+ ├── step-coverage.jsonl.gz heat map labels (only when steps are labelled)
103
+ └── attachments/ screenshots, traces, videos, logs
104
+ ```
105
+
106
+ Screenshots and traces become attachments when Playwright makes them, so set `screenshot` and `trace` in `use` as you
107
+ need them. Traces are the heaviest part: [Retention](#7-retention) deletes them after a few days.
108
+
109
+ ## 3. View the report on your machine
110
+
111
+ After a test run:
112
+
113
+ ```bash
114
+ npx qaily-report show
49
115
  ```
50
116
 
51
- | Option | Default | Description |
117
+ This builds the report from `reports/data` into `report-data` and serves it at <http://127.0.0.1:4173>. Run it again
118
+ after the next test run; it only rebuilds what changed. `--in`, `--out` and `--port` change the folders and the port.
119
+
120
+ **Runs in a bucket** (set up in [section 4](#4-host-the-report)), with the bucket in `report-data/settings.json` and an
121
+ AWS CLI login ([section 5](#5-storage-and-credentials)). Use one of these, depending on where the report data lives:
122
+
123
+ | Your setup | Command | What it does |
52
124
  |---|---|---|
53
- | `outputDir` | `'./reports/data'` | Where run folders are written |
54
- | `archived` | `true` | Write `.jsonl.gz` (true) or plain `.jsonl` (false) |
55
- | `environment` | none | Environment name, e.g. `'staging'`; lanes can match it |
56
- | `category` | none | What kind of run this is, e.g. `'ui:web'`; lanes can match it |
57
- | `tag` | none | Suite name shown on the run, e.g. `'@regression'` |
58
- | `coverage.levels` | `['Component', 'Test Name', 'Feature', 'Area']` | Step annotation types used as the heat map's four levels |
59
- | `includeAttachmentChecksums` | `false` | SHA-256 of each attachment |
60
- | `maxErrorMessageLength` | `500` | Error messages are cut at this length |
61
- | `afterJsonl` | none | Called after the run's files are written |
125
+ | Mode 2: Pages + bucket for artifacts | `show --sync` | Downloads the run folders (no attachments), builds the report on your machine, serves it |
126
+ | Mode 3: all data in the bucket | `show --pull` | Downloads the report data CI published, serves it. No build |
62
127
 
63
- Label steps for the coverage heat map (optional, see [docs/heatmap.md](docs/heatmap.md)):
128
+ ```bash
129
+ npx qaily-report show --sync --aws-profile qaily
130
+ npx qaily-report show --pull --aws-profile qaily
131
+ ```
64
132
 
65
- ```ts
66
- import { coverage } from '@qaily/playwright-html-reporter';
133
+ - **`--sync`** works in both modes. It shows only runs whose folders are still in the bucket: runs whose folders were
134
+ deleted by retention are missing. In mode 3, use it to see runs CI hasn't published yet.
135
+ - **`--pull`** is faster and includes runs whose folders have expired, but it's only as fresh as CI's last `publish`.
136
+ In mode 2 there's nothing to pull: CI deploys the data with the site.
137
+ - **Not both:** pull, then sync, is what CI does in mode 3 to keep history and add the newest runs. To look at the
138
+ report, one is enough, so `show` refuses both together.
67
139
 
68
- await test.step('Archive board', async (step) => {
69
- coverage(step, { component: 'Permissions', scenario: 'Workspace owners', feature: 'Boards', area: 'Archive' });
70
- // ...
71
- });
140
+ Screenshots and traces stay in the bucket: the local server signs links to them with your login, which never reaches
141
+ the browser. The bucket needs no CORS rules for this.
142
+
143
+ **Without tests:** `npx qaily-report demo && npx qaily-report serve` shows 120 synthetic runs.
144
+
145
+ **Settings:** open Settings in the report. **For me** changes apply to your browser only. Under **For the team**,
146
+ **Save to settings.json** writes `report-data/settings.json`. Commit that file: CI uses it ([section 6](#6-settings)).
147
+ Add the rest of `report-data/` to `.gitignore`:
148
+
149
+ ```gitignore
150
+ report-data/*
151
+ !report-data/settings.json
152
+ !report-data/coverage-catalog.json
153
+ ```
154
+
155
+ ## 4. Host the report
156
+
157
+ Three ways to share the report, here with GitHub Actions and GitHub Pages. Every workflow is in
158
+ [docs/workflows/](docs/workflows) ready to copy, and the steps are plain `qaily-report` commands, so other CI systems and
159
+ static hosts work the same way.
160
+
161
+ | | 1. Pages only | 2. Pages + bucket for artifacts | 3. All data in the bucket |
162
+ |---|---|---|---|
163
+ | On the site | App, report data, the latest runs' attachments | App and report data | App and `settings.json` only |
164
+ | In the bucket | Nothing | Run folders with attachments | Run folders and the report data |
165
+ | Site rebuilt | Every run | Every run | Once, and when the version or settings change |
166
+ | History kept in | A `report-data` branch | The bucket's run folders | The bucket |
167
+ | Dashboards need credentials | No | No | Yes, every viewer |
168
+ | Attachments need credentials | No | Yes | Yes |
169
+ | Who can read test names and errors | Anyone who can open the site | Anyone who can open the site | Only people with bucket access |
170
+ | Workflow | [pages-only.yml](docs/workflows/pages-only.yml) | [pages-s3-attachments.yml](docs/workflows/pages-s3-attachments.yml) | [s3-data.yml](docs/workflows/s3-data.yml) + [s3-data-site.yml](docs/workflows/s3-data-site.yml) |
171
+
172
+ Choose **1** to start, or when the site's readers may see everything. Choose **2** to keep large artifacts out of the
173
+ site. Choose **3** when the data must stay private and access to the site can't be limited, or to publish new runs without
174
+ redeploying. For all three, set Settings → Pages → Source to **GitHub Actions**.
175
+
176
+ ### Mode 1: GitHub Pages only
177
+
178
+ After the tests, the job adds the run to the report data and deploys the site. The report data is small (rollups and
179
+ JSONL), and is kept in a `report-data` branch between runs; `build` keeps the earlier runs from it. Attachments are
180
+ published only for the runs in the latest deploy.
181
+
182
+ ```yaml
183
+ - run: npx playwright test
184
+ - run: npx qaily-report build --in reports/data --out report-history # report-history: the report-data branch
185
+ - run: |
186
+ cp -r node_modules/@qaily/playwright-html-reporter/app site
187
+ cp -r report-history site/data
188
+ npx qaily-report trace-viewer --out site/trace
189
+ ```
190
+
191
+ [pages-only.yml](docs/workflows/pages-only.yml) has the full job, including the branch and the attachments.
192
+
193
+ ### Mode 2: GitHub Pages + bucket for artifacts
194
+
195
+ The test job uploads the run folder, attachments included. The report job rebuilds the report from every run in the
196
+ bucket, without downloading attachments, and deploys it. Viewers connect credentials to open screenshots and traces.
197
+
198
+ ```yaml
199
+ # test job
200
+ - run: npx playwright test
201
+ - run: npx qaily-report upload # reports/data → bucket, .sync-complete marker last
202
+ # report job
203
+ - run: npx qaily-report sync # run folders from the bucket, no attachments, then build
204
+ - run: |
205
+ cp -r node_modules/@qaily/playwright-html-reporter/app site
206
+ cp -r report-data site/data
207
+ npx qaily-report trace-viewer --out site/trace
208
+ ```
209
+
210
+ `report-data/settings.json` names the bucket ([settings for each mode](#settings-for-each-mode)). CI needs a key that
211
+ can upload and list the run folders. If run folders are deleted after a while, add `pull` before `sync` and `publish`
212
+ after it, so the bucket keeps the report data between builds.
213
+
214
+ ### Mode 3: all data in the bucket
215
+
216
+ The site is deployed once, with the app and `settings.json`. After each test run, CI publishes the report data to the
217
+ bucket, and viewers see the new run on their next load.
218
+
219
+ ```yaml
220
+ # after the tests, every run (s3-data.yml)
221
+ - run: npx qaily-report upload
222
+ - run: npx qaily-report pull # what was published last time
223
+ - run: npx qaily-report sync # run folders → rebuild the months that changed
224
+ - run: npx qaily-report publish # report data → bucket, catalog.json last
225
+ # when the version or settings change (s3-data-site.yml)
226
+ - run: |
227
+ cp -r node_modules/@qaily/playwright-html-reporter/app site
228
+ mkdir -p site/data && cp report-data/settings.json site/data/
229
+ npx qaily-report trace-viewer --out site/trace
230
+ ```
231
+
232
+ Viewers paste credentials in Settings → Tokens. Team settings (lanes, thresholds, heat map rules) can then be saved from
233
+ the report to the bucket, with no redeploy ([docs/storage.md](docs/storage.md#7-team-settings-in-the-bucket)). Run folders
234
+ can expire while the dashboards keep every run.
235
+
236
+ ### Reading the bucket from the browser: CORS or the storage bridge
237
+
238
+ In modes 2 and 3 the page reads the bucket from the viewer's browser, which needs the bucket's permission (CORS):
239
+
240
+ - **You can set CORS rules:** allow the site's origin, e.g. `https://<org>.github.io`, for `GET` and `HEAD` (and `PUT` to
241
+ save team settings). Examples for each provider are in [docs/storage.md](docs/storage.md#3-cors).
242
+ - **You can't** (Dell ECS, or no admin rights): use the **storage bridge**. Upload a small page to the bucket once, and set
243
+ Settings → Connections → **Storage access** to **Bridge**:
244
+
245
+ ```bash
246
+ npx qaily-report bridge --origin https://<org>.github.io --aws-profile qaily
247
+ ```
248
+
249
+ The report then reads the bucket through that page, with no CORS rules. Each viewer can switch between Direct and Bridge
250
+ under **For me**. How it works and what it allows: [docs/storage.md](docs/storage.md#9-storage-without-cors-the-storage-bridge).
251
+
252
+ Without CORS or the bridge, mode 2 still shows dashboards, screenshots and downloads (on most storage), but not text
253
+ attachments; mode 3 shows nothing. `qaily-report show` needs neither.
254
+
255
+ **Traces** open in the trace viewer copied next to the report (`qaily-report trace-viewer`, with
256
+ `"traceViewer": "trace/"`). The report hands it the trace file, so the viewer needs no CORS of its own
257
+ ([docs/storage.md](docs/storage.md#10-self-hosted-trace-viewer)).
258
+
259
+ ## 5. Storage and credentials
260
+
261
+ | Storage | `endpoint` | `region` | `pathStyle` | Browser access |
262
+ |---|---|---|---|---|
263
+ | AWS S3 (tested) | empty | the bucket's region | `false` | CORS |
264
+ | Google Cloud Storage (tested) | `https://storage.googleapis.com` | `auto` | `false` | CORS |
265
+ | MinIO | the server's URL | usually `us-east-1` | `true` | CORS (server-wide) |
266
+ | Dell ECS | the server's URL | any, e.g. `us-east-1` | `true` | Storage bridge |
267
+
268
+ The commands log in through the AWS CLI v2, for every provider:
269
+
270
+ ```bash
271
+ aws configure --profile qaily # access key and secret (HMAC key for Google Cloud Storage)
272
+ aws configure sso --profile qaily # or AWS IAM Identity Center; then: aws sso login --profile qaily
273
+ npx qaily-report show --sync --aws-profile qaily # or AWS_PROFILE=qaily, or the AWS_ACCESS_KEY_ID variables
72
274
  ```
73
275
 
74
- ## 2. Build and view the report
276
+ For the browser (Settings → Tokens), paste one of:
75
277
 
76
278
  ```bash
77
- npx qaily-report build --in reports/data --out report-data # rollups for the dashboards
78
- npx qaily-report serve --data report-data --attachments reports/data
79
- # → http://127.0.0.1:4173
279
+ aws configure export-credentials --profile qaily --format process
280
+ aws sts get-session-token --profile qaily --duration-seconds 43200 --query Credentials --output json # 12-hour keys
281
+ ```
282
+
283
+ [docs/storage.md](docs/storage.md) has the rest:
284
+ - provider notes;
285
+ - CORS rules;
286
+ - IAM policies for viewers, CI and the retention job;
287
+ - certificates for internal storage;
288
+ - what the page stores and for how long.
289
+
290
+ ## 6. Settings
291
+
292
+ Settings come in three layers, the later ones winning:
293
+
294
+ 1. **Site:** `settings.json` next to the report data. In this README it's `report-data/settings.json`, committed and
295
+ deployed by CI.
296
+ 2. **Team settings in the bucket** (mode 3): `team-settings.json` in the data folder. **Save to bucket** writes it, and it
297
+ never contains the bucket connection.
298
+ 3. **Personal:** Settings → **For me**, kept in each viewer's browser.
299
+
300
+ Edit them in the report: Settings shows where each value comes from, and **For the team** saves to the right place
301
+ (**Save to settings.json**, **Save to bucket** or **Commit to GitHub**). Or edit the JSON. Unset fields keep their
302
+ defaults.
303
+
304
+ ### Settings for each mode
305
+
306
+ Mode 1, no bucket:
307
+
308
+ ```json
309
+ {
310
+ "lanes": [
311
+ { "id": "main", "label": "Main branch", "match": { "branch": "^main$" } },
312
+ { "id": "other", "label": "Other branches", "match": {} }
313
+ ],
314
+ "connections": { "traceViewer": "trace/" }
315
+ }
316
+ ```
317
+
318
+ Mode 2: the same, plus the bucket:
319
+
320
+ ```json
321
+ {
322
+ "connections": {
323
+ "traceViewer": "trace/",
324
+ "s3": { "bucket": "test-reports", "region": "eu-north-1", "prefix": "jsonl-report" }
325
+ }
326
+ }
327
+ ```
328
+
329
+ Mode 3: the data source and the data folder too. Add `"transport": "bridge"` for the storage bridge:
330
+
331
+ ```json
332
+ {
333
+ "connections": {
334
+ "traceViewer": "trace/",
335
+ "s3": {
336
+ "bucket": "test-reports",
337
+ "region": "eu-north-1",
338
+ "prefix": "jsonl-report",
339
+ "dataSource": "bucket",
340
+ "dataPrefix": "qaily-data"
341
+ }
342
+ }
343
+ }
80
344
  ```
81
345
 
82
- `build` only rebuilds months whose runs changed. Try it without tests: `npx qaily-report demo --out report-data`
83
- writes 120 synthetic runs, with heat map rules and a planned matrix.
346
+ For Google Cloud Storage, MinIO or ECS, add `endpoint` and `pathStyle` from [section 5](#5-storage-and-credentials).
347
+ `prefix` is where run folders go; `dataPrefix` must be outside it.
348
+
349
+ ### Lanes
350
+
351
+ A lane is a line of runs on the Monitoring page and a filter elsewhere. Its `match` fields are all optional: `category`
352
+ and `branch` are regular expressions, and `environment` must match exactly. Each run joins the first lane that matches,
353
+ so list specific lanes first.
354
+
355
+ ```json
356
+ { "id": "web-staging", "label": "Web · Staging", "match": { "category": "^ui:web$", "environment": "staging" } }
357
+ ```
84
358
 
85
- Everything else is set on the report's **Settings** page: lanes, health thresholds, theme, the bucket, heat map rules.
86
- Under **For the team**, the save button says where it writes. **Save to settings.json** writes `report-data/settings.json` when the report is opened through `serve` (on GitHub Pages it
87
- can commit the file instead, see [docs/ai.md](docs/ai.md#drafts-and-publishing)). A report that reads its data from a
88
- bucket keeps team settings there instead (**Save to bucket**), as `team-settings.json` next to the data; the bucket
89
- connection itself always comes from the site's `settings.json` (see
90
- [docs/s3-artifacts.md](docs/s3-artifacts.md#team-settings-in-the-bucket)).
359
+ ### Coverage heat map
91
360
 
92
- ## 3. Share it
361
+ The heat map places steps on four levels: Component, Scenario, Feature and Area. Steps get there in two ways:
93
362
 
94
- Upload each run folder to a bucket from CI, with a `.sync-complete` marker last (the upload step is at the end of
95
- [docs/report-site.example.yml](docs/report-site.example.yml)). Then build the report from the bucket:
363
+ - **Mapping rules** in Settings → Coverage heat map, with no test changes:
364
+ ```json
365
+ { "coverage": { "rules": [ { "step": "^Archive board", "component": "Permissions", "feature": "Boards", "area": "Archive" } ] } }
366
+ ```
367
+ - **Labels on steps**, with Playwright 1.58+:
368
+ ```ts
369
+ import { coverage } from '@qaily/playwright-html-reporter';
370
+
371
+ await test.step('Archive board', async (step) => {
372
+ coverage(step, { component: 'Permissions', scenario: 'Workspace owners', feature: 'Boards', area: 'Archive' });
373
+ });
374
+ ```
375
+
376
+ A planned matrix shows the gaps: `npx qaily-report catalog --csv coverage/matrix.csv` writes
377
+ `report-data/coverage-catalog.json`. Details are in [docs/heatmap.md](docs/heatmap.md).
378
+
379
+ ## 7. Retention
380
+
381
+ Attachments are most of the storage: a run with traces can be 1 GB or more, while its JSONL is a few hundred KB. The
382
+ report is built to keep a year of light data and drop heavy artifacts after a few days. Example: 20 runs a day of
383
+ 1.5 GB is 900 GB a month with everything kept, and about 90 GB with attachments kept for 3 days.
384
+
385
+ | What | How long | How |
386
+ |---|---|---|
387
+ | Attachments | Days | `qaily-report expire-attachments`, on a schedule |
388
+ | Run folders | Weeks, or forever | A lifecycle rule on the run-folder prefix, or keep them |
389
+ | Report data (dashboards) | A year or more | Kept by `build`; `--prune-before YYYY-MM` drops older months |
390
+
391
+ ### Delete old attachments: `expire-attachments`
96
392
 
97
393
  ```bash
98
- npx qaily-report sync --to runs --out report-data # run folders without attachments, then build
394
+ npx qaily-report expire-attachments --older-than 3 --dry-run # what would be deleted
395
+ npx qaily-report expire-attachments --older-than 3 # delete it
99
396
  ```
100
397
 
101
- The bucket comes from `report-data/settings.json` (Settings → For the team → Connections), or from flags:
398
+ It deletes `<prefix>/<run>/attachments/` of runs whose attachments are all older than the given days, and keeps the rest
399
+ of each run folder. The dashboards, and other tools that read the run folders, keep working. The age lives in your
400
+ workflow, not in the bucket, so it works where you can't change the bucket's settings. Run it daily:
401
+ [expire-attachments.yml](docs/workflows/expire-attachments.yml).
402
+
403
+ A lifecycle rule can't do this, because its filter matches the start of a key and `attachments/` is in the middle. In a
404
+ versioned bucket, a delete only hides a file, so also add a rule that deletes old versions, or pass `--all-versions`.
405
+
406
+ The report then says, for an old run, that its attachments are no longer in storage. The test's error, steps and history
407
+ are still shown.
408
+
409
+ ### Delete old run folders: lifecycle rules
410
+
411
+ To delete whole run folders after, say, 30 days, add a lifecycle rule on the run-folder prefix. Dashboards keep those runs
412
+ as long as CI keeps the report data: mode 3 does this already; in mode 2, add `pull` and `publish`
413
+ ([docs/storage.md](docs/storage.md#11-when-run-folders-expire-too)).
414
+
415
+ AWS S3 (`aws s3api put-bucket-lifecycle-configuration --bucket <bucket> --lifecycle-configuration file://lifecycle.json`).
416
+ This replaces the bucket's existing rules, so include any you have:
102
417
 
103
418
  ```json
104
- { "connections": { "s3": { "bucket": "my-test-runs", "region": "eu-north-1", "prefix": "jsonl-report" } } }
419
+ {
420
+ "Rules": [
421
+ { "ID": "run-folders", "Status": "Enabled", "Filter": { "Prefix": "jsonl-report/" }, "Expiration": { "Days": 30 } },
422
+ { "ID": "old-versions", "Status": "Enabled", "Filter": {}, "NoncurrentVersionExpiration": { "NoncurrentDays": 1 } }
423
+ ]
424
+ }
105
425
  ```
106
426
 
107
- Copy `node_modules/@qaily/playwright-html-reporter/app` and your data folder (as `data/`) to any static host.
108
- [docs/report-site.example.yml](docs/report-site.example.yml) does this on GitHub Pages every 30 minutes.
427
+ Google Cloud Storage (`gcloud storage buckets update gs://<bucket> --lifecycle-file=lifecycle.json`):
109
428
 
110
- Dashboard data can live on the site (simplest) or in the bucket (private, viewers sign in with credentials). Both, and
111
- the bucket, CORS and IAM setup for AWS, Google Cloud Storage and MinIO, are in [docs/s3-artifacts.md](docs/s3-artifacts.md).
429
+ ```json
430
+ { "rule": [ { "action": { "type": "Delete" }, "condition": { "age": 30, "matchesPrefix": ["jsonl-report/"] } } ] }
431
+ ```
432
+
433
+ MinIO: `mc ilm rule add --expire-days 30 --prefix "jsonl-report/" <alias>/<bucket>`.
434
+
435
+ Dell ECS accepts the AWS rule through its S3 API (`--endpoint-url`) when the bucket's owner allows it. Otherwise use
436
+ `expire-attachments` only.
437
+
438
+ Rules count days from upload, and providers usually apply them once a day.
439
+
440
+ ## 8. Commands
112
441
 
113
- ## `qaily-report` commands
442
+ Run them as `npx qaily-report <command>`; `npx qaily-report help` lists them. The defaults match the setup above: run
443
+ folders in `reports/data`, report data in `report-data`, and the bucket from `report-data/settings.json`.
114
444
 
115
445
  | Command | What it does |
116
446
  |---|---|
117
- | `build --in <runs> --out <data> [--force] [--prune-before YYYY-MM]` | Turns run folders into the report's data, keeping runs whose folders have expired; `--force` rebuilds every month |
118
- | `serve --data <data> [--attachments <runs>] [--aws-profile p] [--port 4173]` | Local server; signs bucket links with this machine's login |
119
- | `sync --to <runs> --out <data> [--with-attachments]` | Downloads run folders from the bucket (attachments only if asked), then builds |
120
- | `pull` / `publish --out <data> [--team-settings seed\|replace\|skip] [--allow-shrink]` | Bucket data source: fetch the last published data / upload the new one, and team settings when the bucket has none yet. See [expiring run folders](docs/s3-artifacts.md#when-run-folders-expire-too) |
121
- | `expire-attachments --older-than <days> [--dry-run] [--all-versions]` | Deletes attachments of runs older than N days and keeps the rest of each run folder; run it daily. See [short retention](docs/s3-artifacts.md#8-short-retention-for-attachments) |
122
- | `bridge --origin <site origin> [--dry-run]` | Uploads the storage bridge page, for storage whose CORS rules you can't change (e.g. Dell ECS). See [storage bridge](docs/s3-artifacts.md#10-storage-without-cors-the-storage-bridge) |
123
- | `trace-viewer --out <site>/trace [--from <project>]` | Copies Playwright's trace viewer from the test project, so traces open on your own site without CORS. See [trace viewer](docs/s3-artifacts.md#9-self-hosted-trace-viewer) |
124
- | `demo --out <data> [--runs 120] [--days 30]` | Synthetic runs, for trying the report |
125
- | `catalog --csv <matrix.csv> --out <data>/coverage-catalog.json` | Planned coverage matrix for the heat map |
447
+ | `show` | Build from local run folders and serve |
448
+ | `serve` | Serve report data that's already built |
449
+ | `build` | Run folders → report data; only changed months are rebuilt |
450
+ | `upload` | Run folders → bucket, attachments included, marker last |
451
+ | `sync` | Bucket run folders → local folder (no attachments), then build |
452
+ | `pull` | Published report data → local folder |
453
+ | `publish` | Report data → bucket, `catalog.json` last |
454
+ | `expire-attachments` | Delete the attachments of old runs, keep the rest |
455
+ | `bridge` | Upload the storage bridge page |
456
+ | `trace-viewer` | Copy Playwright's trace viewer from the test project |
457
+ | `catalog` | Planned coverage matrix for the heat map |
458
+ | `demo` | Synthetic runs, to try the report |
459
+
460
+ Options, with their defaults:
461
+
462
+ ```text
463
+ show --in reports/data --out report-data --port 4173
464
+ --sync run folders from the bucket instead
465
+ --pull published report data from the bucket instead
466
+ serve --data report-data --attachments <run folders>
467
+ --port 4173 --host 127.0.0.1 --app <built app>
468
+ build --in reports/data --out report-data
469
+ --force rebuild every month
470
+ --prune-before YYYY-MM drop older months for good
471
+ upload --in reports/data --dry-run
472
+ sync --to .qaily/runs --out report-data --with-attachments
473
+ pull --out report-data
474
+ publish --out report-data --allow-shrink
475
+ --team-settings seed|replace|skip team-settings.json
476
+ expire-attachments --older-than <days> (required)
477
+ --dry-run --all-versions
478
+ bridge --origin <site origin> (required) --dry-run
479
+ trace-viewer --out <site>/trace (required) --from <test project>
480
+ catalog --csv <matrix.csv> (required)
481
+ --out report-data/coverage-catalog.json
482
+ demo --out report-data --runs 120 --days 30
483
+ ```
484
+
485
+ Bucket options, for every command that uses a bucket:
126
486
 
127
- Bucket, region, key prefix and endpoint come from `<data>/settings.json`, or flags: `--bucket`, `--region`, `--prefix`,
128
- `--endpoint` (Google Cloud Storage, MinIO), `--data-prefix`. `--aws-profile` picks the AWS CLI profile.
487
+ ```text
488
+ --bucket <name> --region <region> --endpoint <url> --path-style
489
+ --prefix <run folders> --data-prefix <report data>
490
+ --aws-profile <name>
491
+ ```
129
492
 
130
- ## Develop
493
+ ## 9. Develop and release
131
494
 
132
495
  ```bash
133
496
  npm ci
@@ -137,11 +500,8 @@ npm run app:test && npm run app:typecheck
137
500
  npm run pkg:build # dist/ and app/, as published to npm
138
501
  ```
139
502
 
140
- The workspace layout is described in [packages/README.md](packages/README.md).
141
-
142
- ## Releasing
143
-
144
- Actions → **Release** → Run workflow (bump and channel), as for `@qaily/playwright-jsonl-reporter`. See [docs/releasing.md](docs/releasing.md).
503
+ The workspace layout is in [packages/README.md](packages/README.md). Releases: Actions → **Release** → Run workflow,
504
+ see [docs/releasing.md](docs/releasing.md).
145
505
 
146
506
  ## License
147
507