monocart-reporter 2.9.5 → 2.9.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,1177 +1,1180 @@
1
- # Monocart Reporter
2
-
3
- [![](https://img.shields.io/npm/v/monocart-reporter)](https://www.npmjs.com/package/monocart-reporter)
4
- [![](https://devimg.vercel.app/npm/downloads/monocart-reporter?label={total}%20downloads/month)](https://www.npmjs.com/package/monocart-reporter)
5
- ![](https://img.shields.io/librariesio/github/cenfun/monocart-reporter)
6
- ![](https://img.shields.io/github/license/cenfun/monocart-reporter)
7
- ![](https://img.shields.io/github/actions/workflow/status/cenfun/monocart-reporter/static.yml)
8
-
9
-
10
- * A [Playwright](https://github.com/microsoft/playwright) Test [Reporter](https://playwright.dev/docs/test-reporters) (Node.js)
11
- - A `Tree Grid` style test reporter
12
- - Support processing big data with high performance
13
- - Design for customization and extensibility
14
- - Interactive report with grouping and ultra-fast filter
15
- - Timeline Workers Graph
16
- - Monitor CPU and Memory Usage
17
- - Export Data (json)
18
- * [Preview](#preview)
19
- * [Install](#install)
20
- * [Playwright Config](#playwright-config)
21
- * [Examples](#examples)
22
- * [Output](#output)
23
- * [Reporter Options](#reporter-options)
24
- * [View Trace Online](#view-trace-online)
25
- * [Custom Fields Report](#custom-fields-report)
26
- * [Custom Columns](#custom-columns)
27
- - [Column Formatter](#column-formatter)
28
- - [Searchable Fields](#searchable-fields)
29
- * [Custom Fields in Comments](#custom-fields-in-comments)
30
- - [Create Diagrams and Visualizations with Mermaid](#create-diagrams-and-visualizations-with-mermaid)
31
- * [Custom Fields with `setMetadata()`](#custom-fields-with-setmetadata)
32
- * [Custom Data Visitor](#custom-data-visitor)
33
- - [Collect Data from the Title](#collect-data-from-the-title)
34
- - [Collect Data from the Annotations](#collect-data-from-the-annotations)
35
- - [Remove Secrets and Sensitive Data](#remove-secrets-and-sensitive-data)
36
- * [Style Tags](#style-tags)
37
- * [Metadata](#metadata)
38
- * [Trend Chart](#trend-chart)
39
- * [Code Coverage Report](#code-coverage-report)
40
- - [Global Coverage Report](#global-coverage-report)
41
- - [Coverage Options](#coverage-options)
42
- - [Coverage Examples](#coverage-examples)
43
- * [Attach Lighthouse Audit Report](#attach-lighthouse-audit-report)
44
- * [Attach Network Report](#attach-network-report)
45
- * [Global State Management](#global-state-management)
46
- - [Setup Global State](#setup-global-state)
47
- - [Get, Set, and Remove Global Data](#get-set-and-remove-global-data)
48
- - [Send and Receive Messages between Processes](#send-and-receive-messages-between-processes)
49
- * [Merge Shard Reports](#merge-shard-reports)
50
- * [onEnd Hook](#onend-hook)
51
- * [Integration Examples](#integration-examples)
52
- * [Contributing](#contributing)
53
- * [Changelog](CHANGELOG.md)
54
-
55
-
56
- ## Preview
57
- [https://cenfun.github.io/monocart-reporter](https://cenfun.github.io/monocart-reporter)
58
-
59
- ![](/docs/report.gif)
60
-
61
- ![](/docs/cli.png)
62
- (For Github actions, we can enforce color with env: `FORCE_COLOR: true`)
63
-
64
- ## Install
65
- ```sh
66
- npm i -D monocart-reporter
67
- ```
68
-
69
- ## Playwright Config
70
- > Note: Most examples use `CommonJS` by default, please [move to ESM](https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c#how-can-i-move-my-commonjs-project-to-esm) according to your needs.
71
- ```js
72
- // playwright.config.js
73
- module.exports = {
74
- reporter: [
75
- ['list'],
76
- ['monocart-reporter', {
77
- name: "My Test Report",
78
- outputFile: './monocart-report/index.html'
79
- }]
80
- ]
81
- };
82
- ```
83
- Playwright Docs [https://playwright.dev/docs/test-reporters](https://playwright.dev/docs/test-reporters)
84
-
85
- ## Examples
86
- - [tests](/tests/)
87
- - [monocart-reporter-examples](https://github.com/cenfun/monocart-reporter-examples)
88
- - [playwright-reporter-integrations](https://github.com/cenfun/playwright-reporter-integrations)
89
-
90
- ## Output
91
- - path-to/your-filename.html
92
- Single HTML file (data compressed), easy to transfer/deploy or open directly anywhere
93
- > Note: All attachments (screenshots images/videos) will be linked with relative path in report.
94
- - path-to/your-filename.json (requires option `json` is true)
95
- Separated data file which can be used for debugging or data provider (It's included in the above HTML and compressed).
96
- - path-to/your-filename.zip (requires option `zip` is true)
97
- Zip file for merging reports
98
-
99
- ## Reporter Options
100
- - Default options: [lib/default/options.js](./lib/default/options.js)
101
- - Options declaration see `MonocartReporterOptions` [lib/index.d.ts](./lib/index.d.ts)
102
-
103
- ## View Trace Online
104
- > The [Trace Viewer](https://trace.playwright.dev/) requires that the trace file must be loaded over the http:// or https:// protocols without [CORS](https://developer.mozilla.org/en-US/docs/Glossary/CORS) issue.
105
- - Start a local web server with following CLI:
106
- ```sh
107
- # serve and open report
108
- npx monocart show-report <path-to-report>
109
-
110
- # serve report
111
- npx monocart serve-report <path-to-report>
112
- ```
113
- The server add the http header `Access-Control-Allow-Origin: *` to [allow requesting from any origin](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin), it works with `http://localhost:port/` or `http://127.0.0.1:port/`
114
- - To successfully work with other `IP` or `domain`, you can start web server with `https`:
115
- ```sh
116
- npx monocart show-report <path-to-report> --ssl <path-to-key,path-to-cert>
117
- ```
118
- For example: `npx monocart show-report monocart-report/index.html --ssl ssl/key.pem,ssl/cert.pem`
119
-
120
- You can create and install local CA with [mkcert](https://mkcert.dev)
121
- - Using your own trace viewer url with option `traceViewerUrl`:
122
- ```js
123
- // reporter options
124
- {
125
- name: "My Test Report",
126
- outputFile: './monocart-report/index.html',
127
- // defaults to 'https://trace.playwright.dev/?trace={traceUrl}'
128
- traceViewerUrl: 'https://your-own-trace-viewer-url/?trace={traceUrl}'
129
- }
130
- ```
131
-
132
- ## Custom Fields Report
133
- You can add custom fields to the report. for example: Owner, JIRA Key etc.
134
- - First, you need to add [Custom Columns](#custom-columns) for the fields.
135
- - Then, collect data for these fields
136
- - [Custom Fields in Comments](#custom-fields-in-comments)
137
- - [Custom Fields with `setMetadata()`](#custom-fields-with-setmetadata)
138
- - [Custom Data Visitor](#custom-data-visitor)
139
-
140
- ### Custom Columns
141
- The report will be displayed in a `Tree Grid`. The `columns` function is used to customize the grid columns. The column properties following:
142
- - `id` (String) Column id (required)
143
- - `name` (String) Column name, shows in grid header
144
- - `align` (String) left (default), center, right
145
- - `width` (Number) Column width
146
- - `minWidth`, `maxWidth` (Number) Default to 81 ~ 300
147
- - `styleMap` (Object, String) Column style (css)
148
- - `formatter` (Function) [column formatter](#custom-formatter). Arguments: value, rowItem, columnItem, cellNode
149
- - `sortable` (Boolean) Column sortable when click column header name
150
- - `resizable` (Boolean) Column width resizable
151
- - `searchable` (Boolean) Specifies whether the column is searchable
152
- - `markdown` (Boolean) Specifies whether the column needs to use markdown conversion
153
- - `detailed` (Boolean) Specifies whether the column needs to display the layout in detail (horizontal)
154
- - more properties [columnProps](https://cenfun.github.io/turbogrid/api.html#options.columnProps)
155
- ```js
156
- // playwright.config.js
157
- module.exports = {
158
- reporter: [
159
- ['monocart-reporter', {
160
- name: "My Test Report",
161
- outputFile: './monocart-report/index.html',
162
-
163
- // custom columns
164
- columns: (defaultColumns) => {
165
-
166
- // insert custom column(s) before a default column
167
- const index = defaultColumns.findIndex((column) => column.id === 'duration');
168
- defaultColumns.splice(index, 0, {
169
- // define the column in reporter
170
- id: 'owner',
171
- name: 'Owner',
172
- align: 'center',
173
- searchable: true,
174
- styleMap: {
175
- 'font-weight': 'normal'
176
- }
177
- }, {
178
- // another column for JIRA link
179
- id: 'jira',
180
- name: 'JIRA Key',
181
- width: 100,
182
- searchable: true,
183
- styleMap: 'font-weight:normal;',
184
- formatter: (v, rowItem, columnItem) => {
185
- const key = rowItem[columnItem.id];
186
- return `<a href="https://your-jira-url/${key}" target="_blank">${v}</a>`;
187
- }
188
- });
189
-
190
- }
191
- }]
192
- ]
193
- };
194
- ```
195
- #### Column Formatter
196
- > Note: The `formatter` function will be serialized into string via JSON, so closures, contexts, etc. will not work!
197
- ```js
198
- // playwright.config.js
199
- module.exports = {
200
- reporter: [
201
- ['monocart-reporter', {
202
- name: "My Test Report",
203
- outputFile: './monocart-report/index.html',
204
- columns: (defaultColumns) => {
205
-
206
- // duration formatter
207
- const durationColumn = defaultColumns.find((column) => column.id === 'duration');
208
- durationColumn.formatter = function(value, rowItem, columnItem) {
209
- if (typeof value === 'number' && value) {
210
- return `<i>${value.toLocaleString()} ms</i>`;
211
- }
212
- return value;
213
- };
214
-
215
- // title formatter
216
- // Note: The title shows the tree style, it is a complicated HTML structure
217
- // it is recommended to format title base on previous.
218
- const titleColumn = defaultColumns.find((column) => column.id === 'title');
219
- titleColumn.formatter = function(value, rowItem, columnItem, cellNode) {
220
- const perviousFormatter = this.getFormatter('tree');
221
- const v = perviousFormatter(value, rowItem, columnItem, cellNode);
222
- if (rowItem.type === 'step') {
223
- return `${v}<div style="position:absolute;top:0;right:5px;">✅</div>`;
224
- }
225
- return v;
226
- };
227
-
228
- }
229
- }]
230
- ]
231
- };
232
- ```
233
-
234
- #### Searchable Fields
235
- ```js
236
- // playwright.config.js
237
- module.exports = {
238
- reporter: [
239
- ['monocart-reporter', {
240
- name: "My Test Report",
241
- outputFile: './monocart-report/index.html',
242
- columns: (defaultColumns) => {
243
- const locationColumn = defaultColumns.find((column) => column.id === 'location');
244
- locationColumn.searchable = true;
245
- }
246
- }]
247
- ]
248
- };
249
- ```
250
-
251
- ### Custom Fields in Comments
252
- > The code comments are good enough to provide extra information without breaking existing code, and no dependencies, clean, easy to read, etc.
253
- - First, enable option `customFieldsInComments` to `true`
254
- ```js
255
- // playwright.config.js
256
- module.exports = {
257
- reporter: [
258
- ['monocart-reporter', {
259
- // enable/disable custom fields in comments. Defaults to true.
260
- customFieldsInComments: true
261
- }]
262
- ]
263
- };
264
- ```
265
-
266
- - Then, add comments for the tests
267
- > Note: Each comment item must start with `@` which is similar to [JSDoc](https://jsdoc.app/).
268
-
269
- For example, adding `owner` and `jira` to the cases, steps, and suites. or updating the value if the field exists like `title`
270
- ```js
271
- /**
272
- * for file (comment file in the first line)
273
- * @owner FO
274
- */
275
- const { test, expect } = require('@playwright/test');
276
-
277
- /**
278
- * for case
279
- * @owner Kevin
280
- * @jira MCR-16888
281
- */
282
- test('case title', () => {
283
-
284
- });
285
-
286
- /**
287
- * @description multiple lines text description
288
- multiple lines text description
289
- multiple lines text description
290
- * @jira MCR-16888
291
- */
292
- test('case description', () => {
293
-
294
- });
295
-
296
- /**
297
- * for describe suite
298
- * @owner Mark
299
- * @jira MCR-16900
300
- */
301
- test.describe('suite title', () => {
302
-
303
- test('case title', ({ browserName }, testInfo) => {
304
-
305
- /**
306
- * rewrite assert step title "expect.toBe" to
307
- * @title my custom assert step title
308
- * @annotations important
309
- */
310
- expect(testInfo).toBe(test.info());
311
-
312
- // @owner Steve
313
- await test.step('step title', () => {
314
-
315
- });
316
-
317
- });
318
-
319
- });
320
-
321
- /**
322
- * rewrite "beforeAll hook" title to
323
- * @title do something before all
324
- */
325
- test.beforeAll(() => {
326
-
327
- });
328
-
329
- /**
330
- * rewrite "beforeEach hook" title to
331
- * @title do something before each
332
- */
333
- test.beforeEach(() => {
334
-
335
- });
336
- ```
337
-
338
- #### Create Diagrams and Visualizations with [Mermaid](https://mermaid.js.org/)
339
- - Enable Mermaid
340
- ```js
341
- // playwright.config.js
342
- module.exports = {
343
- reporter: [
344
- ['monocart-reporter', {
345
- name: "My Test Report",
346
- outputFile: './monocart-report/index.html',
347
- mermaid: {
348
- // mermaid script url, using mermaid CDN: https://www.jsdelivr.com/package/npm/mermaid
349
- scriptSrc: 'https://cdn.jsdelivr.net/npm/mermaid@latest/dist/mermaid.min.js',
350
- // mermaid config: https://mermaid.js.org/config/schema-docs/config.html
351
- config: {
352
- startOnLoad: false
353
- }
354
- }
355
- }]
356
- ]
357
- };
358
- ```
359
- - Write Mermaid code in markdown:
360
- ````js
361
- /**
362
- * @description Sequence diagram for Monocart Reporter
363
- ```mermaid
364
- flowchart LR
365
-
366
- A[Hard] -->|Text| B(Round)
367
- B --> C{Decision}
368
- C -->|One| D[Result 1]
369
- C -->|Two| E[Result 2]
370
- ```
371
- */
372
- test('case description', () => {
373
-
374
- });
375
- ````
376
- see [Mermaid doc](https://mermaid.js.org/syntax/flowchart.html)
377
-
378
-
379
- ### Custom Fields with `setMetadata()`
380
- Using `comments` is only applicable to statically created tests, while using API `setMetadata()` can be applicable to all situations, which is including dynamically created tests.
381
- ```js
382
- const { test } = require('@playwright/test');
383
- const { setMetadata } = require('monocart-reporter');
384
- test.describe('Data Driven Tests with setMetadata(data, testInfo)', () => {
385
- const list = [{
386
- title: 'Example Case 1 Data Driven Test',
387
- owner: 'Jensen',
388
- jira: 'MCR-16889',
389
- }, {
390
- title: 'Example Case 2 Data Driven Test',
391
- owner: 'Mark',
392
- jira: 'MCR-16899'
393
- }];
394
- list.forEach((item, i) => {
395
- test(item.title, () => {
396
- setMetadata({
397
- owner: item.owner,
398
- jira: item.jira
399
- }, test.info());
400
-
401
- //expect(1).toBe(1);
402
-
403
- });
404
- });
405
- });
406
- ```
407
-
408
- ### Custom Data Visitor
409
- The `visitor` function will be executed for each row item (suite, case and step). Arguments:
410
- - `data` data item (suite/case/step) for reporter, you can rewrite some of its properties or add more
411
- - `metadata` original data object from Playwright test, could be one of [Suite](https://playwright.dev/docs/api/class-suite), [TestCase](https://playwright.dev/docs/api/class-testcase) or [TestStep](https://playwright.dev/docs/api/class-teststep)
412
-
413
- #### Collect Data from the Title
414
- For example, we want to parse out the jira key from the title:
415
- ```js
416
- test('[MCR-123] collect data from the title', () => {
417
-
418
- });
419
- ```
420
- You can simply use regular expressions to parse and get jira key:
421
- ```js
422
- // playwright.config.js
423
- module.exports = {
424
- reporter: [
425
- ['monocart-reporter', {
426
- name: "My Test Report",
427
- outputFile: './monocart-report/index.html',
428
- visitor: (data, metadata) => {
429
- // [MCR-123] collect data from the title
430
- const matchResult = metadata.title.match(/\[(.+)\]/);
431
- if (matchResult && matchResult[1]) {
432
- data.jira = matchResult[1];
433
- }
434
- }
435
- }]
436
- ]
437
- };
438
- ```
439
- multiple matches example: [collect-data](https://github.com/cenfun/monocart-reporter-examples/tree/main/tests/collect-data)
440
-
441
- #### Collect Data from the Annotations
442
- It should be easier than getting from title. see [custom annotations](https://playwright.dev/docs/test-annotations#custom-annotations) via `test.info().annotations`
443
- ```js
444
- test('collect data from the annotations', () => {
445
- test.info().annotations.push({
446
- type: "jira",
447
- description: "MCR-123"
448
- })
449
- });
450
- ```
451
- ```js
452
- // playwright.config.js
453
- module.exports = {
454
- reporter: [
455
- ['monocart-reporter', {
456
- name: "My Test Report",
457
- outputFile: './monocart-report/index.html',
458
- visitor: (data, metadata) => {
459
- // collect data from the annotations
460
- if (metadata.annotations) {
461
- const jiraItem = metadata.annotations.find((item) => item.type === 'jira');
462
- if (jiraItem && jiraItem.description) {
463
- data.jira = jiraItem.description;
464
- }
465
- }
466
- }
467
- }]
468
- ]
469
- };
470
- ```
471
-
472
- #### Remove Secrets and Sensitive Data
473
- > The report may hosted outside of the organization’s internal boundaries, security becomes a big issue. Any secrets or sensitive data, such as usernames, passwords, tokens and API keys, should be handled with extreme care. The following example is removing the password and token from the report data with the string replacement in `visitor` function.
474
- ```js
475
- // playwright.config.js
476
- module.exports = {
477
- reporter: [
478
- ['monocart-reporter', {
479
- name: "My Test Report",
480
- outputFile: './monocart-report/index.html',
481
- visitor: (data, metadata) => {
482
- const mySecrets = [process.env.PASSWORD, process.env.TOKEN];
483
- mySecrets.forEach((secret) => {
484
- // remove from title
485
- data.title = data.title.replace(secret, '***');
486
- // remove from logs
487
- if (data.logs) {
488
- data.logs = data.logs.map((item) => item.replace(secret, '***'));
489
- }
490
- // remove from errors
491
- if (data.errors) {
492
- data.errors = data.errors.map((item) => item.replace(secret, '***'));
493
- }
494
- });
495
- }
496
- }]
497
- ]
498
- };
499
- ```
500
- see example: [remove-secrets](https://github.com/cenfun/monocart-reporter-examples/tree/main/tests/remove-secrets)
501
-
502
- ## Style Tags
503
-
504
- * Add tag to test/describe title ( starts with `@` )
505
- ```js
506
- test('test title @smoke @critical', () => { });
507
- test.describe('describe title @smoke @critical', () => { });
508
-
509
- // new syntax for tag in playwright v1.42.0
510
- test('test title', { tag: ['@smoke', '@critical'] }, () => { });
511
- test.describe('describe title', { tag: ['@smoke', '@critical'] }, () => { });
512
- ```
513
-
514
- * Custom tag style
515
- ```js
516
- // playwright.config.js
517
- module.exports = {
518
- reporter: [
519
- ['monocart-reporter', {
520
- name: "My Test Report",
521
- outputFile: './monocart-report/index.html',
522
- tags: {
523
- smoke: {
524
- style: {
525
- background: '#6F9913'
526
- },
527
- description: 'This is Smoke Test'
528
- },
529
- critical: {
530
- background: '#c00'
531
- }
532
- }
533
- }]
534
- ]
535
- };
536
- ```
537
-
538
- * Put style tags in new column
539
- ```js
540
- module.exports = {
541
- reporter: [
542
- ['list'],
543
- ['monocart-reporter', {
544
- name: "My Test Report",
545
- outputFile: './monocart-report/index.html',
546
- tags: {
547
- // ...
548
- },
549
- columns: (defaultColumns) => {
550
-
551
- // disable title tags
552
- defaultColumns.find((column) => column.id === 'title').titleTagsDisabled = true;
553
-
554
- // add tags column
555
- const index = defaultColumns.findIndex((column) => column.id === 'type');
556
- defaultColumns.splice(index, 0, {
557
- id: 'tags',
558
- name: 'Tags',
559
- width: 150,
560
- formatter: 'tags'
561
- });
562
- }
563
- }]
564
- ]
565
- };
566
- ```
567
- see [example](https://github.com/cenfun/monocart-reporter-examples/tree/main/tests/tags-column)
568
-
569
-
570
- ## Metadata
571
- > All metadata will be listed in the report in a key/value format.
572
- - Global level `metadata`
573
- ```js
574
- // playwright.config.js
575
- module.exports = {
576
- globalSetup: require.resolve('./common/global-setup.js'),
577
- metadata: {
578
- product: 'Monocart',
579
- env: 'STG',
580
- type: 'Regression',
581
- executor: 'Mono',
582
-
583
- // test home page object model
584
- url: 'https://www.npmjs.org/package/monocart-reporter'
585
- },
586
- reporter: [
587
- ['monocart-reporter', {
588
- name: "My Test Report",
589
- outputFile: './monocart-report/index.html'
590
- }]
591
- ]
592
- };
593
- ```
594
-
595
- - Project level `metadata`
596
- ```js
597
- // playwright.config.js
598
- module.exports = {
599
- projects: [
600
- {
601
- name: 'Desktop Chromium',
602
- use: {
603
- browserName: 'chromium'
604
- },
605
- metadata: {
606
- projectData: 'project level metadata',
607
- owner: 'PO',
608
- link: 'https://github.com/cenfun/monocart-reporter'
609
- }
610
- }
611
- ]
612
- }
613
- ```
614
-
615
- - Collect metadata in [global setup or teardown](https://playwright.dev/docs/test-global-setup-teardown)
616
- ```js
617
- // ./common/global-setup.js
618
- import { chromium } from '@playwright/test';
619
- export default async (config) => {
620
- const metadata = config.metadata;
621
- // collect data and save to global metadata
622
- const browser = await chromium.launch();
623
- const chromiumVersion = await browser.version();
624
- metadata.chromiumVersion = chromiumVersion;
625
- };
626
- ```
627
-
628
- - Collect metadata in a test
629
- > Playwright Test runs tests in [parallel](https://playwright.dev/docs/test-parallel) with isolate test data by default, so we need to utilize [global state management](#global-state-management) to collect metadata in a test.
630
-
631
- - Enable global state, see [Setup Global State](#setup-global-state)
632
- - Update global state in a test
633
- ```js
634
- const { test } = require('@playwright/test');
635
- const { useState } = require('monocart-reporter');
636
-
637
- const state = useState({
638
- // port: 8130
639
- });
640
-
641
- test('test metadata url', async ({ page }) => {
642
- const url = await page.evaluate(() => window.location.href);
643
- await state.set('url', url);
644
- });
645
- ```
646
-
647
- ## Trend Chart
648
- > Note: The trend chart requires historical data generally stored in the server database. There is a serverless solution which is connecting and collecting historical trend data from previous report data before test every time.
649
- - If a report is generated in the same place every time, you can simply connect the data with the report JSON path (the data is not 100% safe if there is any runtime error, the previous output dir will be empty by Playwright but the reporter processing not finish)
650
- ```js
651
- // playwright.config.js
652
- module.exports = {
653
- reporter: [
654
- ['monocart-reporter', {
655
- name: "My Test Report",
656
- outputFile: './monocart-report/index.html',
657
- // connect previous report data for trend chart
658
- trend: './monocart-report/index.json'
659
- }]
660
- ]
661
- };
662
- ```
663
- - Recommended: resolve the data by yourself (could be requested from the server), required data fields:
664
- - `date` (Number) the previous test date in milliseconds
665
- - `duration` (Number) the previous test duration
666
- - `summary` (Object) the previous test summary
667
- - `trends` (Array) historical data list, but except the previous self
668
- ```js
669
- // playwright.config.js
670
- module.exports = {
671
- reporter: [
672
- ['monocart-reporter', {
673
- name: "My Test Report",
674
- outputFile: './monocart-report/index.html',
675
- // connect previous report data for trend chart
676
- trend: async () => {
677
- const previousReportData = await readDataFromSomeWhere("path-to/report.json");
678
- // do some data filtering to previous trends
679
- previousReportData.trends = previousReportData.trends.filter((item) => {
680
- // remove data a week ago
681
- return item.date > (Date.now() - 7 * 24 * 60 * 60 * 1000)
682
- });
683
- return previousReportData;
684
- }
685
- }]
686
- ]
687
- };
688
- ```
689
-
690
- ## Code Coverage Report
691
- The reporter integrates [monocart-coverage-reports](https://github.com/cenfun/monocart-coverage-reports) for coverage reports, there are two APIs:
692
- - `addCoverageReport(data, testInfo)` Add coverage to global coverage report from a test. see [Global Coverage Report](#global-coverage-report)
693
- - `data` There are two supported data inputs: [Istanbul](https://github.com/cenfun/monocart-coverage-reports?#collecting-istanbul-coverage-data) (Object) or [V8](https://github.com/cenfun/monocart-coverage-reports?#collecting-v8-coverage-data) (Array)
694
- - `testInfo` see [TestInfo](https://playwright.dev/docs/api/class-testinfo)
695
- - `attachCoverageReport(data, testInfo, options)` Attach a coverage report to a test. Arguments:
696
- - `data` same as above
697
- - `testInfo` same as above
698
- - `options` (Object) see [Coverage Options](#coverage-options)
699
-
700
- ![](/docs/v8.gif)
701
-
702
- ### Global Coverage Report
703
- The global coverage report will merge all coverages into one global report after all the tests are finished.
704
- - The global coverage options see [Coverage Options](#coverage-options)
705
- ```js
706
- // playwright.config.js
707
- module.exports = {
708
- reporter: [
709
- ['monocart-reporter', {
710
- name: "My Test Report",
711
- outputFile: './monocart-report/index.html',
712
- // global coverage report options
713
- coverage: {
714
- entryFilter: (entry) => true,
715
- sourceFilter: (sourcePath) => sourcePath.search(/src\/.+/) !== -1,
716
- }
717
- }]
718
- ]
719
- };
720
- ```
721
- - Adding coverage data for each test with [automatic fixtures](https://playwright.dev/docs/test-fixtures#automatic-fixtures)
722
- ```js
723
- // fixtures.js for v8 coverage
724
- import { test as testBase, expect } from '@playwright/test';
725
- import { addCoverageReport } from 'monocart-reporter';
726
-
727
- const test = testBase.extend({
728
- autoTestFixture: [async ({ page }, use) => {
729
-
730
- // NOTE: it depends on your project name
731
- const isChromium = test.info().project.name === 'Desktop Chromium';
732
-
733
- // console.log('autoTestFixture setup...');
734
- // coverage API is chromium only
735
- if (isChromium) {
736
- await Promise.all([
737
- page.coverage.startJSCoverage({
738
- resetOnNavigation: false
739
- }),
740
- page.coverage.startCSSCoverage({
741
- resetOnNavigation: false
742
- })
743
- ]);
744
- }
745
-
746
- await use('autoTestFixture');
747
-
748
- // console.log('autoTestFixture teardown...');
749
- if (isChromium) {
750
- const [jsCoverage, cssCoverage] = await Promise.all([
751
- page.coverage.stopJSCoverage(),
752
- page.coverage.stopCSSCoverage()
753
- ]);
754
- const coverageList = [... jsCoverage, ... cssCoverage];
755
- // console.log(coverageList.map((item) => item.url));
756
- await addCoverageReport(coverageList, test.info());
757
- }
758
-
759
- }, {
760
- scope: 'test',
761
- auto: true
762
- }]
763
- });
764
- export { test, expect };
765
- ```
766
- - Adding server side coverage on global teardown
767
- > For example, a Node.js web server start at the beginning of the test with the env `NODE_V8_COVERAGE=dir`, the V8 coverage data will be saved to `dir` with calling API `v8.takeCoverage()` manually or terminating server gracefully. On global teardown, reading all from `dir` and adding them to global coverage report. For Node.js, the V8 coverage data requires appending source manually.
768
- ```js
769
- // global-teardown.js
770
- import fs from 'fs';
771
- import path from 'path';
772
- import { fileURLToPath } from 'url';
773
- import { addCoverageReport } from 'monocart-reporter';
774
-
775
- export default async (config) => {
776
-
777
- const dir = "your-v8-coverage-data-dir";
778
-
779
- const files = fs.readdirSync(dir);
780
- for (const filename of files) {
781
- const content = fs.readFileSync(path.resolve(dir, filename)).toString('utf-8');
782
- const json = JSON.parse(content);
783
- let coverageList = json.result;
784
- coverageList = coverageList.filter((entry) => entry.url && entry.url.startsWith('file:'));
785
-
786
- // appending source
787
- coverageList.forEach((entry) => {
788
- entry.source = fs.readFileSync(fileURLToPath(entry.url)).toString('utf8');
789
- });
790
-
791
- // there is no test info on teardown, just mock one with required config
792
- const mockTestInfo = {
793
- config
794
- };
795
- await addCoverageReport(coverageList, mockTestInfo);
796
- }
797
- }
798
- ```
799
- see [Collecting V8 Coverage Data from Node.js](https://github.com/cenfun/monocart-coverage-reports?#collecting-v8-coverage-data-from-nodejs)
800
-
801
- ### Coverage Options
802
- - Default [options](https://github.com/cenfun/monocart-coverage-reports?#options)
803
- - [Available Reports](https://github.com/cenfun/monocart-coverage-reports?#available-reports)
804
- - [Using entryFilter and sourceFilter to filter the results for V8 report](https://github.com/cenfun/monocart-coverage-reports?#using-entryfilter-and-sourcefilter-to-filter-the-results-for-v8-report)
805
- - Checking thresholds with [onEnd Hook](https://github.com/cenfun/monocart-coverage-reports?#onend-hook)
806
- - More details see [monocart-coverage-reports](https://github.com/cenfun/monocart-coverage-reports)
807
-
808
- ### Coverage Examples
809
- - For Playwright component testing:
810
- - [playwright-ct-vue](https://github.com/cenfun/playwright-ct-vue)
811
- - [playwright-ct-react](https://github.com/cenfun/playwright-ct-react)
812
- - [playwright-ct-svelte](https://github.com/cenfun/playwright-ct-svelte)
813
- - [nextjs-with-playwright](https://github.com/cenfun/nextjs-with-playwright)
814
- - [code-coverage-with-monocart-reporter](https://github.com/edumserrano/playwright-adventures/blob/main/demos/code-coverage-with-monocart-reporter/)
815
-
816
- ## Attach Lighthouse Audit Report
817
- Attach an audit report with API `attachAuditReport(runnerResult, testInfo)`. Arguments:
818
- - `runnerResult` lighthouse result. see [lighthouse](https://github.com/GoogleChrome/lighthouse/tree/main/docs)
819
- - `testInfo` see [TestInfo](https://playwright.dev/docs/api/class-testinfo)
820
- ```js
821
- const { test, chromium } = require('@playwright/test');
822
- const { attachAuditReport } = require('monocart-reporter');
823
- const lighthouse = require('lighthouse/core/index.cjs');
824
- test('attach lighthouse audit report', async () => {
825
- const port = 9222;
826
- const browser = await chromium.launch({
827
- args: [`--remote-debugging-port=${port}`]
828
- });
829
- const options = {
830
- // logLevel: 'info',
831
- // onlyCategories: ['performance', 'best-practices', 'seo'],
832
- output: 'html',
833
- port
834
- };
835
- const url = 'https://github.com/cenfun/monocart-reporter';
836
- const runnerResult = await lighthouse(url, options);
837
- await browser.close();
838
- await attachAuditReport(runnerResult, test.info());
839
- });
840
-
841
- ```
842
-
843
- ## Attach Network Report
844
- Attach a network report with API `attachNetworkReport(har, testInfo)`. Arguments:
845
- - `har` HAR path (String) or HAR file buffer (Buffer). see [HAR 1.2 Spec](http://www.softwareishard.com/blog/har-12-spec/)
846
- - `testInfo` see [TestInfo](https://playwright.dev/docs/api/class-testinfo)
847
-
848
- Generate HAR with `recordHar` option in browser.newContext() (see example: [report-network.spec.js](https://github.com/cenfun/monocart-reporter/blob/main/tests/report-network/report-network.spec.js) preview [report](https://cenfun.github.io/monocart-reporter/network-1a18723ee59b36867898/index.html))
849
-
850
- ```js
851
- const fs = require('fs');
852
- const path = require('path');
853
- const { test } = require('@playwright/test');
854
- const { attachNetworkReport } = require('monocart-reporter');
855
- let context;
856
- test.describe('attach network report 1', () => {
857
-
858
- const harPath = path.resolve('.temp/network-report1.har');
859
- if (fs.existsSync(harPath)) {
860
- // remove previous
861
- fs.rmSync(harPath);
862
- }
863
-
864
- test('first, open page', async ({ browser }) => {
865
- context = await browser.newContext({
866
- recordHar: {
867
- path: harPath
868
- }
869
- });
870
- const page = await context.newPage();
871
- await page.goto('https://github.com/cenfun/monocart-reporter');
872
- });
873
-
874
- test('next, run test cases', async () => {
875
-
876
- });
877
-
878
- test('finally, attach HAR', async () => {
879
- // Close context to ensure HAR is saved to disk.
880
- await context.close();
881
- await attachNetworkReport(harPath, test.info());
882
- });
883
- });
884
- ```
885
- Generate HAR with [playwright-har](https://github.com/janzaremski/playwright-har)
886
- ```js
887
- import { test } from '@playwright/test';
888
- import { attachNetworkReport } from 'monocart-reporter';
889
- import { PlaywrightHar } from 'playwright-har';
890
-
891
- const harPath = path.resolve('.temp/network-report2.har');
892
- if (fs.existsSync(harPath)) {
893
- // remove previous
894
- fs.rmSync(harPath);
895
- }
896
-
897
- test('first, open page', async ({ browser }) => {
898
- const context = await browser.newContext();
899
- const page = await context.newPage();
900
- playwrightHar = new PlaywrightHar(page);
901
- await playwrightHar.start();
902
- await page.goto('https://github.com/cenfun/monocart-reporter');
903
- });
904
-
905
- test('next, run test cases', async () => {
906
-
907
- });
908
-
909
- test('finally, attach HAR', async () => {
910
- await playwrightHar.stop(harPath);
911
- await attachNetworkReport(harPath, test.info());
912
- });
913
- ```
914
- Preview [Network HTML Report](https://cenfun.github.io/monocart-reporter/network-da7f5b4cceb1e6280782/index.html)
915
-
916
- ## Global State Management
917
- When tests are executed in [isolation](https://playwright.dev/docs/browser-contexts) mode, the reporter and each test may run in a different process, they cannot share data with each other. we can start a local WebSocket server to serve the global data, and read/write the global data with `useState` API from a test.
918
- ### Setup Global State
919
- ```js
920
- module.exports = {
921
- reporter: [
922
- ['list'],
923
- ['monocart-reporter', {
924
- name: "My Test Report",
925
- outputFile: './monocart-report/index.html',
926
- state: {
927
- data: {
928
- count: 0
929
- },
930
- server: {
931
- // port: 8130
932
- },
933
- onClose: (data, config) => {
934
- // save state data to global metadata
935
- Object.assign(config.metadata, data);
936
- }
937
- }
938
- }]
939
- ]
940
- };
941
- ```
942
- ### Get, Set, and Remove Global Data
943
- ```js
944
- const { test } = require('@playwright/test');
945
- const { useState } = require('monocart-reporter');
946
- test('state test', async ({ browserName }) => {
947
- const state = useState({
948
- // port: 8130
949
- });
950
-
951
- const count = await state.get('count');
952
- console.log('count', count);
953
-
954
- await state.set('count', count + 1);
955
-
956
- await state.set({
957
- browser: browserName,
958
- someKey: 'some value'
959
- });
960
-
961
- const [browser, someKey] = await state.get('browser', 'someKey');
962
- console.log(browser, someKey);
963
-
964
- await state.remove('someKey');
965
-
966
- const all = await state.get();
967
- console.log(all);
968
- });
969
- ```
970
- ### Send and Receive Messages between Processes
971
- - send message and receive response from a test (child process)
972
- ```js
973
- const { test } = require('@playwright/test');
974
- const { useState } = require('monocart-reporter');
975
- test('state test send message', async () => {
976
- const state = useState({
977
- // port: 8130
978
- });
979
- const response = await state.send({
980
- testId: test.info().testId,
981
- data: 'my test data'
982
- });
983
- console.log('receive response on client', response);
984
- });
985
- ```
986
- - receive message and send response from global state (main process)
987
- ```js
988
- module.exports = {
989
- reporter: [
990
- ['list'],
991
- ['monocart-reporter', {
992
- name: "My Test Report",
993
- outputFile: './monocart-report/index.html',
994
- state: {
995
- onReceive: function(message) {
996
- const test = this.getTest(message.testId);
997
- if (test) {
998
- // current test
999
- }
1000
- console.log('receive message on server', message);
1001
- return {
1002
- data: 'my response data'
1003
- };
1004
- }
1005
- }
1006
- }]
1007
- ]
1008
- };
1009
- ```
1010
- see example: [Allow specified test cases to run in sequence mode with lock/unlock state](https://github.com/cenfun/monocart-reporter-examples/tree/main/tests/global-state)
1011
-
1012
- ## Merge Shard Reports
1013
- There will be multiple reports to be generated if Playwright test executes in sharding mode. for example:
1014
- ```sh
1015
- npx playwright test --shard=1/3
1016
- npx playwright test --shard=2/3
1017
- npx playwright test --shard=3/3
1018
- ```
1019
- There are 3 reports will be generated.
1020
-
1021
- ### Using `merge` API to merge all reports into one
1022
- > Note: One more suite level "shard" will be added, its title will be the machine hostname, and the summary will be restated. All attachments will be copied to the merged output directory.
1023
- ```js
1024
- import { merge } from 'monocart-reporter';
1025
-
1026
- // json file path
1027
- const reportDataList = [
1028
- 'path-to/shard1/index.json',
1029
- 'path-to/shard2/index.json',
1030
- 'path-to/shard3/index.json'
1031
-
1032
- // Or load zip file directly if the output files is zipped
1033
- // 'path-to/shard1/index.zip',
1034
- // 'path-to/shard2/index.zip',
1035
- // 'path-to/shard3/index.zip'
1036
- ];
1037
-
1038
- await merge(reportDataList, {
1039
- name: 'My Merged Report',
1040
- outputFile: 'merged-report/index.html',
1041
- onEnd: async (reportData, helper) => {
1042
- // send email or third party integration
1043
- }
1044
- });
1045
- ```
1046
-
1047
- > Note: The coverage reports will be merged automatically if we specify the `raw` report in coverage options:
1048
- ```js
1049
- // global coverage options
1050
- coverage: {
1051
- reports: [
1052
- // for merging coverage reports
1053
- 'raw'
1054
- // we can merge and zip the raw report files
1055
- // ['raw', { merge: true, zip: true }]
1056
- ]
1057
- }
1058
- ```
1059
- see example [merge.js](https://github.com/cenfun/monocart-reporter-examples/blob/main/scripts/merge.js)
1060
-
1061
- ### Using `merge` CLI
1062
- ```sh
1063
- # NOTE: The asterisk(*) is a special character which is interpreted by some operating systems (Mac and Linux), please put it in quotes
1064
- npx monocart merge '<glob-patterns>'
1065
-
1066
- # -o --outputFile
1067
- npx monocart merge 'path-to/shard*/index.json' -o merged-reports/index.html
1068
-
1069
- # -c --config
1070
- npx monocart merge 'path-to/shard*/my-report.zip' -c mr.config.js
1071
- ```
1072
- The default config files (In order of priority)
1073
- - mr.config.js
1074
- - mr.config.cjs
1075
- - mr.config.mjs
1076
- - mr.config.json
1077
- - mr.config.ts
1078
-
1079
- Preload for TypeScript config file:
1080
- - It requires node 18.19.0+
1081
- - Installing tsx: `npm i -D tsx`
1082
- - Using the `--import tsx` flag
1083
- - see [comment](https://github.com/cenfun/monocart-reporter/issues/145#issuecomment-2365460013)
1084
-
1085
-
1086
- ## onEnd Hook
1087
- The `onEnd` function will be executed after report generated. Arguments:
1088
- - `reportData` all report data, properties:
1089
- - `name` (String) report name
1090
- - `date` (Number) start date in milliseconds
1091
- - `dateH` (String) human-readable date with `Date.toLocaleString()`
1092
- - `duration` (Number) test duration in milliseconds
1093
- - `durationH` (String) human-readable duration
1094
- - `summary` (Object) test summary, includes `tests`, `suites`, `steps`, etc.
1095
- - `rows` and `columns` (Array) all rows and columns data, both are tree structure, see [detail](https://cenfun.github.io/turbogrid/api.html#data)
1096
- - `tags` (Object) tag collection
1097
- - `metadata` (Object) metadata collection
1098
- - `system` (Object) system information
1099
- - `trends` (Array) historical trend data
1100
- - `caseTypes` and `suiteTypes` (Array)
1101
- - `cwd`, `outputDir` and `outputFile` (String)
1102
- - `htmlPath`, `jsonPath` and `summaryTable` (String)
1103
- - ...
1104
- - `helper` APIs:
1105
- - `helper.find(callback)` Find item like array `find` function.
1106
- - `helper.filter(callback)` Filter list like array `filter` function.
1107
- - `helper.forEach(callback)` Iterate all rows of data (suites/cases/steps), return `break` will break the iteration.
1108
- - `helper.sendEmail(emailOptions)`
1109
-
1110
- ```js
1111
- // playwright.config.js
1112
- module.exports = {
1113
- reporter: [
1114
- ['monocart-reporter', {
1115
- name: "My Test Report",
1116
- outputFile: './monocart-report/index.html',
1117
- // async hook after report data generated
1118
- onEnd: async (reportData, helper) => {
1119
- // console.log(reportData.summary);
1120
-
1121
- // find a test by title
1122
- const myCase = helper.find((item, parent) => item.type === 'case' && item.title.includes('inline tag'));
1123
- console.log(myCase && myCase.title);
1124
-
1125
- // find a suite by title
1126
- const mySuite = helper.find((item, parent) => item.type === 'suite' && item.title.includes('new syntax'));
1127
- console.log(mySuite && mySuite.title);
1128
-
1129
- // filter failed cases
1130
- const failedCases = helper.filter((item, parent) => item.type === 'case' && item.caseType === 'failed');
1131
- console.log(failedCases.map((it) => it.title));
1132
-
1133
- // Iterate all items
1134
- helper.forEach((item, parent) => {
1135
- // do something
1136
- });
1137
-
1138
-
1139
- }
1140
- }]
1141
- ]
1142
- };
1143
- ```
1144
-
1145
- ## Integration Examples
1146
- By using the `onEnd` hook, we can integrate Playwright report with any other tools, such as:
1147
- - [Email](https://github.com/cenfun/playwright-reporter-integrations/tree/main/send-email)
1148
- - [Testrail](https://github.com/cenfun/playwright-reporter-integrations/blob/main/testrail)
1149
- - [Qase](https://github.com/cenfun/playwright-reporter-integrations/tree/main/qase)
1150
- - [Jira + Zephyr](https://github.com/cenfun/playwright-reporter-integrations/tree/main/zephyr-scale)
1151
- - [Jira + Xray](https://github.com/cenfun/playwright-reporter-integrations/blob/main/xray)
1152
- - [Slack](https://github.com/cenfun/playwright-reporter-integrations/tree/main/slack-webhook)
1153
- - [Discord](https://github.com/cenfun/playwright-reporter-integrations/blob/main/discord-webhook)
1154
- - [Teams](https://github.com/cenfun/playwright-reporter-integrations/blob/main/teams-webhook)
1155
- - [BrowserStack](https://github.com/cenfun/playwright-reporter-integrations/blob/main/browserstack)
1156
- - [Github Actions Summary](https://github.com/cenfun/playwright-reporter-integrations/blob/main/github-actions-summary)
1157
- - [Dingtalk](https://github.com/cenfun/playwright-reporter-integrations/tree/main/dingtalk-webhook)/[Weixin](https://github.com/cenfun/playwright-reporter-integrations/tree/main/weixin-webhook)/[Feishu](https://github.com/cenfun/playwright-reporter-integrations/tree/main/feishu-webhook)
1158
-
1159
- See [playwright-reporter-integrations](https://github.com/cenfun/playwright-reporter-integrations)
1160
-
1161
- ## Contributing
1162
- ```sh
1163
- # Node.js 20+
1164
- npm install starfall-cli -g
1165
- npm install
1166
-
1167
- npm run build
1168
- npm run test
1169
-
1170
- npm run dev
1171
- ```
1172
- ### Dependencies
1173
- - UI Framework [Vue 3](https://github.com/vuejs/core)
1174
- - Lightweight UI Components [vine-ui](https://github.com/cenfun/vine-ui)
1175
- - High Performance Grid [turbogrid](https://github.com/cenfun/turbogrid)
1176
- - String compress/decompress [lz-utils](https://github.com/cenfun/lz-utils)
1177
- - Coverage Reporter [monocart-coverage-reports](https://github.com/cenfun/monocart-coverage-reports)
1
+ # Monocart Reporter
2
+
3
+ [![](https://img.shields.io/npm/v/monocart-reporter)](https://www.npmjs.com/package/monocart-reporter)
4
+ [![](https://devimg.vercel.app/npm/downloads/monocart-reporter?label={total}%20downloads/month)](https://www.npmjs.com/package/monocart-reporter)
5
+ ![](https://img.shields.io/librariesio/github/cenfun/monocart-reporter)
6
+ ![](https://img.shields.io/github/license/cenfun/monocart-reporter)
7
+ ![](https://img.shields.io/github/actions/workflow/status/cenfun/monocart-reporter/static.yml)
8
+
9
+
10
+ * A [Playwright](https://github.com/microsoft/playwright) Test [Reporter](https://playwright.dev/docs/test-reporters) (Node.js)
11
+ - A `Tree Grid` style test reporter
12
+ - Support processing big data with high performance
13
+ - Design for customization and extensibility
14
+ - Interactive report with grouping and ultra-fast filter
15
+ - Timeline Workers Graph
16
+ - Monitor CPU and Memory Usage
17
+ - Export Data (json)
18
+ * [Preview](#preview)
19
+ * [Install](#install)
20
+ * [Playwright Config](#playwright-config)
21
+ * [Examples](#examples)
22
+ * [Output](#output)
23
+ * [Reporter Options](#reporter-options)
24
+ * [View Trace Online](#view-trace-online)
25
+ * [Custom Fields Report](#custom-fields-report)
26
+ * [Custom Columns](#custom-columns)
27
+ - [Column Formatter](#column-formatter)
28
+ - [Searchable Fields](#searchable-fields)
29
+ * [Custom Fields in Comments](#custom-fields-in-comments)
30
+ - [Create Diagrams and Visualizations with Mermaid](#create-diagrams-and-visualizations-with-mermaid)
31
+ * [Custom Fields with `setMetadata()`](#custom-fields-with-setmetadata)
32
+ * [Custom Data Visitor](#custom-data-visitor)
33
+ - [Collect Data from the Title](#collect-data-from-the-title)
34
+ - [Collect Data from the Annotations](#collect-data-from-the-annotations)
35
+ - [Remove Secrets and Sensitive Data](#remove-secrets-and-sensitive-data)
36
+ * [Style Tags](#style-tags)
37
+ * [Metadata](#metadata)
38
+ * [Trend Chart](#trend-chart)
39
+ * [Code Coverage Report](#code-coverage-report)
40
+ - [Global Coverage Report](#global-coverage-report)
41
+ - [Coverage Options](#coverage-options)
42
+ - [Coverage Examples](#coverage-examples)
43
+ * [Attach Lighthouse Audit Report](#attach-lighthouse-audit-report)
44
+ * [Attach Network Report](#attach-network-report)
45
+ * [Global State Management](#global-state-management)
46
+ - [Setup Global State](#setup-global-state)
47
+ - [Get, Set, and Remove Global Data](#get-set-and-remove-global-data)
48
+ - [Send and Receive Messages between Processes](#send-and-receive-messages-between-processes)
49
+ * [Merge Shard Reports](#merge-shard-reports)
50
+ * [onEnd Hook](#onend-hook)
51
+ * [Integration Examples](#integration-examples)
52
+ * [Contributing](#contributing)
53
+ * [Changelog](CHANGELOG.md)
54
+
55
+
56
+ ## Preview
57
+ [https://cenfun.github.io/monocart-reporter](https://cenfun.github.io/monocart-reporter)
58
+
59
+ ![](/docs/report.gif)
60
+
61
+ ![](/docs/cli.png)
62
+ (For Github actions, we can enforce color with env: `FORCE_COLOR: true`)
63
+
64
+ ## Install
65
+ ```sh
66
+ npm i -D monocart-reporter
67
+ ```
68
+
69
+ ## Playwright Config
70
+ > Note: Most examples use `CommonJS` by default, please [move to ESM](https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c#how-can-i-move-my-commonjs-project-to-esm) according to your needs.
71
+ ```js
72
+ // playwright.config.js
73
+ module.exports = {
74
+ reporter: [
75
+ ['list'],
76
+ ['monocart-reporter', {
77
+ name: "My Test Report",
78
+ outputFile: './monocart-report/index.html'
79
+ }]
80
+ ]
81
+ };
82
+ ```
83
+ Playwright Docs [https://playwright.dev/docs/test-reporters](https://playwright.dev/docs/test-reporters)
84
+
85
+ ## Examples
86
+ - [tests](/tests/)
87
+ - [monocart-reporter-examples](https://github.com/cenfun/monocart-reporter-examples)
88
+ - [playwright-reporter-integrations](https://github.com/cenfun/playwright-reporter-integrations)
89
+
90
+ ## Output
91
+ - path-to/your-filename.html
92
+ Single HTML file (data compressed), easy to transfer/deploy or open directly anywhere
93
+ > Note: All attachments (screenshots images/videos) will be linked with relative path in report.
94
+ - path-to/your-filename.json (requires option `json` is true)
95
+ Separated data file which can be used for debugging or data provider (It's included in the above HTML and compressed).
96
+ - path-to/your-filename.zip (requires option `zip` is true)
97
+ Zip file for merging reports
98
+
99
+ ## Reporter Options
100
+ - Default options: [lib/default/options.js](./lib/default/options.js)
101
+ - Options declaration see `MonocartReporterOptions` [lib/index.d.ts](./lib/index.d.ts)
102
+
103
+ ## View Trace Online
104
+ > The [Trace Viewer](https://trace.playwright.dev/) requires that the trace file must be loaded over the http:// or https:// protocols without [CORS](https://developer.mozilla.org/en-US/docs/Glossary/CORS) issue.
105
+ - Start a local web server with following CLI:
106
+ ```sh
107
+ # serve and open report
108
+ npx monocart show-report <path-to-report>
109
+
110
+ # serve report
111
+ npx monocart serve-report <path-to-report>
112
+ ```
113
+ The server add the http header `Access-Control-Allow-Origin: *` to [allow requesting from any origin](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin), it works with `http://localhost:port/` or `http://127.0.0.1:port/`
114
+ - To successfully work with other `IP` or `domain`, you can start web server with `https`:
115
+ ```sh
116
+ npx monocart show-report <path-to-report> --ssl <path-to-key,path-to-cert>
117
+ ```
118
+ For example: `npx monocart show-report monocart-report/index.html --ssl ssl/key.pem,ssl/cert.pem`
119
+
120
+ You can create and install local CA with [mkcert](https://mkcert.dev)
121
+ - Using your own trace viewer url with option `traceViewerUrl`:
122
+ ```js
123
+ // reporter options
124
+ {
125
+ name: "My Test Report",
126
+ outputFile: './monocart-report/index.html',
127
+ // defaults to 'https://trace.playwright.dev/?trace={traceUrl}'
128
+ traceViewerUrl: 'https://your-own-trace-viewer-url/?trace={traceUrl}'
129
+ }
130
+ ```
131
+
132
+ ## Custom Fields Report
133
+ You can add custom fields to the report. for example: Owner, JIRA Key etc.
134
+ - First, you need to add [Custom Columns](#custom-columns) for the fields.
135
+ - Then, collect data for these fields
136
+ - [Custom Fields in Comments](#custom-fields-in-comments)
137
+ - [Custom Fields with `setMetadata()`](#custom-fields-with-setmetadata)
138
+ - [Custom Data Visitor](#custom-data-visitor)
139
+
140
+ ### Custom Columns
141
+ The report will be displayed in a `Tree Grid`. The `columns` function is used to customize the grid columns. The column properties following:
142
+ - `id` (String) Column id (required)
143
+ - `name` (String) Column name, shows in grid header
144
+ - `align` (String) left (default), center, right
145
+ - `width` (Number) Column width
146
+ - `minWidth`, `maxWidth` (Number) Default to 81 ~ 300
147
+ - `styleMap` (Object, String) Column style (css)
148
+ - `formatter` (Function) [column formatter](#custom-formatter). Arguments: value, rowItem, columnItem, cellNode
149
+ - `sortable` (Boolean) Column sortable when click column header name
150
+ - `resizable` (Boolean) Column width resizable
151
+ - `searchable` (Boolean) Specifies whether the column is searchable
152
+ - `markdown` (Boolean) Specifies whether the column needs to use markdown conversion
153
+ - `detailed` (Boolean) Specifies whether the column needs to display the layout in detail (horizontal)
154
+ - more properties [columnProps](https://cenfun.github.io/turbogrid/api.html#options.columnProps)
155
+ ```js
156
+ // playwright.config.js
157
+ module.exports = {
158
+ reporter: [
159
+ ['monocart-reporter', {
160
+ name: "My Test Report",
161
+ outputFile: './monocart-report/index.html',
162
+
163
+ // custom columns
164
+ columns: (defaultColumns) => {
165
+
166
+ // insert custom column(s) before a default column
167
+ const index = defaultColumns.findIndex((column) => column.id === 'duration');
168
+ defaultColumns.splice(index, 0, {
169
+ // define the column in reporter
170
+ id: 'owner',
171
+ name: 'Owner',
172
+ align: 'center',
173
+ searchable: true,
174
+ styleMap: {
175
+ 'font-weight': 'normal'
176
+ }
177
+ }, {
178
+ // another column for JIRA link
179
+ id: 'jira',
180
+ name: 'JIRA Key',
181
+ width: 100,
182
+ searchable: true,
183
+ styleMap: 'font-weight:normal;',
184
+ formatter: (v, rowItem, columnItem) => {
185
+ const key = rowItem[columnItem.id];
186
+ return `<a href="https://your-jira-url/${key}" target="_blank">${v}</a>`;
187
+ }
188
+ });
189
+
190
+ }
191
+ }]
192
+ ]
193
+ };
194
+ ```
195
+ #### Column Formatter
196
+ > Note: The `formatter` function will be serialized into string via JSON, so closures, contexts, etc. will not work!
197
+ ```js
198
+ // playwright.config.js
199
+ module.exports = {
200
+ reporter: [
201
+ ['monocart-reporter', {
202
+ name: "My Test Report",
203
+ outputFile: './monocart-report/index.html',
204
+ columns: (defaultColumns) => {
205
+
206
+ // duration formatter
207
+ const durationColumn = defaultColumns.find((column) => column.id === 'duration');
208
+ durationColumn.formatter = function(value, rowItem, columnItem) {
209
+ if (typeof value === 'number' && value) {
210
+ return `<i>${value.toLocaleString()} ms</i>`;
211
+ }
212
+ return value;
213
+ };
214
+
215
+ // title formatter
216
+ // Note: The title shows the tree style, it is a complicated HTML structure
217
+ // it is recommended to format title base on previous.
218
+ const titleColumn = defaultColumns.find((column) => column.id === 'title');
219
+ titleColumn.formatter = function(value, rowItem, columnItem, cellNode) {
220
+ const perviousFormatter = this.getFormatter('tree');
221
+ const v = perviousFormatter(value, rowItem, columnItem, cellNode);
222
+ if (rowItem.type === 'step') {
223
+ return `${v}<div style="position:absolute;top:0;right:5px;">✅</div>`;
224
+ }
225
+ return v;
226
+ };
227
+
228
+ }
229
+ }]
230
+ ]
231
+ };
232
+ ```
233
+
234
+ #### Searchable Fields
235
+ ```js
236
+ // playwright.config.js
237
+ module.exports = {
238
+ reporter: [
239
+ ['monocart-reporter', {
240
+ name: "My Test Report",
241
+ outputFile: './monocart-report/index.html',
242
+ columns: (defaultColumns) => {
243
+ const locationColumn = defaultColumns.find((column) => column.id === 'location');
244
+ locationColumn.searchable = true;
245
+ }
246
+ }]
247
+ ]
248
+ };
249
+ ```
250
+
251
+ ### Custom Fields in Comments
252
+ > The code comments are good enough to provide extra information without breaking existing code, and no dependencies, clean, easy to read, etc.
253
+ - First, enable option `customFieldsInComments` to `true`
254
+ ```js
255
+ // playwright.config.js
256
+ module.exports = {
257
+ reporter: [
258
+ ['monocart-reporter', {
259
+ // enable/disable custom fields in comments. Defaults to true.
260
+ customFieldsInComments: true
261
+ }]
262
+ ]
263
+ };
264
+ ```
265
+
266
+ - Then, add comments for the tests
267
+ > Note: Each comment item must start with `@` which is similar to [JSDoc](https://jsdoc.app/).
268
+
269
+ For example, adding `owner` and `jira` to the cases, steps, and suites. or updating the value if the field exists like `title`
270
+ ```js
271
+ /**
272
+ * for file (comment file in the first line)
273
+ * @owner FO
274
+ */
275
+ const { test, expect } = require('@playwright/test');
276
+
277
+ /**
278
+ * for case
279
+ * @owner Kevin
280
+ * @jira MCR-16888
281
+ */
282
+ test('case title', () => {
283
+
284
+ });
285
+
286
+ /**
287
+ * @description multiple lines text description
288
+ multiple lines text description
289
+ multiple lines text description
290
+ * @jira MCR-16888
291
+ */
292
+ test('case description', () => {
293
+
294
+ });
295
+
296
+ /**
297
+ * for describe suite
298
+ * @owner Mark
299
+ * @jira MCR-16900
300
+ */
301
+ test.describe('suite title', () => {
302
+
303
+ test('case title', ({ browserName }, testInfo) => {
304
+
305
+ /**
306
+ * rewrite assert step title "expect.toBe" to
307
+ * @title my custom assert step title
308
+ * @annotations important
309
+ */
310
+ expect(testInfo).toBe(test.info());
311
+
312
+ // @owner Steve
313
+ await test.step('step title', () => {
314
+
315
+ });
316
+
317
+ });
318
+
319
+ });
320
+
321
+ /**
322
+ * rewrite "beforeAll hook" title to
323
+ * @title do something before all
324
+ */
325
+ test.beforeAll(() => {
326
+
327
+ });
328
+
329
+ /**
330
+ * rewrite "beforeEach hook" title to
331
+ * @title do something before each
332
+ */
333
+ test.beforeEach(() => {
334
+
335
+ });
336
+ ```
337
+
338
+ #### Create Diagrams and Visualizations with [Mermaid](https://mermaid.js.org/)
339
+ - Enable Mermaid
340
+ ```js
341
+ // playwright.config.js
342
+ module.exports = {
343
+ reporter: [
344
+ ['monocart-reporter', {
345
+ name: "My Test Report",
346
+ outputFile: './monocart-report/index.html',
347
+ mermaid: {
348
+ // mermaid script url, using mermaid CDN: https://www.jsdelivr.com/package/npm/mermaid
349
+ scriptSrc: 'https://cdn.jsdelivr.net/npm/mermaid@latest/dist/mermaid.min.js',
350
+ // mermaid config: https://mermaid.js.org/config/schema-docs/config.html
351
+ config: {
352
+ startOnLoad: false
353
+ }
354
+ }
355
+ }]
356
+ ]
357
+ };
358
+ ```
359
+ - Write Mermaid code in markdown:
360
+ ````js
361
+ /**
362
+ * @description Sequence diagram for Monocart Reporter
363
+ ```mermaid
364
+ flowchart LR
365
+
366
+ A[Hard] -->|Text| B(Round)
367
+ B --> C{Decision}
368
+ C -->|One| D[Result 1]
369
+ C -->|Two| E[Result 2]
370
+ ```
371
+ */
372
+ test('case description', () => {
373
+
374
+ });
375
+ ````
376
+ see [Mermaid doc](https://mermaid.js.org/syntax/flowchart.html)
377
+
378
+
379
+ ### Custom Fields with `setMetadata()`
380
+ Using `comments` is only applicable to statically created tests, while using API `setMetadata()` can be applicable to all situations, which is including dynamically created tests.
381
+ ```js
382
+ const { test } = require('@playwright/test');
383
+ const { setMetadata } = require('monocart-reporter');
384
+ test.describe('Data Driven Tests with setMetadata(data, testInfo)', () => {
385
+ const list = [{
386
+ title: 'Example Case 1 Data Driven Test',
387
+ owner: 'Jensen',
388
+ jira: 'MCR-16889',
389
+ }, {
390
+ title: 'Example Case 2 Data Driven Test',
391
+ owner: 'Mark',
392
+ jira: 'MCR-16899'
393
+ }];
394
+ list.forEach((item, i) => {
395
+ test(item.title, () => {
396
+ setMetadata({
397
+ owner: item.owner,
398
+ jira: item.jira
399
+ }, test.info());
400
+
401
+ //expect(1).toBe(1);
402
+
403
+ });
404
+ });
405
+ });
406
+ ```
407
+
408
+ ### Custom Data Visitor
409
+ The `visitor` function will be executed for each row item (suite, case and step). Arguments:
410
+ - `data` data item (suite/case/step) for reporter, you can rewrite some of its properties or add more
411
+ - `metadata` original data object from Playwright test, could be one of [Suite](https://playwright.dev/docs/api/class-suite), [TestCase](https://playwright.dev/docs/api/class-testcase) or [TestStep](https://playwright.dev/docs/api/class-teststep)
412
+
413
+ #### Collect Data from the Title
414
+ For example, we want to parse out the jira key from the title:
415
+ ```js
416
+ test('[MCR-123] collect data from the title', () => {
417
+
418
+ });
419
+ ```
420
+ You can simply use regular expressions to parse and get jira key:
421
+ ```js
422
+ // playwright.config.js
423
+ module.exports = {
424
+ reporter: [
425
+ ['monocart-reporter', {
426
+ name: "My Test Report",
427
+ outputFile: './monocart-report/index.html',
428
+ visitor: (data, metadata) => {
429
+ // [MCR-123] collect data from the title
430
+ const matchResult = metadata.title.match(/\[(.+)\]/);
431
+ if (matchResult && matchResult[1]) {
432
+ data.jira = matchResult[1];
433
+ }
434
+ }
435
+ }]
436
+ ]
437
+ };
438
+ ```
439
+ multiple matches example: [collect-data](https://github.com/cenfun/monocart-reporter-examples/tree/main/tests/collect-data)
440
+
441
+ #### Collect Data from the Annotations
442
+ It should be easier than getting from title. see [custom annotations](https://playwright.dev/docs/test-annotations#custom-annotations) via `test.info().annotations`
443
+ ```js
444
+ test('collect data from the annotations', () => {
445
+ test.info().annotations.push({
446
+ type: "jira",
447
+ description: "MCR-123"
448
+ })
449
+ });
450
+ ```
451
+ ```js
452
+ // playwright.config.js
453
+ module.exports = {
454
+ reporter: [
455
+ ['monocart-reporter', {
456
+ name: "My Test Report",
457
+ outputFile: './monocart-report/index.html',
458
+ visitor: (data, metadata) => {
459
+ // collect data from the annotations
460
+ if (metadata.annotations) {
461
+ const jiraItem = metadata.annotations.find((item) => item.type === 'jira');
462
+ if (jiraItem && jiraItem.description) {
463
+ data.jira = jiraItem.description;
464
+ }
465
+ }
466
+ }
467
+ }]
468
+ ]
469
+ };
470
+ ```
471
+
472
+ #### Remove Secrets and Sensitive Data
473
+ > The report may hosted outside of the organization’s internal boundaries, security becomes a big issue. Any secrets or sensitive data, such as usernames, passwords, tokens and API keys, should be handled with extreme care. The following example is removing the password and token from the report data with the string replacement in `visitor` function.
474
+ ```js
475
+ // playwright.config.js
476
+ module.exports = {
477
+ reporter: [
478
+ ['monocart-reporter', {
479
+ name: "My Test Report",
480
+ outputFile: './monocart-report/index.html',
481
+ visitor: (data, metadata) => {
482
+ const mySecrets = [process.env.PASSWORD, process.env.TOKEN];
483
+ mySecrets.forEach((secret) => {
484
+ // remove from title
485
+ data.title = data.title.replace(secret, '***');
486
+ // remove from logs
487
+ if (data.logs) {
488
+ data.logs = data.logs.map((item) => item.replace(secret, '***'));
489
+ }
490
+ // remove from errors
491
+ if (data.errors) {
492
+ data.errors = data.errors.map((item) => item.replace(secret, '***'));
493
+ }
494
+ });
495
+ }
496
+ }]
497
+ ]
498
+ };
499
+ ```
500
+ see example: [remove-secrets](https://github.com/cenfun/monocart-reporter-examples/tree/main/tests/remove-secrets)
501
+
502
+ ## Style Tags
503
+
504
+ * Add tag to test/describe title ( starts with `@` )
505
+ ```js
506
+ test('test title @smoke @critical', () => { });
507
+ test.describe('describe title @smoke @critical', () => { });
508
+
509
+ // new syntax for tag in playwright v1.42.0
510
+ test('test title', { tag: ['@smoke', '@critical'] }, () => { });
511
+ test.describe('describe title', { tag: ['@smoke', '@critical'] }, () => { });
512
+ ```
513
+
514
+ * Custom tag style
515
+ ```js
516
+ // playwright.config.js
517
+ module.exports = {
518
+ reporter: [
519
+ ['monocart-reporter', {
520
+ name: "My Test Report",
521
+ outputFile: './monocart-report/index.html',
522
+ tags: {
523
+ smoke: {
524
+ style: {
525
+ background: '#6F9913'
526
+ },
527
+ description: 'This is Smoke Test'
528
+ },
529
+ critical: {
530
+ background: '#c00'
531
+ }
532
+ }
533
+ }]
534
+ ]
535
+ };
536
+ ```
537
+
538
+ * Put style tags in new column
539
+ ```js
540
+ module.exports = {
541
+ reporter: [
542
+ ['list'],
543
+ ['monocart-reporter', {
544
+ name: "My Test Report",
545
+ outputFile: './monocart-report/index.html',
546
+ tags: {
547
+ // ...
548
+ },
549
+ columns: (defaultColumns) => {
550
+
551
+ // disable title tags
552
+ defaultColumns.find((column) => column.id === 'title').titleTagsDisabled = true;
553
+
554
+ // add tags column
555
+ const index = defaultColumns.findIndex((column) => column.id === 'type');
556
+ defaultColumns.splice(index, 0, {
557
+ id: 'tags',
558
+ name: 'Tags',
559
+ width: 150,
560
+ formatter: 'tags'
561
+ });
562
+ }
563
+ }]
564
+ ]
565
+ };
566
+ ```
567
+ see [example](https://github.com/cenfun/monocart-reporter-examples/tree/main/tests/tags-column)
568
+
569
+
570
+ ## Metadata
571
+ > All metadata will be listed in the report in a key/value format.
572
+ - Global level `metadata`
573
+ ```js
574
+ // playwright.config.js
575
+ module.exports = {
576
+ globalSetup: require.resolve('./common/global-setup.js'),
577
+ metadata: {
578
+ product: 'Monocart',
579
+ env: 'STG',
580
+ type: 'Regression',
581
+ executor: 'Mono',
582
+
583
+ // test home page object model
584
+ url: 'https://www.npmjs.org/package/monocart-reporter'
585
+ },
586
+ reporter: [
587
+ ['monocart-reporter', {
588
+ name: "My Test Report",
589
+ outputFile: './monocart-report/index.html'
590
+ }]
591
+ ]
592
+ };
593
+ ```
594
+
595
+ - Project level `metadata`
596
+ ```js
597
+ // playwright.config.js
598
+ module.exports = {
599
+ projects: [
600
+ {
601
+ name: 'Desktop Chromium',
602
+ use: {
603
+ browserName: 'chromium'
604
+ },
605
+ metadata: {
606
+ projectData: 'project level metadata',
607
+ owner: 'PO',
608
+ link: 'https://github.com/cenfun/monocart-reporter'
609
+ }
610
+ }
611
+ ]
612
+ }
613
+ ```
614
+
615
+ - Collect metadata in [global setup or teardown](https://playwright.dev/docs/test-global-setup-teardown)
616
+ ```js
617
+ // ./common/global-setup.js
618
+ import { chromium } from '@playwright/test';
619
+ export default async (config) => {
620
+ const metadata = config.metadata;
621
+ // collect data and save to global metadata
622
+ const browser = await chromium.launch();
623
+ const chromiumVersion = await browser.version();
624
+ metadata.chromiumVersion = chromiumVersion;
625
+ };
626
+ ```
627
+
628
+ - Collect metadata in a test
629
+ > Playwright Test runs tests in [parallel](https://playwright.dev/docs/test-parallel) with isolate test data by default, so we need to utilize [global state management](#global-state-management) to collect metadata in a test.
630
+
631
+ - Enable global state, see [Setup Global State](#setup-global-state)
632
+ - Update global state in a test
633
+ ```js
634
+ const { test } = require('@playwright/test');
635
+ const { useState } = require('monocart-reporter');
636
+
637
+ const state = useState({
638
+ // port: 8130
639
+ });
640
+
641
+ test('test metadata url', async ({ page }) => {
642
+ const url = await page.evaluate(() => window.location.href);
643
+ await state.set('url', url);
644
+ });
645
+ ```
646
+
647
+ ## Trend Chart
648
+ > Note: The trend chart requires historical data generally stored in the server database. There is a serverless solution which is connecting and collecting historical trend data from previous report data before test every time.
649
+ - If a report is generated in the same place every time, you can simply connect the data with the report JSON path (the data is not 100% safe if there is any runtime error, the previous output dir will be empty by Playwright but the reporter processing not finish)
650
+ ```js
651
+ // playwright.config.js
652
+ module.exports = {
653
+ reporter: [
654
+ ['monocart-reporter', {
655
+ name: "My Test Report",
656
+ outputFile: './monocart-report/index.html',
657
+ // connect previous report data for trend chart
658
+ trend: './monocart-report/index.json'
659
+ }]
660
+ ]
661
+ };
662
+ ```
663
+ - Recommended: resolve the data by yourself (could be requested from the server), required data fields:
664
+ - `date` (Number) the previous test date in milliseconds
665
+ - `duration` (Number) the previous test duration
666
+ - `summary` (Object) the previous test summary
667
+ - `trends` (Array) historical data list, but except the previous self
668
+ ```js
669
+ // playwright.config.js
670
+ module.exports = {
671
+ reporter: [
672
+ ['monocart-reporter', {
673
+ name: "My Test Report",
674
+ outputFile: './monocart-report/index.html',
675
+ // connect previous report data for trend chart
676
+ trend: async () => {
677
+ const previousReportData = await readDataFromSomeWhere("path-to/report.json");
678
+ // do some data filtering to previous trends
679
+ previousReportData.trends = previousReportData.trends.filter((item) => {
680
+ // remove data a week ago
681
+ return item.date > (Date.now() - 7 * 24 * 60 * 60 * 1000)
682
+ });
683
+ return previousReportData;
684
+ }
685
+ }]
686
+ ]
687
+ };
688
+ ```
689
+
690
+ ## Code Coverage Report
691
+ The reporter integrates [monocart-coverage-reports](https://github.com/cenfun/monocart-coverage-reports) for coverage reports, there are two APIs:
692
+ - `addCoverageReport(data, testInfo)` Add coverage to global coverage report from a test. see [Global Coverage Report](#global-coverage-report)
693
+ - `data` There are two supported data inputs: [Istanbul](https://github.com/cenfun/monocart-coverage-reports?#collecting-istanbul-coverage-data) (Object) or [V8](https://github.com/cenfun/monocart-coverage-reports?#collecting-v8-coverage-data) (Array)
694
+ - `testInfo` see [TestInfo](https://playwright.dev/docs/api/class-testinfo)
695
+ - `attachCoverageReport(data, testInfo, options)` Attach a coverage report to a test. Arguments:
696
+ - `data` same as above
697
+ - `testInfo` same as above
698
+ - `options` (Object) see [Coverage Options](#coverage-options)
699
+
700
+ ![](/docs/v8.gif)
701
+
702
+ ### Global Coverage Report
703
+ The global coverage report will merge all coverages into one global report after all the tests are finished.
704
+ - The global coverage options see [Coverage Options](#coverage-options)
705
+ ```js
706
+ // playwright.config.js
707
+ module.exports = {
708
+ reporter: [
709
+ ['monocart-reporter', {
710
+ name: "My Test Report",
711
+ outputFile: './monocart-report/index.html',
712
+ // global coverage report options
713
+ coverage: {
714
+ entryFilter: (entry) => true,
715
+ sourceFilter: (sourcePath) => sourcePath.search(/src\/.+/) !== -1,
716
+ }
717
+ }]
718
+ ]
719
+ };
720
+ ```
721
+ - Adding coverage data for each test with [automatic fixtures](https://playwright.dev/docs/test-fixtures#automatic-fixtures)
722
+ ```js
723
+ // fixtures.js for v8 coverage
724
+ import { test as testBase, expect } from '@playwright/test';
725
+ import { addCoverageReport } from 'monocart-reporter';
726
+
727
+ const test = testBase.extend({
728
+ autoTestFixture: [async ({ page }, use) => {
729
+
730
+ // NOTE: it depends on your project name
731
+ const isChromium = test.info().project.name === 'Desktop Chromium';
732
+
733
+ // console.log('autoTestFixture setup...');
734
+ // coverage API is chromium only
735
+ if (isChromium) {
736
+ await Promise.all([
737
+ page.coverage.startJSCoverage({
738
+ resetOnNavigation: false
739
+ }),
740
+ page.coverage.startCSSCoverage({
741
+ resetOnNavigation: false
742
+ })
743
+ ]);
744
+ }
745
+
746
+ await use('autoTestFixture');
747
+
748
+ // console.log('autoTestFixture teardown...');
749
+ if (isChromium) {
750
+ const [jsCoverage, cssCoverage] = await Promise.all([
751
+ page.coverage.stopJSCoverage(),
752
+ page.coverage.stopCSSCoverage()
753
+ ]);
754
+ const coverageList = [... jsCoverage, ... cssCoverage];
755
+ // console.log(coverageList.map((item) => item.url));
756
+ await addCoverageReport(coverageList, test.info());
757
+ }
758
+
759
+ }, {
760
+ scope: 'test',
761
+ auto: true
762
+ }]
763
+ });
764
+ export { test, expect };
765
+ ```
766
+ - Adding server side coverage on global teardown
767
+ > For example, a Node.js web server start at the beginning of the test with the env `NODE_V8_COVERAGE=dir`, the V8 coverage data will be saved to `dir` with calling API `v8.takeCoverage()` manually or terminating server gracefully. On global teardown, reading all from `dir` and adding them to global coverage report. For Node.js, the V8 coverage data requires appending source manually.
768
+ ```js
769
+ // global-teardown.js
770
+ import fs from 'fs';
771
+ import path from 'path';
772
+ import { fileURLToPath } from 'url';
773
+ import { addCoverageReport } from 'monocart-reporter';
774
+
775
+ export default async (config) => {
776
+
777
+ const dir = "your-v8-coverage-data-dir";
778
+
779
+ const files = fs.readdirSync(dir);
780
+ for (const filename of files) {
781
+ const content = fs.readFileSync(path.resolve(dir, filename)).toString('utf-8');
782
+ const json = JSON.parse(content);
783
+ let coverageList = json.result;
784
+ coverageList = coverageList.filter((entry) => entry.url && entry.url.startsWith('file:'));
785
+
786
+ // appending source
787
+ coverageList.forEach((entry) => {
788
+ entry.source = fs.readFileSync(fileURLToPath(entry.url)).toString('utf8');
789
+ });
790
+
791
+ // there is no test info on teardown, just mock one with required config
792
+ const mockTestInfo = {
793
+ config
794
+ };
795
+ await addCoverageReport(coverageList, mockTestInfo);
796
+ }
797
+ }
798
+ ```
799
+ see [Collecting V8 Coverage Data from Node.js](https://github.com/cenfun/monocart-coverage-reports?#collecting-v8-coverage-data-from-nodejs)
800
+
801
+ ### Coverage Options
802
+ - Default [options](https://github.com/cenfun/monocart-coverage-reports?#options)
803
+ - [Available Reports](https://github.com/cenfun/monocart-coverage-reports?#available-reports)
804
+ - [Using entryFilter and sourceFilter to filter the results for V8 report](https://github.com/cenfun/monocart-coverage-reports?#using-entryfilter-and-sourcefilter-to-filter-the-results-for-v8-report)
805
+ - Checking thresholds with [onEnd Hook](https://github.com/cenfun/monocart-coverage-reports?#onend-hook)
806
+ - More details see [monocart-coverage-reports](https://github.com/cenfun/monocart-coverage-reports)
807
+
808
+ ### Coverage Examples
809
+ - For Playwright component testing:
810
+ - [playwright-ct-vue](https://github.com/cenfun/playwright-ct-vue)
811
+ - [playwright-ct-react](https://github.com/cenfun/playwright-ct-react)
812
+ - [playwright-ct-svelte](https://github.com/cenfun/playwright-ct-svelte)
813
+ - [nextjs-with-playwright](https://github.com/cenfun/nextjs-with-playwright)
814
+ - [code-coverage-with-monocart-reporter](https://github.com/edumserrano/playwright-adventures/blob/main/demos/code-coverage-with-monocart-reporter/)
815
+
816
+ ## Attach Lighthouse Audit Report
817
+ Attach an audit report with API `attachAuditReport(runnerResult, testInfo)`. Arguments:
818
+ - `runnerResult` lighthouse result. see [lighthouse](https://github.com/GoogleChrome/lighthouse/tree/main/docs)
819
+ - `testInfo` see [TestInfo](https://playwright.dev/docs/api/class-testinfo)
820
+ ```js
821
+ const { test, chromium } = require('@playwright/test');
822
+ const { attachAuditReport } = require('monocart-reporter');
823
+ const lighthouse = require('lighthouse/core/index.cjs');
824
+ test('attach lighthouse audit report', async () => {
825
+ const port = 9222;
826
+ const browser = await chromium.launch({
827
+ args: [`--remote-debugging-port=${port}`]
828
+ });
829
+ const options = {
830
+ // logLevel: 'info',
831
+ // onlyCategories: ['performance', 'best-practices', 'seo'],
832
+ output: 'html',
833
+ port
834
+ };
835
+ const url = 'https://github.com/cenfun/monocart-reporter';
836
+ const runnerResult = await lighthouse(url, options);
837
+ await browser.close();
838
+ await attachAuditReport(runnerResult, test.info());
839
+ });
840
+
841
+ ```
842
+
843
+ ## Attach Network Report
844
+ Attach a network report with API `attachNetworkReport(har, testInfo)`. Arguments:
845
+ - `har` HAR path (String) or HAR file buffer (Buffer). see [HAR 1.2 Spec](http://www.softwareishard.com/blog/har-12-spec/)
846
+ - `testInfo` see [TestInfo](https://playwright.dev/docs/api/class-testinfo)
847
+
848
+ Generate HAR with `recordHar` option in browser.newContext() (see example: [report-network.spec.js](https://github.com/cenfun/monocart-reporter/blob/main/tests/report-network/report-network.spec.js) preview [report](https://cenfun.github.io/monocart-reporter/network-1a18723ee59b36867898/index.html))
849
+
850
+ ```js
851
+ const fs = require('fs');
852
+ const path = require('path');
853
+ const { test } = require('@playwright/test');
854
+ const { attachNetworkReport } = require('monocart-reporter');
855
+ let context;
856
+ test.describe('attach network report 1', () => {
857
+
858
+ const harPath = path.resolve('.temp/network-report1.har');
859
+ if (fs.existsSync(harPath)) {
860
+ // remove previous
861
+ fs.rmSync(harPath);
862
+ }
863
+
864
+ test('first, open page', async ({ browser }) => {
865
+ context = await browser.newContext({
866
+ recordHar: {
867
+ path: harPath
868
+ }
869
+ });
870
+ const page = await context.newPage();
871
+ await page.goto('https://github.com/cenfun/monocart-reporter');
872
+ });
873
+
874
+ test('next, run test cases', async () => {
875
+
876
+ });
877
+
878
+ test('finally, attach HAR', async () => {
879
+ // Close context to ensure HAR is saved to disk.
880
+ await context.close();
881
+ await attachNetworkReport(harPath, test.info());
882
+ });
883
+ });
884
+ ```
885
+ Generate HAR with [playwright-har](https://github.com/janzaremski/playwright-har)
886
+ ```js
887
+ import { test } from '@playwright/test';
888
+ import { attachNetworkReport } from 'monocart-reporter';
889
+ import { PlaywrightHar } from 'playwright-har';
890
+
891
+ const harPath = path.resolve('.temp/network-report2.har');
892
+ if (fs.existsSync(harPath)) {
893
+ // remove previous
894
+ fs.rmSync(harPath);
895
+ }
896
+
897
+ test('first, open page', async ({ browser }) => {
898
+ const context = await browser.newContext();
899
+ const page = await context.newPage();
900
+ playwrightHar = new PlaywrightHar(page);
901
+ await playwrightHar.start();
902
+ await page.goto('https://github.com/cenfun/monocart-reporter');
903
+ });
904
+
905
+ test('next, run test cases', async () => {
906
+
907
+ });
908
+
909
+ test('finally, attach HAR', async () => {
910
+ await playwrightHar.stop(harPath);
911
+ await attachNetworkReport(harPath, test.info());
912
+ });
913
+ ```
914
+ Preview [Network HTML Report](https://cenfun.github.io/monocart-reporter/network-da7f5b4cceb1e6280782/index.html)
915
+
916
+ ## Global State Management
917
+ When tests are executed in [isolation](https://playwright.dev/docs/browser-contexts) mode, the reporter and each test may run in a different process, they cannot share data with each other. we can start a local WebSocket server to serve the global data, and read/write the global data with `useState` API from a test.
918
+ ### Setup Global State
919
+ ```js
920
+ module.exports = {
921
+ reporter: [
922
+ ['list'],
923
+ ['monocart-reporter', {
924
+ name: "My Test Report",
925
+ outputFile: './monocart-report/index.html',
926
+ state: {
927
+ data: {
928
+ count: 0
929
+ },
930
+ server: {
931
+ // port: 8130
932
+ },
933
+ onClose: (data, config) => {
934
+ // save state data to global metadata
935
+ Object.assign(config.metadata, data);
936
+ }
937
+ }
938
+ }]
939
+ ]
940
+ };
941
+ ```
942
+ ### Get, Set, and Remove Global Data
943
+ ```js
944
+ const { test } = require('@playwright/test');
945
+ const { useState } = require('monocart-reporter');
946
+ test('state test', async ({ browserName }) => {
947
+ const state = useState({
948
+ // port: 8130
949
+ });
950
+
951
+ const count = await state.get('count');
952
+ console.log('count', count);
953
+
954
+ await state.set('count', count + 1);
955
+
956
+ await state.set({
957
+ browser: browserName,
958
+ someKey: 'some value'
959
+ });
960
+
961
+ const [browser, someKey] = await state.get('browser', 'someKey');
962
+ console.log(browser, someKey);
963
+
964
+ await state.remove('someKey');
965
+
966
+ const all = await state.get();
967
+ console.log(all);
968
+ });
969
+ ```
970
+ ### Send and Receive Messages between Processes
971
+ - send message and receive response from a test (child process)
972
+ ```js
973
+ const { test } = require('@playwright/test');
974
+ const { useState } = require('monocart-reporter');
975
+ test('state test send message', async () => {
976
+ const state = useState({
977
+ // port: 8130
978
+ });
979
+ const response = await state.send({
980
+ testId: test.info().testId,
981
+ data: 'my test data'
982
+ });
983
+ console.log('receive response on client', response);
984
+ });
985
+ ```
986
+ - receive message and send response from global state (main process)
987
+ ```js
988
+ module.exports = {
989
+ reporter: [
990
+ ['list'],
991
+ ['monocart-reporter', {
992
+ name: "My Test Report",
993
+ outputFile: './monocart-report/index.html',
994
+ state: {
995
+ onReceive: function(message) {
996
+ const test = this.getTest(message.testId);
997
+ if (test) {
998
+ // current test
999
+ }
1000
+ console.log('receive message on server', message);
1001
+ return {
1002
+ data: 'my response data'
1003
+ };
1004
+ }
1005
+ }
1006
+ }]
1007
+ ]
1008
+ };
1009
+ ```
1010
+ see example: [Allow specified test cases to run in sequence mode with lock/unlock state](https://github.com/cenfun/monocart-reporter-examples/tree/main/tests/global-state)
1011
+
1012
+ ## Merge Shard Reports
1013
+ There will be multiple reports to be generated if Playwright test executes in sharding mode. for example:
1014
+ ```sh
1015
+ npx playwright test --shard=1/3
1016
+ npx playwright test --shard=2/3
1017
+ npx playwright test --shard=3/3
1018
+ ```
1019
+ There are 3 reports will be generated.
1020
+
1021
+ ### Using `merge` API to merge all reports into one
1022
+ > Note: One more suite level "shard" will be added, its title will be the machine hostname, and the summary will be restated. All attachments will be copied to the merged output directory.
1023
+ ```js
1024
+ import { merge } from 'monocart-reporter';
1025
+
1026
+ // json file path
1027
+ const reportDataList = [
1028
+ 'path-to/shard1/index.json',
1029
+ 'path-to/shard2/index.json',
1030
+ 'path-to/shard3/index.json'
1031
+
1032
+ // Or load zip file directly if the output files is zipped
1033
+ // 'path-to/shard1/index.zip',
1034
+ // 'path-to/shard2/index.zip',
1035
+ // 'path-to/shard3/index.zip'
1036
+ ];
1037
+
1038
+ await merge(reportDataList, {
1039
+ name: 'My Merged Report',
1040
+ outputFile: 'merged-report/index.html',
1041
+ onEnd: async (reportData, helper) => {
1042
+ // send email or third party integration
1043
+ }
1044
+ });
1045
+ ```
1046
+
1047
+ > Note: The coverage reports will be merged automatically if we specify the `raw` report in coverage options:
1048
+ ```js
1049
+ // global coverage options
1050
+ coverage: {
1051
+ reports: [
1052
+ // for merging coverage reports
1053
+ 'raw'
1054
+ // we can merge and zip the raw report files
1055
+ // ['raw', { merge: true, zip: true }]
1056
+ ]
1057
+ }
1058
+ ```
1059
+ see example [merge.js](https://github.com/cenfun/monocart-reporter-examples/blob/main/scripts/merge.js)
1060
+
1061
+ ### Using `merge` CLI
1062
+ ```sh
1063
+ npx monocart merge <glob-patterns>
1064
+
1065
+ # -o --outputFile
1066
+ npx monocart merge path-to/shard*/index.json -o merged-reports/index.html
1067
+
1068
+ # -c --config
1069
+ npx monocart merge path-to/shard*/my-report.zip -c mr.config.js
1070
+
1071
+ # NOTE: The asterisk(*) is a special character which is interpreted by some operating systems
1072
+ # For example: Mac and Linux, please put it in quotes, but NOT for Windows
1073
+ npx monocart merge 'path-to/shard*/*.zip'
1074
+ ```
1075
+ The default config files (In order of priority)
1076
+ - mr.config.js
1077
+ - mr.config.cjs
1078
+ - mr.config.mjs
1079
+ - mr.config.json
1080
+ - mr.config.ts
1081
+
1082
+ Preload for TypeScript config file:
1083
+ - It requires node 18.19.0+
1084
+ - Installing tsx: `npm i -D tsx`
1085
+ - Using the `--import tsx` flag
1086
+ - see [comment](https://github.com/cenfun/monocart-reporter/issues/145#issuecomment-2365460013)
1087
+
1088
+
1089
+ ## onEnd Hook
1090
+ The `onEnd` function will be executed after report generated. Arguments:
1091
+ - `reportData` all report data, properties:
1092
+ - `name` (String) report name
1093
+ - `date` (Number) start date in milliseconds
1094
+ - `dateH` (String) human-readable date with `Date.toLocaleString()`
1095
+ - `duration` (Number) test duration in milliseconds
1096
+ - `durationH` (String) human-readable duration
1097
+ - `summary` (Object) test summary, includes `tests`, `suites`, `steps`, etc.
1098
+ - `rows` and `columns` (Array) all rows and columns data, both are tree structure, see [detail](https://cenfun.github.io/turbogrid/api.html#data)
1099
+ - `tags` (Object) tag collection
1100
+ - `metadata` (Object) metadata collection
1101
+ - `system` (Object) system information
1102
+ - `trends` (Array) historical trend data
1103
+ - `caseTypes` and `suiteTypes` (Array)
1104
+ - `cwd`, `outputDir` and `outputFile` (String)
1105
+ - `htmlPath`, `jsonPath` and `summaryTable` (String)
1106
+ - ...
1107
+ - `helper` APIs:
1108
+ - `helper.find(callback)` Find item like array `find` function.
1109
+ - `helper.filter(callback)` Filter list like array `filter` function.
1110
+ - `helper.forEach(callback)` Iterate all rows of data (suites/cases/steps), return `break` will break the iteration.
1111
+ - `helper.sendEmail(emailOptions)`
1112
+
1113
+ ```js
1114
+ // playwright.config.js
1115
+ module.exports = {
1116
+ reporter: [
1117
+ ['monocart-reporter', {
1118
+ name: "My Test Report",
1119
+ outputFile: './monocart-report/index.html',
1120
+ // async hook after report data generated
1121
+ onEnd: async (reportData, helper) => {
1122
+ // console.log(reportData.summary);
1123
+
1124
+ // find a test by title
1125
+ const myCase = helper.find((item, parent) => item.type === 'case' && item.title.includes('inline tag'));
1126
+ console.log(myCase && myCase.title);
1127
+
1128
+ // find a suite by title
1129
+ const mySuite = helper.find((item, parent) => item.type === 'suite' && item.title.includes('new syntax'));
1130
+ console.log(mySuite && mySuite.title);
1131
+
1132
+ // filter failed cases
1133
+ const failedCases = helper.filter((item, parent) => item.type === 'case' && item.caseType === 'failed');
1134
+ console.log(failedCases.map((it) => it.title));
1135
+
1136
+ // Iterate all items
1137
+ helper.forEach((item, parent) => {
1138
+ // do something
1139
+ });
1140
+
1141
+
1142
+ }
1143
+ }]
1144
+ ]
1145
+ };
1146
+ ```
1147
+
1148
+ ## Integration Examples
1149
+ By using the `onEnd` hook, we can integrate Playwright report with any other tools, such as:
1150
+ - [Email](https://github.com/cenfun/playwright-reporter-integrations/tree/main/send-email)
1151
+ - [Testrail](https://github.com/cenfun/playwright-reporter-integrations/blob/main/testrail)
1152
+ - [Qase](https://github.com/cenfun/playwright-reporter-integrations/tree/main/qase)
1153
+ - [Jira + Zephyr](https://github.com/cenfun/playwright-reporter-integrations/tree/main/zephyr-scale)
1154
+ - [Jira + Xray](https://github.com/cenfun/playwright-reporter-integrations/blob/main/xray)
1155
+ - [Slack](https://github.com/cenfun/playwright-reporter-integrations/tree/main/slack-webhook)
1156
+ - [Discord](https://github.com/cenfun/playwright-reporter-integrations/blob/main/discord-webhook)
1157
+ - [Teams](https://github.com/cenfun/playwright-reporter-integrations/blob/main/teams-webhook)
1158
+ - [BrowserStack](https://github.com/cenfun/playwright-reporter-integrations/blob/main/browserstack)
1159
+ - [Github Actions Summary](https://github.com/cenfun/playwright-reporter-integrations/blob/main/github-actions-summary)
1160
+ - [Dingtalk](https://github.com/cenfun/playwright-reporter-integrations/tree/main/dingtalk-webhook)/[Weixin](https://github.com/cenfun/playwright-reporter-integrations/tree/main/weixin-webhook)/[Feishu](https://github.com/cenfun/playwright-reporter-integrations/tree/main/feishu-webhook)
1161
+
1162
+ See [playwright-reporter-integrations](https://github.com/cenfun/playwright-reporter-integrations)
1163
+
1164
+ ## Contributing
1165
+ ```sh
1166
+ # Node.js 20+
1167
+ npm install starfall-cli -g
1168
+ npm install
1169
+
1170
+ npm run build
1171
+ npm run test
1172
+
1173
+ npm run dev
1174
+ ```
1175
+ ### Dependencies
1176
+ - UI Framework [Vue 3](https://github.com/vuejs/core)
1177
+ - Lightweight UI Components [vine-ui](https://github.com/cenfun/vine-ui)
1178
+ - High Performance Grid [turbogrid](https://github.com/cenfun/turbogrid)
1179
+ - String compress/decompress [lz-utils](https://github.com/cenfun/lz-utils)
1180
+ - Coverage Reporter [monocart-coverage-reports](https://github.com/cenfun/monocart-coverage-reports)