@alex_neo/playwright-azure-reporter 1.20.2 → 1.21.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +20 -0
- package/README.md +101 -0
- package/dist/compat.d.ts +73 -0
- package/dist/compat.js +140 -0
- package/dist/compat.js.map +1 -0
- package/dist/playwright-azure-reporter.d.ts +77 -0
- package/dist/playwright-azure-reporter.js +326 -4
- package/dist/playwright-azure-reporter.js.map +1 -1
- package/dist/test-steps.d.ts +87 -0
- package/dist/test-steps.js +209 -0
- package/dist/test-steps.js.map +1 -0
- package/package.json +3 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,23 @@
|
|
|
1
|
+
## v1.21.0
|
|
2
|
+
|
|
3
|
+
Released: 2026-10-17
|
|
4
|
+
|
|
5
|
+
### Features
|
|
6
|
+
- feat(reporter): add the feature matrix and a Playwright compatibility policy (fc2d4c9)
|
|
7
|
+
- feat(reporter): align published steps with the test case work item (84ef0bb)
|
|
8
|
+
- feat(reporter): publish test.step results with a PATCH in every publishing mode (cff5756)
|
|
9
|
+
- feat(reporter): add pure mapping of Playwright steps to Azure DevOps step results (a3aa9b3)
|
|
10
|
+
- feat(reporter): add publishTestStepResults option to publish test.step outcomes to Azure DevOps (928a45b)
|
|
11
|
+
|
|
12
|
+
### Bug Fixes
|
|
13
|
+
- fix(tests): keep reporter state out of the environment of inline runs (386f026)
|
|
14
|
+
- fix(tests): correct file extension assertions for JPEG in reporter utils (016e1d7)
|
|
15
|
+
|
|
16
|
+
### Other Changes
|
|
17
|
+
- docs(readme): make the Playwright Compatibility section fit the README column (d8cc338)
|
|
18
|
+
- docs(readme): describe publishTestSteps and the step matching rules (1f2f2ce)
|
|
19
|
+
- chore(deps): drop dependency bump from PR, keep package.json and yarn.lock as on main (eb53d95)
|
|
20
|
+
|
|
1
21
|
## v1.20.2
|
|
2
22
|
|
|
3
23
|
Released: 2026-08-07
|
package/README.md
CHANGED
|
@@ -17,12 +17,14 @@ A Playwright reporter that publishes test results to Azure DevOps Test Plans.
|
|
|
17
17
|
- [Authentication](#authentication)
|
|
18
18
|
- [Publishing Modes](#publishing-modes)
|
|
19
19
|
- [Retry Results](#retry-results)
|
|
20
|
+
- [Test Step Results](#test-step-results)
|
|
20
21
|
- [Attachments & Logs](#attachments--logs)
|
|
21
22
|
- [Test Point Mapping](#test-point-mapping)
|
|
22
23
|
- [Test Case ID Matching](#test-case-id-matching)
|
|
23
24
|
- [Test Case Summary](#test-case-summary)
|
|
24
25
|
- [Auto-Mark as Automated](#auto-mark-as-automated)
|
|
25
26
|
- [CI/CD Integration](#cicd-integration)
|
|
27
|
+
- [Playwright Compatibility](#playwright-compatibility)
|
|
26
28
|
- [Releases](#releases)
|
|
27
29
|
|
|
28
30
|
## Quick Start
|
|
@@ -180,6 +182,7 @@ For more examples, see [Authentication Examples](tests/examples/authType-example
|
|
|
180
182
|
|-------|--------|---------|
|
|
181
183
|
| Test Management | Read & Write | Creating/updating test runs, publishing results, uploading attachments |
|
|
182
184
|
| Work Items | Read & Write | Marking test cases as automated (only if `autoMarkTestCasesAsAutomated` is enabled) |
|
|
185
|
+
| Work Items | Read | Aligning published steps with the test case (only if `publishTestSteps` is enabled with `alignWithTestCase`, the default) |
|
|
183
186
|
| Project and Team | Read | Accessing project information |
|
|
184
187
|
|
|
185
188
|
Alternatively, use a token with **Full access** scope.
|
|
@@ -231,6 +234,75 @@ Control how retry attempts are published with `publishRetryResults`:
|
|
|
231
234
|
}
|
|
232
235
|
```
|
|
233
236
|
|
|
237
|
+
### Test Step Results
|
|
238
|
+
|
|
239
|
+
Publish the outcome of every `test.step()` as a step result of the Azure DevOps test case result, so the **Steps** view of a result shows which steps passed, failed or were never reached.
|
|
240
|
+
|
|
241
|
+
```typescript
|
|
242
|
+
{
|
|
243
|
+
publishTestSteps: true,
|
|
244
|
+
}
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
or, with options:
|
|
248
|
+
|
|
249
|
+
```typescript
|
|
250
|
+
{
|
|
251
|
+
publishTestSteps: {
|
|
252
|
+
enabled: true,
|
|
253
|
+
categories: ['test.step'], // Playwright step categories to publish
|
|
254
|
+
testStepIdMatcher: /\[(\d+)\]/, // explicit step number in the step title
|
|
255
|
+
maxDepth: 1, // 1 = top-level steps only
|
|
256
|
+
alignWithTestCase: true, // read the test case steps from Azure DevOps
|
|
257
|
+
},
|
|
258
|
+
}
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
| Sub-option | Type | Default | Description |
|
|
262
|
+
|------------|------|---------|-------------|
|
|
263
|
+
| `enabled` | `boolean` | `false` | Enable step publishing (required in the object form) |
|
|
264
|
+
| `categories` | `string[]` | `['test.step']` | Playwright step categories to publish. Add `'expect'` or `'pw:api'` to publish assertions or Playwright calls as steps |
|
|
265
|
+
| `testStepIdMatcher` | `string \| RegExp` | `/\[(\d+)\]/` | Extracts an explicit step number from the step title; group 1 is the number |
|
|
266
|
+
| `maxDepth` | `number` | `1` | How deep nested steps are collected. `1` publishes top-level steps only |
|
|
267
|
+
| `alignWithTestCase` | `boolean` | `true` | Read `Microsoft.VSTS.TCM.Steps` of the test case and map onto its real step ids. Requires the **Work Items (Read)** token scope |
|
|
268
|
+
|
|
269
|
+
#### How steps are matched
|
|
270
|
+
|
|
271
|
+
1. Each published step gets a **step number**: an explicit `[N]` in the title wins, otherwise the number continues after the previous step (`1, 2, 3, ...`). The number refers to the position of the step in the Azure DevOps test case (1-based).
|
|
272
|
+
2. With `alignWithTestCase: true` (default) the reporter reads the steps of the test case work item once per run and maps number `N` onto the id of the N-th step. This keeps results correct after steps were deleted or reordered in Azure DevOps. Work item steps the test never reached are published as `NotExecuted`, with a note naming the step that failed.
|
|
273
|
+
3. Without alignment (option `false`, missing token scope, or the work item cannot be read) number `N` maps to step id `N + 1`, which is the id Azure DevOps assigns to steps that were never deleted or reordered.
|
|
274
|
+
|
|
275
|
+
```typescript
|
|
276
|
+
test('[31] login flow', async ({ page }) => {
|
|
277
|
+
await test.step('[1] navigate to login page', async () => {
|
|
278
|
+
await page.goto('/login');
|
|
279
|
+
});
|
|
280
|
+
|
|
281
|
+
await test.step('[2] fill credentials', async () => {
|
|
282
|
+
await page.fill('#username', 'user');
|
|
283
|
+
await page.fill('#password', 'pass');
|
|
284
|
+
});
|
|
285
|
+
|
|
286
|
+
// No number: continues as step 3
|
|
287
|
+
await test.step('submit form', async () => {
|
|
288
|
+
await page.click('#submit');
|
|
289
|
+
});
|
|
290
|
+
});
|
|
291
|
+
```
|
|
292
|
+
|
|
293
|
+
Each step result carries the outcome (`Passed`, `Failed`, `NotExecuted`), the error message, the duration, the start and end time, and the Playwright step title as comment. A step counts as failed when it or any nested step has an error, so a failed `expect.soft()` inside a `test.step()` marks that step `Failed` while later steps keep their own outcome. With Playwright 1.63 or newer, `test.step(title, body, { params, subtitle })` adds the params as iteration parameters and the subtitle to the comment (see [Playwright Compatibility](#playwright-compatibility)).
|
|
294
|
+
|
|
295
|
+
Step results are sent with a second request after the test result is created (Azure DevOps ignores step details on the request that creates a result). Works in all publishing modes (`testResult`, `testRun`, `testRunADO`). A failed step request is logged and never fails the publishing of the test result itself.
|
|
296
|
+
|
|
297
|
+
A `Test steps summary` line is printed at the end of the run. Steps that could not be published are logged with the reason:
|
|
298
|
+
|
|
299
|
+
- the test case has fewer steps than the test;
|
|
300
|
+
- two steps resolve to the same number;
|
|
301
|
+
- the step maps to a **Shared Steps** reference in the test case (shared steps are not supported);
|
|
302
|
+
- the test case has no steps at all (nothing is published for it).
|
|
303
|
+
|
|
304
|
+
> **Note:** `[N]` in a step title and `[N]` in a test title are matched independently: `testCaseIdMatcher` only looks at test titles, `testStepIdMatcher` only at step titles.
|
|
305
|
+
|
|
234
306
|
### Attachments & Logs
|
|
235
307
|
|
|
236
308
|
| Option | Type | Default | Description |
|
|
@@ -489,6 +561,35 @@ Automatically update Azure DevOps test case work items to reflect their automati
|
|
|
489
561
|
displayName: 'Print test run id'
|
|
490
562
|
```
|
|
491
563
|
|
|
564
|
+
## Playwright Compatibility
|
|
565
|
+
|
|
566
|
+
Some reporter features need data that only newer Playwright versions provide. On startup the reporter reads the installed Playwright version (from `@playwright/test/package.json` next to your project, or from the reporter's own location) and checks the features you enabled against this matrix:
|
|
567
|
+
|
|
568
|
+
| Feature | Playwright | Level | If Playwright is older |
|
|
569
|
+
|:---|:---|:---|:---|
|
|
570
|
+
| `publishTestSteps`<br>TestResult.steps and the TestStep API | >= 1.10.0 | required | step publishing is switched off |
|
|
571
|
+
| `publishTestSteps.params`<br>TestStep.params and TestStep.subtitle (test.step options) | >= 1.63.0 | enrichment | steps are published without iteration parameters and subtitles |
|
|
572
|
+
| `testCaseIdZone.title.tags`<br>TestCase.tags, so test case ids can be declared with tag: ["@[123]"] | >= 1.42.0 | enrichment | ids are read from test titles only |
|
|
573
|
+
|
|
574
|
+
- **required**: the feature cannot work below that version.
|
|
575
|
+
- **enrichment**: the feature works, but part of its output is unavailable below that version.
|
|
576
|
+
|
|
577
|
+
Set `compatibility` to choose what happens when the installed Playwright is older than an enabled feature needs:
|
|
578
|
+
|
|
579
|
+
- `'lenient'` (default): a required feature is switched off with a warning. A missing enrichment is reported once and the feature keeps running.
|
|
580
|
+
- `'strict'`: any mismatch is a configuration error. The reporter throws on startup and Playwright exits before the first test runs.
|
|
581
|
+
|
|
582
|
+
```typescript
|
|
583
|
+
{
|
|
584
|
+
publishTestSteps: true,
|
|
585
|
+
compatibility: 'strict', // fail the run instead of silently publishing less
|
|
586
|
+
}
|
|
587
|
+
```
|
|
588
|
+
|
|
589
|
+
When the Playwright version cannot be detected, the check is skipped in both modes and logged at debug level (`AZUREPWDEBUG=1`).
|
|
590
|
+
|
|
591
|
+
<!-- The table above is generated from src/compat.ts: run `yarn build && yarn feature-matrix` and paste the output when the matrix changes. A test fails when README and the matrix differ. -->
|
|
592
|
+
|
|
492
593
|
## Releases
|
|
493
594
|
|
|
494
595
|
For detailed information about stable releases, beta releases, and version management, see [RELEASES.md](RELEASES.md).
|
package/dist/compat.d.ts
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Feature matrix: which reporter features need which Playwright version, and what to do when
|
|
3
|
+
* the installed Playwright is older than a feature requires.
|
|
4
|
+
*
|
|
5
|
+
* Two policies exist:
|
|
6
|
+
* - `lenient` (default): a `required` feature that cannot work is switched off with a warning,
|
|
7
|
+
* an `enrichment` that cannot work is reported once and the feature keeps running without it.
|
|
8
|
+
* - `strict`: any unmet requirement is a configuration error. The reporter throws from its
|
|
9
|
+
* constructor, which makes Playwright exit before the first test runs. Errors thrown from the
|
|
10
|
+
* reporter hooks are swallowed by Playwright, so the constructor is the only place where a
|
|
11
|
+
* hard stop is possible.
|
|
12
|
+
*/
|
|
13
|
+
export type CompatibilityPolicy = 'strict' | 'lenient';
|
|
14
|
+
export type FeatureLevel = 'required' | 'enrichment';
|
|
15
|
+
export interface FeatureRequirement {
|
|
16
|
+
/** Stable identifier, also used in log messages. */
|
|
17
|
+
id: string;
|
|
18
|
+
/** Lowest Playwright version that provides what the feature needs. */
|
|
19
|
+
since: string;
|
|
20
|
+
/**
|
|
21
|
+
* `required`: the feature cannot work at all below `since`.
|
|
22
|
+
* `enrichment`: the feature works, but part of its output is unavailable below `since`.
|
|
23
|
+
*/
|
|
24
|
+
level: FeatureLevel;
|
|
25
|
+
/** What Playwright provides from `since` on. */
|
|
26
|
+
provides: string;
|
|
27
|
+
/** What happens below `since` in lenient mode. */
|
|
28
|
+
fallback: string;
|
|
29
|
+
}
|
|
30
|
+
/**
|
|
31
|
+
* The feature matrix. Keep entries sorted by `id`.
|
|
32
|
+
* README.md "Playwright compatibility" is generated from this list with `yarn feature-matrix`.
|
|
33
|
+
*/
|
|
34
|
+
export declare const FEATURE_MATRIX: readonly FeatureRequirement[];
|
|
35
|
+
export interface PlaywrightVersionInfo {
|
|
36
|
+
version: string;
|
|
37
|
+
/** Path of the package.json the version was read from. */
|
|
38
|
+
source: string;
|
|
39
|
+
}
|
|
40
|
+
export interface CompatibilityIssue {
|
|
41
|
+
feature: FeatureRequirement;
|
|
42
|
+
actual: PlaywrightVersionInfo;
|
|
43
|
+
}
|
|
44
|
+
export type ParsedVersion = [number, number, number];
|
|
45
|
+
export declare function parseVersion(version: string): ParsedVersion | undefined;
|
|
46
|
+
/** Returns a negative number when `a` is older than `b`, 0 when equal, positive when newer. */
|
|
47
|
+
export declare function compareVersions(a: string, b: string): number;
|
|
48
|
+
export declare function satisfies(version: string, since: string): boolean;
|
|
49
|
+
type Resolver = (request: string, options: {
|
|
50
|
+
paths: string[];
|
|
51
|
+
}) => string;
|
|
52
|
+
/**
|
|
53
|
+
* Finds the Playwright installation the reporter runs with. Resolution starts from the working
|
|
54
|
+
* directory (the user's project) and then from the reporter's own location, which covers both
|
|
55
|
+
* a project install and a monorepo with hoisted dependencies.
|
|
56
|
+
*/
|
|
57
|
+
export declare function detectPlaywrightVersion(searchPaths?: string[], resolver?: Resolver): PlaywrightVersionInfo | undefined;
|
|
58
|
+
/**
|
|
59
|
+
* Checks the requested features against the installed Playwright version.
|
|
60
|
+
* Unknown feature ids are ignored, unknown versions produce no issues.
|
|
61
|
+
*/
|
|
62
|
+
export declare function evaluateCompatibility(actual: PlaywrightVersionInfo | undefined, requestedFeatureIds: Iterable<string>, matrix?: readonly FeatureRequirement[]): CompatibilityIssue[];
|
|
63
|
+
export declare function describeIssue(issue: CompatibilityIssue): string;
|
|
64
|
+
/** The message of the error thrown in strict mode. */
|
|
65
|
+
export declare function formatStrictModeError(issues: CompatibilityIssue[]): string;
|
|
66
|
+
/**
|
|
67
|
+
* Markdown table of the feature matrix, used for README.md.
|
|
68
|
+
*
|
|
69
|
+
* Four columns on purpose: GitHub scrolls tables sideways when they are wider than the README column.
|
|
70
|
+
* The feature id and what it needs share one cell (id on the first line) instead of getting a column each.
|
|
71
|
+
*/
|
|
72
|
+
export declare function formatFeatureMatrix(matrix?: readonly FeatureRequirement[]): string;
|
|
73
|
+
export {};
|
package/dist/compat.js
ADDED
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
/**
|
|
3
|
+
* Feature matrix: which reporter features need which Playwright version, and what to do when
|
|
4
|
+
* the installed Playwright is older than a feature requires.
|
|
5
|
+
*
|
|
6
|
+
* Two policies exist:
|
|
7
|
+
* - `lenient` (default): a `required` feature that cannot work is switched off with a warning,
|
|
8
|
+
* an `enrichment` that cannot work is reported once and the feature keeps running without it.
|
|
9
|
+
* - `strict`: any unmet requirement is a configuration error. The reporter throws from its
|
|
10
|
+
* constructor, which makes Playwright exit before the first test runs. Errors thrown from the
|
|
11
|
+
* reporter hooks are swallowed by Playwright, so the constructor is the only place where a
|
|
12
|
+
* hard stop is possible.
|
|
13
|
+
*/
|
|
14
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
15
|
+
exports.FEATURE_MATRIX = void 0;
|
|
16
|
+
exports.parseVersion = parseVersion;
|
|
17
|
+
exports.compareVersions = compareVersions;
|
|
18
|
+
exports.satisfies = satisfies;
|
|
19
|
+
exports.detectPlaywrightVersion = detectPlaywrightVersion;
|
|
20
|
+
exports.evaluateCompatibility = evaluateCompatibility;
|
|
21
|
+
exports.describeIssue = describeIssue;
|
|
22
|
+
exports.formatStrictModeError = formatStrictModeError;
|
|
23
|
+
exports.formatFeatureMatrix = formatFeatureMatrix;
|
|
24
|
+
/**
|
|
25
|
+
* The feature matrix. Keep entries sorted by `id`.
|
|
26
|
+
* README.md "Playwright compatibility" is generated from this list with `yarn feature-matrix`.
|
|
27
|
+
*/
|
|
28
|
+
exports.FEATURE_MATRIX = [
|
|
29
|
+
{
|
|
30
|
+
id: 'publishTestSteps',
|
|
31
|
+
since: '1.10.0',
|
|
32
|
+
level: 'required',
|
|
33
|
+
provides: 'TestResult.steps and the TestStep API',
|
|
34
|
+
fallback: 'step publishing is switched off',
|
|
35
|
+
},
|
|
36
|
+
{
|
|
37
|
+
id: 'publishTestSteps.params',
|
|
38
|
+
since: '1.63.0',
|
|
39
|
+
level: 'enrichment',
|
|
40
|
+
provides: 'TestStep.params and TestStep.subtitle (test.step options)',
|
|
41
|
+
fallback: 'steps are published without iteration parameters and subtitles',
|
|
42
|
+
},
|
|
43
|
+
{
|
|
44
|
+
id: 'testCaseIdZone.title.tags',
|
|
45
|
+
since: '1.42.0',
|
|
46
|
+
level: 'enrichment',
|
|
47
|
+
provides: 'TestCase.tags, so test case ids can be declared with tag: ["@[123]"]',
|
|
48
|
+
fallback: 'ids are read from test titles only',
|
|
49
|
+
},
|
|
50
|
+
];
|
|
51
|
+
function parseVersion(version) {
|
|
52
|
+
const match = /^(\d+)\.(\d+)(?:\.(\d+))?/.exec(version.trim());
|
|
53
|
+
if (!match)
|
|
54
|
+
return undefined;
|
|
55
|
+
return [parseInt(match[1], 10), parseInt(match[2], 10), parseInt(match[3] ?? '0', 10)];
|
|
56
|
+
}
|
|
57
|
+
/** Returns a negative number when `a` is older than `b`, 0 when equal, positive when newer. */
|
|
58
|
+
function compareVersions(a, b) {
|
|
59
|
+
const pa = parseVersion(a);
|
|
60
|
+
const pb = parseVersion(b);
|
|
61
|
+
if (!pa || !pb)
|
|
62
|
+
throw new Error(`Cannot compare versions "${a}" and "${b}"`);
|
|
63
|
+
for (let i = 0; i < 3; i++) {
|
|
64
|
+
if (pa[i] !== pb[i])
|
|
65
|
+
return pa[i] - pb[i];
|
|
66
|
+
}
|
|
67
|
+
return 0;
|
|
68
|
+
}
|
|
69
|
+
function satisfies(version, since) {
|
|
70
|
+
return compareVersions(version, since) >= 0;
|
|
71
|
+
}
|
|
72
|
+
/**
|
|
73
|
+
* Finds the Playwright installation the reporter runs with. Resolution starts from the working
|
|
74
|
+
* directory (the user's project) and then from the reporter's own location, which covers both
|
|
75
|
+
* a project install and a monorepo with hoisted dependencies.
|
|
76
|
+
*/
|
|
77
|
+
function detectPlaywrightVersion(searchPaths = [process.cwd(), __dirname], resolver = (request, options) => require.resolve(request, options)) {
|
|
78
|
+
for (const pkg of ['@playwright/test/package.json', 'playwright/package.json']) {
|
|
79
|
+
let source;
|
|
80
|
+
try {
|
|
81
|
+
source = resolver(pkg, { paths: searchPaths });
|
|
82
|
+
}
|
|
83
|
+
catch {
|
|
84
|
+
continue;
|
|
85
|
+
}
|
|
86
|
+
try {
|
|
87
|
+
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
|
88
|
+
const version = require(source).version;
|
|
89
|
+
if (typeof version === 'string' && parseVersion(version))
|
|
90
|
+
return { version, source };
|
|
91
|
+
}
|
|
92
|
+
catch {
|
|
93
|
+
continue;
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
return undefined;
|
|
97
|
+
}
|
|
98
|
+
/**
|
|
99
|
+
* Checks the requested features against the installed Playwright version.
|
|
100
|
+
* Unknown feature ids are ignored, unknown versions produce no issues.
|
|
101
|
+
*/
|
|
102
|
+
function evaluateCompatibility(actual, requestedFeatureIds, matrix = exports.FEATURE_MATRIX) {
|
|
103
|
+
if (!actual)
|
|
104
|
+
return [];
|
|
105
|
+
const requested = new Set(requestedFeatureIds);
|
|
106
|
+
return matrix
|
|
107
|
+
.filter((feature) => requested.has(feature.id) && !satisfies(actual.version, feature.since))
|
|
108
|
+
.map((feature) => ({ feature, actual }));
|
|
109
|
+
}
|
|
110
|
+
function describeIssue(issue) {
|
|
111
|
+
const { feature } = issue;
|
|
112
|
+
return `${feature.id} requires Playwright >= ${feature.since} (${feature.provides}); ${feature.level} feature, fallback: ${feature.fallback}`;
|
|
113
|
+
}
|
|
114
|
+
/** The message of the error thrown in strict mode. */
|
|
115
|
+
function formatStrictModeError(issues) {
|
|
116
|
+
const { actual } = issues[0];
|
|
117
|
+
const lines = issues.map((issue) => ` - ${describeIssue(issue)}`);
|
|
118
|
+
return (`[playwright-azure-reporter] Playwright ${actual.version} (${actual.source}) does not satisfy the enabled features:\n` +
|
|
119
|
+
`${lines.join('\n')}\n` +
|
|
120
|
+
`Upgrade Playwright, disable the feature(s), or set compatibility: 'lenient'.`);
|
|
121
|
+
}
|
|
122
|
+
/** Keeps a value on one table row: pipes would split the cell, newlines would end the row. */
|
|
123
|
+
function escapeTableCell(value) {
|
|
124
|
+
return value.replace(/\|/g, '\\|').replace(/\s*\r?\n\s*/g, ' ');
|
|
125
|
+
}
|
|
126
|
+
/**
|
|
127
|
+
* Markdown table of the feature matrix, used for README.md.
|
|
128
|
+
*
|
|
129
|
+
* Four columns on purpose: GitHub scrolls tables sideways when they are wider than the README column.
|
|
130
|
+
* The feature id and what it needs share one cell (id on the first line) instead of getting a column each.
|
|
131
|
+
*/
|
|
132
|
+
function formatFeatureMatrix(matrix = exports.FEATURE_MATRIX) {
|
|
133
|
+
const header = '| Feature | Playwright | Level | If Playwright is older |\n|:---|:---|:---|:---|';
|
|
134
|
+
const rows = matrix.map((f) => {
|
|
135
|
+
const feature = `\`${f.id}\`<br>${escapeTableCell(f.provides)}`;
|
|
136
|
+
return `| ${feature} | >= ${f.since} | ${f.level} | ${escapeTableCell(f.fallback)} |`;
|
|
137
|
+
});
|
|
138
|
+
return [header, ...rows].join('\n');
|
|
139
|
+
}
|
|
140
|
+
//# sourceMappingURL=compat.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"compat.js","sourceRoot":"","sources":["../src/compat.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;GAWG;;;AA+DH,oCAIC;AAGD,0CAQC;AAED,8BAEC;AASD,0DAoBC;AAMD,sDAUC;AAED,sCAGC;AAGD,sDAQC;AAaD,kDAOC;AA7ID;;;GAGG;AACU,QAAA,cAAc,GAAkC;IAC3D;QACE,EAAE,EAAE,kBAAkB;QACtB,KAAK,EAAE,QAAQ;QACf,KAAK,EAAE,UAAU;QACjB,QAAQ,EAAE,uCAAuC;QACjD,QAAQ,EAAE,iCAAiC;KAC5C;IACD;QACE,EAAE,EAAE,yBAAyB;QAC7B,KAAK,EAAE,QAAQ;QACf,KAAK,EAAE,YAAY;QACnB,QAAQ,EAAE,2DAA2D;QACrE,QAAQ,EAAE,gEAAgE;KAC3E;IACD;QACE,EAAE,EAAE,2BAA2B;QAC/B,KAAK,EAAE,QAAQ;QACf,KAAK,EAAE,YAAY;QACnB,QAAQ,EAAE,sEAAsE;QAChF,QAAQ,EAAE,oCAAoC;KAC/C;CACF,CAAC;AAeF,SAAgB,YAAY,CAAC,OAAe;IAC1C,MAAM,KAAK,GAAG,2BAA2B,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,EAAE,CAAC,CAAC;IAC/D,IAAI,CAAC,KAAK;QAAE,OAAO,SAAS,CAAC;IAC7B,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,EAAE,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,GAAG,EAAE,EAAE,CAAC,CAAC,CAAC;AACzF,CAAC;AAED,+FAA+F;AAC/F,SAAgB,eAAe,CAAC,CAAS,EAAE,CAAS;IAClD,MAAM,EAAE,GAAG,YAAY,CAAC,CAAC,CAAC,CAAC;IAC3B,MAAM,EAAE,GAAG,YAAY,CAAC,CAAC,CAAC,CAAC;IAC3B,IAAI,CAAC,EAAE,IAAI,CAAC,EAAE;QAAE,MAAM,IAAI,KAAK,CAAC,4BAA4B,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC;IAC7E,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;QAC3B,IAAI,EAAE,CAAC,CAAC,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC;YAAE,OAAO,EAAE,CAAC,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC,CAAC;IAC5C,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC;AAED,SAAgB,SAAS,CAAC,OAAe,EAAE,KAAa;IACtD,OAAO,eAAe,CAAC,OAAO,EAAE,KAAK,CAAC,IAAI,CAAC,CAAC;AAC9C,CAAC;AAID;;;;GAIG;AACH,SAAgB,uBAAuB,CACrC,cAAwB,CAAC,OAAO,CAAC,GAAG,EAAE,EAAE,SAAS,CAAC,EAClD,WAAqB,CAAC,OAAO,EAAE,OAAO,EAAE,EAAE,CAAC,OAAO,CAAC,OAAO,CAAC,OAAO,EAAE,OAAO,CAAC;IAE5E,KAAK,MAAM,GAAG,IAAI,CAAC,+BAA+B,EAAE,yBAAyB,CAAC,EAAE,CAAC;QAC/E,IAAI,MAAc,CAAC;QACnB,IAAI,CAAC;YACH,MAAM,GAAG,QAAQ,CAAC,GAAG,EAAE,EAAE,KAAK,EAAE,WAAW,EAAE,CAAC,CAAC;QACjD,CAAC;QAAC,MAAM,CAAC;YACP,SAAS;QACX,CAAC;QACD,IAAI,CAAC;YACH,iEAAiE;YACjE,MAAM,OAAO,GAAI,OAAO,CAAC,MAAM,CAA2B,CAAC,OAAO,CAAC;YACnE,IAAI,OAAO,OAAO,KAAK,QAAQ,IAAI,YAAY,CAAC,OAAO,CAAC;gBAAE,OAAO,EAAE,OAAO,EAAE,MAAM,EAAE,CAAC;QACvF,CAAC;QAAC,MAAM,CAAC;YACP,SAAS;QACX,CAAC;IACH,CAAC;IACD,OAAO,SAAS,CAAC;AACnB,CAAC;AAED;;;GAGG;AACH,SAAgB,qBAAqB,CACnC,MAAyC,EACzC,mBAAqC,EACrC,SAAwC,sBAAc;IAEtD,IAAI,CAAC,MAAM;QAAE,OAAO,EAAE,CAAC;IACvB,MAAM,SAAS,GAAG,IAAI,GAAG,CAAC,mBAAmB,CAAC,CAAC;IAC/C,OAAO,MAAM;SACV,MAAM,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,SAAS,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,OAAO,EAAE,OAAO,CAAC,KAAK,CAAC,CAAC;SAC3F,GAAG,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,CAAC,EAAE,OAAO,EAAE,MAAM,EAAE,CAAC,CAAC,CAAC;AAC7C,CAAC;AAED,SAAgB,aAAa,CAAC,KAAyB;IACrD,MAAM,EAAE,OAAO,EAAE,GAAG,KAAK,CAAC;IAC1B,OAAO,GAAG,OAAO,CAAC,EAAE,2BAA2B,OAAO,CAAC,KAAK,KAAK,OAAO,CAAC,QAAQ,MAAM,OAAO,CAAC,KAAK,uBAAuB,OAAO,CAAC,QAAQ,EAAE,CAAC;AAChJ,CAAC;AAED,sDAAsD;AACtD,SAAgB,qBAAqB,CAAC,MAA4B;IAChE,MAAM,EAAE,MAAM,EAAE,GAAG,MAAM,CAAC,CAAC,CAAC,CAAC;IAC7B,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,OAAO,aAAa,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;IACnE,OAAO,CACL,0CAA0C,MAAM,CAAC,OAAO,KAAK,MAAM,CAAC,MAAM,4CAA4C;QACtH,GAAG,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI;QACvB,8EAA8E,CAC/E,CAAC;AACJ,CAAC;AAED,8FAA8F;AAC9F,SAAS,eAAe,CAAC,KAAa;IACpC,OAAO,KAAK,CAAC,OAAO,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC,OAAO,CAAC,cAAc,EAAE,GAAG,CAAC,CAAC;AAClE,CAAC;AAED;;;;;GAKG;AACH,SAAgB,mBAAmB,CAAC,SAAwC,sBAAc;IACxF,MAAM,MAAM,GAAG,kFAAkF,CAAC;IAClG,MAAM,IAAI,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE;QAC5B,MAAM,OAAO,GAAG,KAAK,CAAC,CAAC,EAAE,SAAS,eAAe,CAAC,CAAC,CAAC,QAAQ,CAAC,EAAE,CAAC;QAChE,OAAO,KAAK,OAAO,SAAS,CAAC,CAAC,KAAK,MAAM,CAAC,CAAC,KAAK,MAAM,eAAe,CAAC,CAAC,CAAC,QAAQ,CAAC,IAAI,CAAC;IACxF,CAAC,CAAC,CAAC;IACH,OAAO,CAAC,MAAM,EAAE,GAAG,IAAI,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACtC,CAAC"}
|
|
@@ -1,11 +1,27 @@
|
|
|
1
1
|
import { Reporter, TestCase, TestResult } from '@playwright/test/reporter';
|
|
2
2
|
import * as TestInterfaces from 'azure-devops-node-api/interfaces/TestInterfaces';
|
|
3
|
+
import { CompatibilityPolicy } from './compat';
|
|
3
4
|
declare const attachmentTypesArray: readonly ["screenshot", "video", "trace"];
|
|
4
5
|
type TAttachmentType = Array<(typeof attachmentTypesArray)[number] | RegExp>;
|
|
5
6
|
type TTestRunConfig = Omit<TestInterfaces.RunCreateModel, 'name' | 'automated' | 'plan' | 'pointIds'> | undefined;
|
|
6
7
|
type TPublishTestResults = 'testResult' | 'testRun' | 'testRunADO';
|
|
7
8
|
type TPublishRetryResults = 'all' | 'last';
|
|
8
9
|
type TTestCaseIdZone = 'title' | 'annotation';
|
|
10
|
+
type TPublishTestStepsOptions = {
|
|
11
|
+
enabled?: boolean;
|
|
12
|
+
/** Playwright step categories to publish. Default: `['test.step']`. */
|
|
13
|
+
categories?: string[];
|
|
14
|
+
/** Extracts an explicit step number from the step title, group 1 is the number. Default: `/\[(\d+)\]/`. */
|
|
15
|
+
testStepIdMatcher?: string | RegExp;
|
|
16
|
+
/** Nesting depth of steps to publish. Default: 1 (top-level steps only). */
|
|
17
|
+
maxDepth?: number;
|
|
18
|
+
/**
|
|
19
|
+
* Read the steps of the test case work item (`Microsoft.VSTS.TCM.Steps`) and map Playwright steps onto
|
|
20
|
+
* their real ids; work item steps the test never reached are reported as NotExecuted.
|
|
21
|
+
* Requires the "Work Items (Read)" token scope. Default: true.
|
|
22
|
+
*/
|
|
23
|
+
alignWithTestCase?: boolean;
|
|
24
|
+
};
|
|
9
25
|
export interface AzureReporterOptions {
|
|
10
26
|
token?: string;
|
|
11
27
|
authType?: 'pat' | 'accessToken' | 'managedIdentity' | undefined;
|
|
@@ -43,6 +59,13 @@ export interface AzureReporterOptions {
|
|
|
43
59
|
automatedTestType?: (testCase: TestCase) => string;
|
|
44
60
|
};
|
|
45
61
|
publishRetryResults?: TPublishRetryResults;
|
|
62
|
+
publishTestSteps?: boolean | TPublishTestStepsOptions;
|
|
63
|
+
/**
|
|
64
|
+
* What to do when the installed Playwright is older than an enabled feature requires
|
|
65
|
+
* (see the feature matrix in README). `lenient` (default) switches the feature off or degrades it
|
|
66
|
+
* with a log line; `strict` throws on startup so the run never starts.
|
|
67
|
+
*/
|
|
68
|
+
compatibility?: CompatibilityPolicy;
|
|
46
69
|
}
|
|
47
70
|
declare class AzureDevOpsReporter implements Reporter {
|
|
48
71
|
private _logger;
|
|
@@ -103,9 +126,27 @@ declare class AzureDevOpsReporter implements Reporter {
|
|
|
103
126
|
private _workItemApi;
|
|
104
127
|
private _autoMarkStats;
|
|
105
128
|
private _publishRetryResults;
|
|
129
|
+
private _publishTestSteps;
|
|
130
|
+
private _testStepsOptions;
|
|
131
|
+
private _testStepsStats;
|
|
132
|
+
private _compatibility;
|
|
133
|
+
private _playwrightVersion;
|
|
134
|
+
private _testStepsAlignWithTestCase;
|
|
135
|
+
private _workItemStepsCache;
|
|
136
|
+
private _testStepsAlignmentWarned;
|
|
137
|
+
private _testCasesWithoutSteps;
|
|
106
138
|
private _unmatched;
|
|
107
139
|
constructor(options: AzureReporterOptions);
|
|
140
|
+
private _initPublishTestSteps;
|
|
108
141
|
_validateOptions(options: AzureReporterOptions): void;
|
|
142
|
+
/** Feature ids from the feature matrix that the current options turn on. */
|
|
143
|
+
private _requestedFeatures;
|
|
144
|
+
private _disableFeature;
|
|
145
|
+
/**
|
|
146
|
+
* Checks the enabled features against the installed Playwright (feature matrix, see src/compat.ts).
|
|
147
|
+
* Throws in strict mode: this runs from the constructor, so Playwright exits before any test runs.
|
|
148
|
+
*/
|
|
149
|
+
private _applyCompatibility;
|
|
109
150
|
private _getAuthenticationHandler;
|
|
110
151
|
private _refreshTokenIfNeeded;
|
|
111
152
|
private _fetchConfigurationNames;
|
|
@@ -162,6 +203,40 @@ declare class AzureDevOpsReporter implements Reporter {
|
|
|
162
203
|
private _uploadAttachmentsFunc;
|
|
163
204
|
private _uploadLogsAttachmentsFunc;
|
|
164
205
|
private _mapToAzureState;
|
|
206
|
+
/**
|
|
207
|
+
* Reads the steps of a test case work item, once per test case per run.
|
|
208
|
+
* Resolves to `undefined` when the work item cannot be read, in which case
|
|
209
|
+
* step ids fall back to the positional N + 1 scheme.
|
|
210
|
+
*/
|
|
211
|
+
private _getWorkItemSteps;
|
|
212
|
+
/**
|
|
213
|
+
* Maps the Playwright steps of a test result to an Azure DevOps iteration for one test case.
|
|
214
|
+
* Returns `undefined` when step publishing is disabled or the test has no publishable steps.
|
|
215
|
+
*/
|
|
216
|
+
private _buildTestStepsIteration;
|
|
217
|
+
/**
|
|
218
|
+
* Builds the iteration for every test point of a test result, keyed by test point id.
|
|
219
|
+
* Test points of the same test case share one iteration object.
|
|
220
|
+
*/
|
|
221
|
+
private _buildTestStepsIterationsByTestPoint;
|
|
222
|
+
/**
|
|
223
|
+
* Finds the created result id for each test point through the results query,
|
|
224
|
+
* the same way attachments are matched in testRun mode.
|
|
225
|
+
*/
|
|
226
|
+
private _resolveResultIdsByTestPoint;
|
|
227
|
+
/**
|
|
228
|
+
* testResult mode: publishes step results on the results just created for one test.
|
|
229
|
+
* When every test point shares the same iteration, all created ids are PATCHed directly;
|
|
230
|
+
* otherwise result ids are resolved by test point first.
|
|
231
|
+
*/
|
|
232
|
+
private _publishTestStepsForCreatedResults;
|
|
233
|
+
/**
|
|
234
|
+
* Publishes step results with a PATCH on already created test results.
|
|
235
|
+
* Azure DevOps ignores `iterationDetails` on the POST that creates a result,
|
|
236
|
+
* so steps are always sent as a second request (Results - Update).
|
|
237
|
+
*/
|
|
238
|
+
private _publishTestStepsIterations;
|
|
239
|
+
private _countPublishedSteps;
|
|
165
240
|
private _createTestCaseResult;
|
|
166
241
|
private _prepareLogAttachments;
|
|
167
242
|
private _publishCaseResult;
|
|
@@ -171,4 +246,6 @@ declare class AzureDevOpsReporter implements Reporter {
|
|
|
171
246
|
private _setAzurePWTestRunId;
|
|
172
247
|
private _publishTestCaseSummaryAttachment;
|
|
173
248
|
}
|
|
249
|
+
export type { CompatibilityPolicy, FeatureRequirement } from './compat';
|
|
250
|
+
export { FEATURE_MATRIX, formatFeatureMatrix } from './compat';
|
|
174
251
|
export default AzureDevOpsReporter;
|