playwright-qase-reporter 2.1.3 → 2.1.4

package/README.md

@@ -176,6 +176,7 @@ Reporter options (\* - required):
 - `testops.run.complete` - Whether the run should be completed
 - `framework.browser.addAsParameter` - Whether to add the browser name as a parameter, default - `false`
 - `framework.browser.parameterName` - The name of the parameter to add the browser name to, default - `browser`
+- `framework.markAsFlaky` - Whether to mark tests as flaky if they passed after retries, default - `false`
 
 Example `playwright.config.js` config:
 
@@ -206,6 +207,7 @@ const config = {
             addAsParameter: true,
             parameterName: 'Browser Name',
           },
+          markAsFlaky: true,
         },
       },
     ],

package/changelog.md

@@ -1,3 +1,20 @@
+# playwright-qase-reporter@2.1.4
+
+## What's new
+
+Support marking tests as flaky if they passed after retries.
+
+```ts
+[
+  'playwright-qase-reporter',
+  {
+    framework: {
+      markAsFlaky: true,
+    },
+  },
+],
+```
+
 # playwright-qase-reporter@2.1.3
 
 ## What's new

package/dist/configSchema.js

@@ -26,7 +26,11 @@ exports.configSchema = {
                                     nullable: true,
                                 },
                             }
-                        }
+                        },
+                        markAsFlaky: {
+                            type: 'boolean',
+                            nullable: true,
+                        },
                     }
                 }
             }

package/dist/options.d.ts

@@ -3,4 +3,5 @@ export type ReporterOptionsType = {
         addAsParameter?: boolean;
         parameterName?: string;
     };
+    markAsFlaky?: boolean;
 };

package/dist/reporter.js

@@ -311,6 +311,10 @@ class PlaywrightQaseReporter {
                 suites = suites.filter(suite => suite !== browser);
             }
         }
+        // if markAsFlaky is true and the test passed after retries, mark the test as flaky
+        if (this.options.markAsFlaky && result.status === 'passed' && result.retry > 0) {
+            testCaseMetadata.fields['is_flaky'] = 'true';
+        }
         const testTitle = this.removeQaseIdsFromTitle(test.title);
         const testResult = {
             attachments: testCaseMetadata.attachments,

package/docs/usage.md

@@ -1,317 +1,286 @@
-# Qase Syntax
+# Qase Integration in Playwright
 
-> [**Click here**](../../examples/playwright/test/) to view Example tests for the following syntax.
+This guide demonstrates how to integrate Qase with Playwright, providing instructions on how to add Qase IDs, titles,
+fields, suites, comments, and file attachments to your test cases.
 
-Here is the complete list of syntax options available for the reporter:
-- [Qase Id](#qase-id)
-- [Qase Title](#qase-title)
-- [Steps](#steps)
-- [Fields](#fields)
-- [Suite](#suite)
-- [Parameters](#parameters)
-- [Comment](#comment)
-- [Attach](#attach)
-- [Ignore](#ignore)
-
-If you do not use any Qase syntax, the reporter uses the title from the `describe` and `test` functions as the Suite and
-Test case title respectively, when publishing results.
+---
 
-<br>
+## Adding QaseID to a Test
 
-### Import Statement
+To associate a QaseID with a test in Playwright, use the `qase` function. This function accepts a single integer
+representing the test's ID in Qase.
 
----
-Add the following statement at the beginning of your spec file, before any tests.
+### Example
 
 ```javascript
 import { qase } from 'playwright-qase-reporter';
-```
 
-<br>
+test(qase(1, 'test'), async ({ page }) => {
+  await page.goto('https://example.com');
+});
 
-### Example Spec file
+test(qase([1, 2, 3], 'test'), async ({ page }) => {
+  await page.goto('https://example.com');
+});
+```
 
 ---
 
+## Adding a Title to a Test
+
+You can provide a title for your test using the `qase.title` function. The function accepts a string, which will be
+used as the test's title in Qase. If no title is provided, the test method name will be used by default.
+
+### Example
+
 ```javascript
 import { qase } from 'playwright-qase-reporter';
 
-describe('Suite title', () => {
-
-  test(qase(1, "This is the test name"), async () => {
-    qase.title("This overrides the test name");
-    qase.suite("Suite name");
-    qase.fields({ 'severity': 'high', 'priority': 'medium' });
-    qase.attach({ paths: './tests/examples/attachments/test-file.txt' });
-    qase.comment("A comment for this result");
-    qase.ignore(); // doesn't report his result to Qase.
-    qase.parameters({ Username: "@test" });
-    qase.groupParameters({ Username: username, Password: "123" });
-    await test.step('Test step title', async () => {
-      // step logic 
-    });
-  });
+test('test', async ({ page }) => {
+  qase.title('Title');
+  await page.goto('https://example.com');
 });
 ```
 
-<br>
+---
 
-### Qase ID
+## Adding Fields to a Test
 
----
+The `qase.fields` function allows you to add additional metadata to a test case. You can specify multiple fields to
+enhance test case information in Qase.
 
-Qase IDs can be defined using two different methods. It is best to select one method and consistently use it throughout
-your tests. The first method is recommended.
+### System Fields
 
-Only one Qase Id can be linked to a test.
+- `description` — Description of the test case.
+- `preconditions` — Preconditions for the test case.
+- `postconditions` — Postconditions for the test case.
+- `severity` — Severity of the test case (e.g., `critical`, `major`).
+- `priority` — Priority of the test case (e.g., `high`, `low`).
+- `layer` — Test layer (e.g., `UI`, `API`).
 
-**Inline with the `test` Function**:
+### Example
 
 ```javascript
-test(qase(1, 'A simple test'), () => {
-  // Test logic here
+import { qase } from 'playwright-qase-reporter';
+
+test('test', async ({ page }) => {
+  qase.fields({ description: "Description", preconditions: "Preconditions" });
+  await page.goto('https://example.com');
 });
 ```
 
-**Inside the `test` Body**:
+---
+
+## Adding a Suite to a Test
+
+To assign a suite or sub-suite to a test, use the `qase.suite` function. It can receive a suite name, and optionally a
+sub-suite, both as strings.
+
+### Example
 
 ```javascript
-test('A simple test', () => {
-  qase.id(1);
-  // Test logic here
-});
-```
+import { qase } from 'playwright-qase-reporter';
 
-**In the test's annotations**:
+test('test', async ({ page }) => {
+  qase.suite("Suite 01");
+  await page.goto('https://example.com');
+});
 
-```js
-test('A simple test',
-  {
-    annotation: { type: 'QaseID', description: '1' },
-  },
-  async () => {
-    // Test logic here
-  });
+test('test', async ({ page }) => {
+  qase.suite("Suite 01\tSuite 02");
+  await page.goto('https://example.com');
+});
 ```
 
-<br>
+---
 
-### Qase Title
+## Ignoring a Test in Qase
 
----
+To exclude a test from being reported to Qase (while still executing the test in Playwright), use the `qase.ignore`
+function. The test will run, but its result will not be sent to Qase.
 
-The `qase.title()` method is used to set the title of a test case, both when creating a new test case from the result,
-and when updating the title of an existing test case - *if used with `qase.id()`.*
+### Example
 
 ```javascript
-test("This won't appear in Qase", () => {
-  qase.title("This text will be the title of the test, in Qase");
-  // Test logic here
+import { qase } from 'playwright-qase-reporter';
+
+test('test', async ({ page }) => {
+  qase.ignore();
+  await page.goto('https://example.com');
 });
 ```
 
-If you don’t explicitly set a title using this method, the title specified in the `test(..)` function will be used for
-creating new test cases. However, if this method is defined, it always takes precedence and overrides the title from the
-`test(..)` function.
-
-<br>
-
-### Steps
-
 ---
 
-The reporter uses the title from the `test.step` function as the step title. By providing clear and descriptive step
-names, you make it easier to understand the test’s flow when reviewing the test case.
+## Adding a Comment to a Test
 
-Additionally, these steps get their own result in the Qase Test run, offering a well-organized summary of the test flow.
-This helps quickly identify the cause of any failures.
+You can attach comments to the test results in Qase using the `qase.comment` function. The comment will be displayed
+alongside the test execution details in Qase.
+
+### Example
 
 ```javascript
-test('A Test case with steps, updated from code', async () => {
-  await test.step('Initialize the environment', async () => {
-    // Set up test environment
-  });
-  await test.step('Test Core Functionality of the app', async () => {
-    // Exercise core functionality
-  });
+import { qase } from 'playwright-qase-reporter';
 
-  await test.step('Verify Expected Behavior of the app', async () => {
-    // Assert expected behavior
-  });
+test('test', async ({ page }) => {
+  qase.comment("Some comment");
+  await page.goto('https://example.com');
 });
 ```
 
-You also can use the `qase.step()` method to set the expected result and data of the step, which will be used in the
-Qase report.
+---
+
+## Attaching Files to a Test
+
+To attach files to a test result, use the `qase.attach` function. This method supports attaching one or multiple files,
+along with optional file names, comments, and file types.
+
+### Example
 
 ```javascript
-test('A Test case with steps, updated from code', async () => {
-  await test.step(qase.step('Initialize the environment'), async () => {
-    // Set up test environment
-  });
-  await test.step(qase.step('Test Core Functionality of the app', 'expected result'), async () => {
-    // Exercise core functionality
-  });
+import { qase } from 'playwright-qase-reporter';
 
-  await test.step(qase.step('Verify Expected Behavior of the app', 'expected result', 'data'), async () => {
-    // Assert expected behavior
-  });
+test('test', async ({ page }) => {
+  qase.attach({ name: 'attachment.txt', content: 'Hello, world!', contentType: 'text/plain' });
+  qase.attach({ paths: '/path/to/file' });
+  qase.attach({ paths: ['/path/to/file', '/path/to/another/file'] });
+  await page.goto('https://example.com');
 });
 ```
 
-<br>
-
-### Fields
+## Adding Parameters to a Test
 
----
+You can add parameters to a test case using the `qase.parameters` function. This function accepts an object with
+parameter names and values.
 
-You can define the `description`, `pre-conditions`, `post-conditions`, and fields such as `severity`, `priority`, and
-`layer` using this method, which enables you to specify and maintain the context of the case directly within your code.
+### Example
 
 ```javascript
-test('Maintain your test meta-data from code', async () => {
-  qase.fields({
-    severity: 'high',
-    priority: 'medium',
-    layer: 'api',
-    precondition: 'add your precondition',
-    postcondition: 'add your postcondition',
-    description: `Code it quick, fix it slow,
-                    Tech debt grows where shortcuts go,
-                    Refactor later? Ha! We know.`
-  })
-  //  test logic here
+import { qase } from 'playwright-qase-reporter';
+
+test('test', async ({ page }) => {
+  qase.parameters({ param1: 'value1', param2: 'value2' });
+  await page.goto('https://example.com');
 });
 ```
 
-<br>
-
-### Suite
+## Adding Group Parameters to a Test
 
----
+To add group parameters to a test case, use the `qase.groupParameters` function. This function accepts an object with
+group parameter names and values.
 
-You can use this method to nest the resulting test cases in a particular suite. There's something to note here – suites,
-unlike test cases, are not identified uniquely by the Reporter. Therefore, when defining an existing suite - the title
-of the suite is used for matching.
+### Example
 
-```js
-test('Test with a defined suite', () => {
-  qase.suite('Suite defined with qase.suite()');
-});
+```javascript
+import { qase } from 'playwright-qase-reporter';
 
-/*
- *  Or, nest multiple levels of suites. 
- *  `\t` is used for dividing each suite name.
- */
-test('Test with a nested suite', () => {
-  qase.suite('Application\tAuthentication\tLogin\tEdge_case');
-  //  test logic here
+test('test', async ({ page }) => {
+  qase.parameters({ param1: 'value1', param2: 'value2' });
+  qase.groupParameters({ param3: 'value3', param4: 'value4' });
+  await page.goto('https://example.com');
 });
-
-test('Test with a suite from annotations',
-  {
-    annotation: {
-      type: 'QaseSuite', 
-      description: 'Suite defined in annotation',
-    },
-  },
-  async () => {
-    // Test logic here
-  });
 ```
 
-<br>
+## Adding Steps to a Test
 
-### Parameters
+You can add steps to a test case using the `qase.step` function. This function accepts a string, which will be used as
+the step description in Qase.
 
----
-Parameters are a great way to make your tests more dynamic, reusable, and data-driven. By defining parameters in this
-method, you can ensure only one test case with all the parameters is created in your Qase project, avoiding duplication.
+### Example
 
 ```javascript
-const testCases = [
-  { browser: "Chromium", username: "@alice", password: "123" },
-  { browser: "Firefox", username: "@bob", password: "456" },
-  { browser: "Webkit", username: "@charlie", password: "789" },
-];
-
-testCases.forEach(({ browser, username, password, }) => {
-  test(`Test login with ${browser}`, async () => {
-    qase.title("Verify if page loads on all browsers");
-
-    qase.parameters({ Browser: browser });  // Single parameter
-    // test logic
-
-    testCases.forEach(({ username, password }) => {
-      test(`Test login with ${username} using qase.groupParameters`, () => {
-        qase.title("Verify if user is able to login with their username.");
-
-        qase.groupParameters({  // Group parameters
-          Username: username,
-          Password: password,
-        });
-        // test logic
-      });
-    });
+import { qase } from 'playwright-qase-reporter';
+
+test('test', async ({ page }) => {
+  await test.step(qase.step('Some step'), async () => {
+    // some actions
   });
+  await page.goto('https://example.com');
 });
 ```
 
-<br>
+## Annotations
 
-### Comment
+Playwright Qase reporter supports test annotations for setting Qase IDs, titles, and suites.
 
----
-In addition to `test.step()`, this method can be used to provide any additional context to your test, it helps maintiain
-the code by clarifying the expected result of the test.
-
-```js
-test("A test case with qase.comment()", () => {
-  /*
-   * Please note, this comment is added to a Result, not to the Test case.
-   */
-  qase.comment("This comment is added to the result");
-  // test logic here
-});
-```
+### Example
 
-<br>
+```javascript
+test('test',
+  {
+    annotation: { type: 'QaseID', description: '1' },
+  },
+  async ({ page }) => {
+    await page.goto('https://example.com');
+  });
 
-### Attach
+test('test',
+  {
+    annotation: { type: 'QaseSuite', description: 'Suite defined in annotation' },
+  },
+  async ({ page }) => {
+    await page.goto('https://example.com');
+  });
+```
 
 ---
-This method can help attach one, or more files to the test's result. You can also add the file's contents directly from
-code. For example:
 
-```js
-test('Test result with attachment', async () => {
+## Selective execution tests
 
-  // To attach a single file
-  qase.attach({
-    paths: "./tests/examples/attachments/test-file.txt",
-  });
+You can use the `grep` property to select tests to run. You can specify a regular expression to match the Qase IDs in `playwright.config.js` or in command line arguments.
 
-  // Add multiple attachments. 
-  qase.attach({ paths: ['/path/to/file', '/path/to/another/file'] });
+### Configuration-based filtering
 
+```javascript
+const config = {
+  grep: /(Qase ID: 1|2|3)/,
+  reporter: [
+    [
+      'playwright-qase-reporter',
+      {
+        debug: true,
+
+        testops: {
+          api: {
+            token: 'api_key',
+          },
+
+          project: 'project_code',
+          uploadAttachments: true,
+
+          run: {
+            complete: true,
+          },
+        },
+      },
+    ],
+  ],
+};
+
+module.exports = config;
+```
 
-  // Upload file's contents directly from code.
-  qase.attach({ name: 'attachment.txt', content: 'Hello, world!', contentType: 'text/plain' });
-  // test logic here
-});
+### Command line filtering
+
+```bash
+# Run tests with specific Qase IDs
+npx playwright test --grep "(Qase ID: 1|2|3)"
 ```
 
-<br>
+### Using qasectl for test plan filtering
 
-### Ignore
+If you use `qase([Id], 'Test name')` syntax for test case IDs, you can use `qasectl` for getting prepared regex with all Qase IDs from your test plan. See [qasectl](https://github.com/qase-tms/qasectl/blob/main/docs/command.md#get-filtered-results) for more information.
 
----
-If this method is added, the reporter will exclude the test’s result from the report sent to Qase. While the test will
-still executed by Playwright, its result will not be considered by the reporter.
+For example, if you have a test plan with ID 123, you can get the regular expression with all Qase IDs from it with the following command:
 
-```js
-test('This test is executed using Playwright; however, it is NOT reported to Qase', () => {
-  qase.ignore();
-//  test logic here
-});
+```bash
+qasectl testops filter --project PROJ --token <token> --planID 123 --framework playwright --output qase.env --verbose
 ```
+
+Specify result to run command:
+
+```bash
+npx playwright test --grep "$(cat qase.env | grep QASE_FILTERED_RESULTS | cut -d'=' -f2)"
+```
+
+Only tests with Qase IDs from the file will be run and reported to Qase.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "playwright-qase-reporter",
-  "version": "2.1.3",
+  "version": "2.1.4",
   "description": "Qase TMS Playwright Reporter",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",