@testomatio/reporter 1.0.0 → 1.0.1-6.1

package/Changelog.md

@@ -1,6 +1,97 @@
+<!-- pending release updates -->
+# 1.0.16.1
+
+Renamed `TESTOMATIO_STACK_FILTER` to `TESTOMATIO_STACK_IGNORE`
+
+# 1.0.16
+
+* Addded [stack trace configuration](./docs/stacktrace.md) and documentation:
+
+```
+TESTOMATIO_STACK_IGNORE="tests/support/**.js" <actual-run-command>
+```
+* Jest: fixed reporting tests without a suite title
+
+# 1.0.15
+
+* Attach Run to Jira Issue via `TESTOMATIO_JIRA_ID` env variable:
+
+```
+TESTOMATIO_JIRA_ID=TST-12 <actual run command>
+```
+
+* Mocha - removed requirement of TESTOMATIO API Key
+
+# 1.0.14
+
+* Execute tests by tag names. Use this filter to select tests associated with specific tags.
+  
+  ```bash
+  TESTOMATIO={API_KEY} npx start-test-run -c 'actual run command' --filter 'testomatio:tag-name=smoke'
+  ```
+
+  * more instruction you can find in docs/pipes/testomatio.md
+
+# 1.0.13
+
+* JUnit improvements
+  * Match test from source code by adding Test ID as a comment:
+
+  ```java
+  // @T8acca9eb
+  ```
+  * Match test from output by adding Test ID as output:
+
+  ```java
+  System.out.println("tid://@T8acca9eb");
+  ```
+  * Support for suite before and after output
+  * Improved support for artifacts
+
+# 1.0.12
+
+& Logger refactoring by @olexandr13 in #208
+* fix undefined logs by @olexandr13 in #210
+
+# 1.0.11
+
+* fix steps duplication for codecept report by @olexandr13 in #209
+
+# 1.0.10
+
+* Added `TESTOMATIO_PUBLISH=1` variable to automatically publish run report
+
+# 1.0.9
+
+* Support XUnit format
+* Improved support for parametrized Java tests
+
+# 1.0.8 
+
+* Fixed `Can't read push of undefined` when logging steps
+
+# 1.0.6
+
+* Testomat.io. Auto-detect current build url and report it to Testomat.io. Manually url can be set with `BUILD_URL` variable:
+
+```
+BUILD_URL=https://.... TESTOMATIO=apiKey <actual test command>
+```
+
+# 1.0.5 
+
+* Fix "create tests" params processing for testomatio pipe
+
+# 1.0.4
+
+* Fixed parallel run
+
+# 1.0.3
+
+* Fixed reporting parallel runs
+
 # 1.0.0
 
-<!-- pending release updates -->
 
 * Added [`TESTOMATIO_SHARED_RUN` option](https://github.com/testomatio/reporter/blob/master/docs/pipes.md#reporting-parallel-execution-to-to-same-run) to use a shared run for parallel executions
 * Reworked [documentation](https://github.com/testomatio/reporter/tree/master#readme).

package/README.md

@@ -11,12 +11,13 @@ Testomat.io Reporter (this npm package) supports:
 
 * 🏄 Integarion with all popular [JavaScript/TypeScript frameworks](./docs/frameworks.md)
 * 🗄️ Screenshots, videos, traces [uploaded into S3 bucket](./docs/artifacts.md)
-* 🔎 Stack traces and error messages
-* 🐙 [GitHub](./docs/pipes.md#github-pipe) & [GitLab](./docs/pipes.md#gitlab-pipe) integration
+* 🔎 [Stack traces](./docs/stacktrace.md) and error messages
+* 🐙 [GitHub](./docs/pipes/github.md) & [GitLab](./docs/pipes/gitlab.md) integration
 * 🚅 Realtime reports
 * 🗃️ Other test frameworks supported via [JUNit XML](./docs/junit.md)
 * 🚶‍♀️ Steps *(work in progress)*
-* ☁️  Custom properties and metadata *(work in progress)*
+* 📄 [Logger](./docs/logger.md) *(work in progress, supports Jest for now)*
+* ☁️ Custom properties and metadata *(work in progress)*
 * 💯 Free & open-source.
 * 📊 Public and private Run reports on cloud via [Testomat.io App](https://testomat.io) 👇
 
@@ -64,7 +65,7 @@ yarn add @testomatio/reporter --dev
 
 ### 1️⃣ Attach Reporter to the Test Runner
 
-* ####  [Playwright](./docs/frameworks.md#playwright)
+* #### [Playwright](./docs/frameworks.md#playwright)
 * #### [CodeceptJS](./docs/frameworks.md#CodeceptJS)
 * #### [Cypress](./docs/frameworks.md#Cypress)
 * #### [Jest](./docs/frameworks.md#Jest)
@@ -84,13 +85,13 @@ or any [other via JUnit](./docs/junit.md) report....
 
 ### 2️⃣ Configure Reports
 
-* [Create report on Testomat.io](./docs/pipes.md#testomatio-pipe).
-* [Create brief summary report for GitHub Pull Request](./docs/pipes.md#github-pipe) 👇
-* [Create brief summary report for GitLab Merge Request](./docs/pipes.md#gitlab-pipe).
-* [Configure other pipes](./docs/pipes.md) for other ways to process test results output.
+* [Create report on Testomat.io](./docs/pipes/testomatio.md).
+* [Create brief summary report for GitHub Pull Request](./docs/pipes/github.md) 👇
+* [Create brief summary report for GitLab Merge Request](./docs/pipes/gitlab.md).
+* [Configure other pipes](./docs/pipes/md) for other ways to process test results output.
 
 
-![GitHub report](./docs/images/github.png)
+![GitHub report](./docs/pipes/images/github.png)
 
 GitHub report published as a comment to Pull Request:
 
@@ -99,7 +100,11 @@ GitHub report published as a comment to Pull Request:
 1. Create bucket on AWS, Google Cloud, or any other cloud storage provider supporting S3 protocol. 
 2. [Pass S3 credentials](./docs/artifacts.md) to reporter to enable artifacts uploading.
 
-### 4️⃣ Add to CI Pipeline
+### 4️⃣ Use Logger
+
+Intercept your logger messages or log anything with our [Logger](./docs/logger.md) (_work in progress_).
+
+### 5️⃣ Add to CI Pipeline
 
 After you tested reporter locally add it to your CI pipeline.
 
@@ -116,9 +121,14 @@ Bring this reporter on CI and never lose test results again!
 
 * 🛠️ [Frameworks](./docs/frameworks.md)
 * ⛲ [Pipes](./docs/pipes.md)
+  * [Testomat.io](./docs/pipes/testomatio.md)
+  * [GitHub](./docs/pipes/github.md)
+  * [Gitlab](./docs/pipes/gitlab.md)
+  * [CSV](./docs/pipes/csv.md)
 * 📓 [JUnit](./docs/junit.md)
 * 🗄️ [Artifacts](./docs/artifacts.md)
 * 🔂 [Workflows](./docs/workflows.md)
+* 🖊️ [Logger](./docs/logger.md)
 
 ## Development
 

package/lib/{ArtifactStorage.js → _ArtifactStorageOld.js}

@@ -4,27 +4,32 @@ const fs = require('fs');
 const os = require('os');
 const uuid = require('uuid');
 const { TESTOMAT_ARTIFACT_SUFFIX } = require('./constants');
-const { specificTestInfo } = require('./util');
+const { specificTestInfo } = require('./utils/utils');
 
 class ArtifactStorage {
-
+  
   constructor(params) {
-    this._tmpPrefix = "tsmt_reporter";
     this.isFile = false;
     this.isMemory = true;
     this.storage = params?.toFile;
 
     if (this.storage) {
+      // this is fine to use when you start saving artifacts;
+      // but when you need to retrieve them, you will not know if there is "isFile" param,
+      // because you need to create a new instance of ArtifactStorage inside client
+      // so the solution is make it opposite to isMemory (which will be retrieved from global)
       this.isFile = true;
       this.isMemory = false;
-      this.tmpDirFullpath = this.createTestomatTmpDir(this._tmpPrefix);
+      this.tmpDirFullpath = this.createTestomatTmpDir(ArtifactStorage._tmpPrefix);
       debug('SAVE to tmp folder mode enabled!');
     }
   }
 
   static async storeToFile(tmpDirName, artifact, testSuffix = "test") {
+    // this assumes to use multiple files for each test. do we really want this?
     const suffix = TESTOMAT_ARTIFACT_SUFFIX + uuid.v4() + testSuffix;
     const dirpath = join(os.tmpdir(), tmpDirName);
+    // why json?
     const filepath = resolve(dirpath, `${suffix}.json`);
 
     return fs.promises.appendFile(filepath, JSON.stringify(artifact));
@@ -34,6 +39,7 @@ class ArtifactStorage {
     // TODO: mode for different pre-hooks. As variant - "all-tests" || "only_fail"
     // const mode = process.env.TESTOMAT_ARTIFACTS || "only_fail";
 
+    // you save the artifact without specifying its source - test id
     if (Array.isArray(global.testomatioArtifacts)) {
       debug("Saving artifacts to global storage");
 
@@ -71,6 +77,7 @@ class ArtifactStorage {
     return list;
   }
 
+  // returns all the content from all files; do we really need it?
   async tmpContents() {
     const contents = [];
 
@@ -103,8 +110,9 @@ class ArtifactStorage {
     }
   }
 
+  // no need to use multiple dirs; we can implement it later if required
   static tmpTestomatDirNames() {
-    const subname = this._tmpPrefix || "tsmt_reporter";
+    const subname = ArtifactStorage._tmpPrefix || "tsmt_reporter";
 
     return fs.readdirSync(os.tmpdir(), { withFileTypes: true })
       .filter((item) => item.isDirectory())
@@ -113,6 +121,7 @@ class ArtifactStorage {
   }
 
   cleanup() {
+    // when you do a cleanup, I know nothing about isFile param
     if (this.isFile && this.tmpDirFullpath) {
       const tmpDirNames = ArtifactStorage.tmpTestomatDirNames();
 
@@ -128,4 +137,6 @@ class ArtifactStorage {
   }
 }
 
+ArtifactStorage._tmpPrefix = "tsmt_reporter";
+
 module.exports = ArtifactStorage;

package/lib/adapter/codecept.js

@@ -1,9 +1,9 @@
 const debug = require('debug')('@testomatio/reporter:adapter:codeceptjs');
 const chalk = require('chalk');
 const TestomatClient = require('../client');
-const { STATUS, APP_PREFIX } = require('../constants');
+const { STATUS, APP_PREFIX, TESTOMAT_TMP_STORAGE } = require('../constants');
 const upload = require('../fileUploader');
-const Output = require('../output');
+const { parseTest: getIdFromTestTitle, fileSystem } = require('../utils/utils');
 
 if (!global.codeceptjs) {
   // eslint-disable-next-line global-require, import/no-extraneous-dependencies
@@ -16,9 +16,9 @@ let currentMetaStep = [];
 let error;
 let stepShift = 0;
 
-const output = new Output({
-  filterFn: stack => !stack.includes('codeceptjs/lib/output'), // output from codeceptjs
-});
+// const output = new Output({
+//   filterFn: stack => !stack.includes('codeceptjs/lib/output'), // output from codeceptjs
+// });
 
 let stepStart = new Date();
 
@@ -51,24 +51,36 @@ function CodeceptReporter(config) {
 
   recorder.startUnlessRunning();
 
+  global.testomatioRunningEnvironment = 'codeceptjs';
+
   // Listening to events
   event.dispatcher.on(event.all.before, () => {
+    // clear tmp dir
+    fileSystem.clearDir(TESTOMAT_TMP_STORAGE.mainDir);
+
     recorder.add('Creating new run', () => client.createRun());
     videos = [];
     traces = [];
+
+    if (!global.testomatioDataStore) global.testomatioDataStore = {};
   });
 
   event.dispatcher.on(event.test.before, () => {
     recorder.add(() => {
       currentMetaStep = [];
-      output.reset();
-      output.start();
+      // output.reset();
+      // output.start();
       stepShift = 0;
     });
+
+    if (!global.testomatioDataStore) global.testomatioDataStore = {};
+    // reset steps
+    global.testomatioDataStore.steps = [];
   });
 
   event.dispatcher.on(event.test.started, test => {
     testTimeMap[test.id] = Date.now();
+    if (global.testomatioDataStore) global.testomatioDataStore.currentlyRunningTestId = getIdFromTestTitle(test.title);
   });
 
   event.dispatcher.on(event.all.result, async () => {
@@ -97,10 +109,10 @@ function CodeceptReporter(config) {
       suite_title: test.parent && test.parent.title,
       message: testObj.message,
       time: getDuration(test),
-      steps: output.text(),
+      steps: global.testomatioDataStore.steps.join('\n') || null,
       test_id: testId,
     });
-    output.stop();
+    // output.stop();
   });
 
   event.dispatcher.on(event.test.failed, (test, err) => {
@@ -125,7 +137,7 @@ function CodeceptReporter(config) {
         time: 0,
       });
     }
-    output.stop();
+    // output.stop();
   });
 
   event.dispatcher.on(event.test.after, test => {
@@ -141,32 +153,33 @@ function CodeceptReporter(config) {
 
     const files = [];
     if (artifacts.screenshot) files.push({ path: artifacts.screenshot, type: 'image/png' });
-    // todo: video must be uploaded later....    
+    // todo: video must be uploaded later....
 
-    
-    const reportTestPromise = client.addTestRun(STATUS.FAILED, {
-      ...stripExampleFromTitle(title),
-      test_id: testId,
-      suite_title: test.parent && test.parent.title,
-      error,
-      message: testObj.message,
-      time: getDuration(test),
-      files,
-      steps: output.text(),
-    }).then(pipes => {
-      testId = pipes.filter(p => p.pipe.includes('Testomatio'))[0]?.result?.data?.test_id;
-
-      debug("artifacts", artifacts);
-      
-      for (const aid in artifacts) {
-        if (aid.startsWith('video')) videos.push({ testId, title, path: artifacts[aid], type: 'video/webm' });
-        if (aid.startsWith('trace')) traces.push({ testId, title, path: artifacts[aid], type: 'application/zip' });
-      }
-    });
+    const reportTestPromise = client
+      .addTestRun(STATUS.FAILED, {
+        ...stripExampleFromTitle(title),
+        test_id: testId,
+        suite_title: test.parent && test.parent.title,
+        error,
+        message: testObj.message,
+        time: getDuration(test),
+        files,
+        steps: global.testomatioDataStore?.steps?.join('\n') || null,
+      })
+      .then(pipes => {
+        testId = pipes.filter(p => p.pipe.includes('Testomatio'))[0]?.result?.data?.test_id;
+
+        debug('artifacts', artifacts);
+
+        for (const aid in artifacts) {
+          if (aid.startsWith('video')) videos.push({ testId, title, path: artifacts[aid], type: 'video/webm' });
+          if (aid.startsWith('trace')) traces.push({ testId, title, path: artifacts[aid], type: 'application/zip' });
+        }
+      });
 
     reportTestPromises.push(reportTestPromise);
 
-    output.stop();
+    // output.stop();
   });
 
   event.dispatcher.on(event.test.skipped, test => {
@@ -182,7 +195,7 @@ function CodeceptReporter(config) {
       message: testObj.message,
       time: getDuration(test),
     });
-    output.stop();
+    // output.stop();
   });
 
   event.dispatcher.on(event.step.started, step => {
@@ -207,30 +220,37 @@ function CodeceptReporter(config) {
         // eslint-disable-next-line no-continue
         if (!metaSteps[i]) continue;
         if (metaSteps[i].isBDD()) {
-          output.push(repeat(stepShift) + chalk.bold(metaSteps[i].toString()) + metaSteps[i].comment);
+          // output.push(repeat(stepShift) + chalk.bold(metaSteps[i].toString()) + metaSteps[i].comment);
+          global.testomatioDataStore?.steps?.push(
+            repeat(stepShift) + chalk.bold(metaSteps[i].toString()) + metaSteps[i].comment,
+          );
         } else {
-          output.push(repeat(stepShift) + chalk.green.bold(metaSteps[i].toString()));
+          // output.push(repeat(stepShift) + chalk.green.bold(metaSteps[i].toString()));
+          global.testomatioDataStore?.steps?.push(repeat(stepShift) + chalk.green.bold(metaSteps[i].toString()));
         }
       }
     }
     currentMetaStep = metaSteps;
     stepShift = 2 * shift;
 
-    const durationMs = (+new Date() - (+stepStart));
+    const durationMs = +new Date() - (+stepStart);
     let duration = '';
     if (durationMs) {
       duration = repeat(1) + chalk.grey(`(${durationMs}ms)`);
     }
 
     if (step.status === STATUS.FAILED) {
-      output.push(repeat(stepShift) + chalk.red(step.toString()) + duration);
+      // output.push(repeat(stepShift) + chalk.red(step.toString()) + duration);
+      global.testomatioDataStore?.steps?.push(repeat(stepShift) + chalk.red(step.toString()) + duration);
     } else {
-      output.push(repeat(stepShift) + step.toString() + duration);
+      // output.push(repeat(stepShift) + step.toString() + duration);
+      global.testomatioDataStore?.steps?.push(repeat(stepShift) + step.toString() + duration);
     }
   });
 
   event.dispatcher.on(event.step.comment, step => {
-    output.push(chalk.cyan.bold(step.toString()));
+    // output.push(chalk.cyan.bold(step.toString()));
+    global.testomatioDataStore?.steps?.push(chalk.cyan.bold(step.toString()));
   });
 }