@testomatio/reporter 1.2.1-beta → 1.2.1-beta.codecept-id

package/README.md

@@ -1,6 +1,5 @@
 # Testomatio Reporter
 
-
 👋 Hey, do you need some test reporting?
 
 Testomat.io Reporter is a library that integrates with popular **JavaScript and TypeScript** test frameworks to provide a common interface for test reporting. By default, Testomat.io Reporter works with our reporting cloud service [Testomat.io App](https://testomat.io), however it is not locked to it. Reporter can be used as a standalone tool.
@@ -9,18 +8,17 @@ Testomat.io Reporter is a library that integrates with popular **JavaScript and
 
 Testomat.io Reporter (this npm package) supports:
 
-* 🏄 Integarion with all popular [JavaScript/TypeScript frameworks](./docs/frameworks.md)
-* 🗄️ Screenshots, videos, traces [uploaded into S3 bucket](./docs/artifacts.md)
-* 🔎 Stack traces and error messages
-* 🐙 [GitHub](./docs/pipes.md#github-pipe) & [GitLab](./docs/pipes.md#gitlab-pipe) integration
-* 🚅 Realtime reports
-* 🗃️ Other test frameworks supported via [JUNit XML](./docs/junit.md)
-* 🚶‍♀️ Steps *(work in progress)*
-* 📄 Logger *(work in progress, supports Jest for now)*
-* ☁️ Custom properties and metadata *(work in progress)*
-* 💯 Free & open-source.
-* 📊 Public and private Run reports on cloud via [Testomat.io App](https://testomat.io) 👇
-
+- 🏄 Integarion with all popular [JavaScript/TypeScript frameworks](./docs/frameworks.md)
+- 🗄️ Screenshots, videos, traces [uploaded into S3 bucket](./docs/artifacts.md)
+- 🔎 [Stack traces](./docs/stacktrace.md) and error messages
+- 🐙 [GitHub](./docs/pipes/github.md) & [GitLab](./docs/pipes/gitlab.md) integration
+- 🚅 Realtime reports
+- 🗃️ Other test frameworks supported via [JUNit XML](./docs/junit.md)
+- 🚶‍♀️ Steps _(work in progress)_
+- 📄 [Logger](./docs/logger.md) _(work in progress, supports Jest for now)_
+- ☁️ Custom properties and metadata _(work in progress)_
+- 💯 Free & open-source.
+- 📊 Public and private Run reports on cloud via [Testomat.io App](https://testomat.io) 👇
 
 ![](./docs/images/app.png)
 
@@ -29,12 +27,12 @@ Testomat.io Reporter (this npm package) supports:
 Testomat.io Reporter provides common API to store and organize test reports.
 It can receive test result data from any [test framework](./docs/frameworks.md) and send it to different services via [pipes](./docs/pipes.md).
 
-| 🌊 Input         | 📊 Output                                 |
-|---------------|----------------------------------------|
-| Playwright    | Report to GitHub                       |
-| Cypress       | Report to GitLab                       |
-| Jest          | Report to [Testomat.io](https://testomat.io) |
-| ...   | ... your custom report   |
+| 🌊 Input   | 📊 Output                                    |
+| ---------- | -------------------------------------------- |
+| Playwright | Report to GitHub                             |
+| Cypress    | Report to GitLab                             |
+| Jest       | Report to [Testomat.io](https://testomat.io) |
+| ...        | ... your custom report                       |
 
 If you use multiple test frameworks and you need to use one customizable reporter, check Testomat.io Reporter, as you can adjust it once and attach it to all your projects.
 
@@ -46,7 +44,6 @@ Artifacts like screenshots, videos, traces, are **uploaded to your own cloud sto
 
 To enable Testomat.io Reporter install `@testomatio/reporter` package
 
-
 Use one of your favorite package managers:
 
 ```
@@ -65,46 +62,52 @@ yarn add @testomatio/reporter --dev
 
 ### 1️⃣ Attach Reporter to the Test Runner
 
-* #### [Playwright](./docs/frameworks.md#playwright)
-* #### [CodeceptJS](./docs/frameworks.md#CodeceptJS)
-* #### [Cypress](./docs/frameworks.md#Cypress)
-* #### [Jest](./docs/frameworks.md#Jest)
-* #### [Mocha](./docs/frameworks.md#Mocha)
-* #### [WebDriverIO](./docs/frameworks.md#WebDriverIO)
-* #### [TestCafe](./docs/frameworks.md#TestCafe)
-* #### [Detox](./docs/frameworks.md#Detox)
-* #### [Codeception](https://github.com/testomatio/php-reporter)
-* #### [Newman (Postman)](./docs/frameworks.md#Newman)
-* #### [JUnit](./docs/junit.md#junit)
-* #### [NUnit](./docs/junit.md#nunit)
-* #### [PyTest](./docs/junit.md#pytest)
-* #### [PHPUnit](./docs/junit.md#phpunit)
-* #### [Protractor](./docs/frameworks.md#protractor)
-
-or any [other via JUnit](./docs/junit.md) report....
+|                                                 |                                               |                                                           |
+| ----------------------------------------------- | --------------------------------------------- | --------------------------------------------------------- |
+| [Playwright](./docs/frameworks.md#playwright)   | [CodeceptJS](./docs/frameworks.md#CodeceptJS) | [Cypress](./docs/frameworks.md#Cypress)                   |
+| [Jest](./docs/frameworks.md#Jest)               | [Mocha](./docs/frameworks.md#Mocha)           | [WebDriverIO](./docs/frameworks.md#WebDriverIO)           |
+| [TestCafe](./docs/frameworks.md#TestCafe)       | [Detox](./docs/frameworks.md#Detox)           | [Codeception](https://github.com/testomatio/php-reporter) |
+| [Newman (Postman)](./docs/frameworks.md#Newman) | [JUnit](./docs/junit.md#junit)                | [NUnit](./docs/junit.md#nunit)                            |
+| [PyTest](./docs/junit.md#pytest)                | [PHPUnit](./docs/junit.md#phpunit)            | [Protractor](./docs/frameworks.md#protractor)             |
 
-### 2️⃣ Configure Reports
+or **any [other via JUnit](./docs/junit.md)** report....
 
-* [Create report on Testomat.io](./docs/pipes.md#testomatio-pipe).
-* [Create brief summary report for GitHub Pull Request](./docs/pipes.md#github-pipe) 👇
-* [Create brief summary report for GitLab Merge Request](./docs/pipes.md#gitlab-pipe).
-* [Configure other pipes](./docs/pipes.md) for other ways to process test results output.
+### 2️⃣ Configure Reports
 
+- [Create report on Testomat.io](./docs/pipes/testomatio.md).
+- [Create brief summary report for GitHub Pull Request](./docs/pipes/github.md) 👇
+- [Create brief summary report for GitLab Merge Request](./docs/pipes/gitlab.md).
+- [Configure other pipes](./docs/pipes/md) for other ways to process test results output.
 
-![GitHub report](./docs/images/github.png)
+![GitHub report](./docs/pipes/images/github.png)
 
 GitHub report published as a comment to Pull Request:
 
 ### 3️⃣ Enable Artifacts Storage
 
-1. Create bucket on AWS, Google Cloud, or any other cloud storage provider supporting S3 protocol. 
+1. Create bucket on AWS, Google Cloud, or any other cloud storage provider supporting S3 protocol.
 2. [Pass S3 credentials](./docs/artifacts.md) to reporter to enable artifacts uploading.
 
 ### 4️⃣ Use Logger
 
 Intercept your logger messages or log anything with our [Logger](./docs/logger.md) (_work in progress_).
 
-### 5️⃣ Add to CI Pipeline
+### 5️⃣ TESTOMATIO HTML Report
+
+Testomatio now features a custom reporting tool that allows you to generate a standalone HTML/CSS report for easy visualization of your test runs. This mode provides a quick way to gain a clear and visually appealing overview of your test execution.
+
+With our reporter, you can:
+
+- Easily analyze the results of your test runs in a convenient HTML format.
+
+* Visualize data on successful and failed tests, including statistics and error details.
+* Quickly share reports with your team members or stakeholders.
+
+![HTML report](./docs/images/html_reporter_example.gif)
+
+Learn more about generating HTML reports [here](./docs/pipes/html.md)
+
+### 6️⃣ Add to CI Pipeline
 
 After you tested reporter locally add it to your CI pipeline.
 
@@ -116,20 +119,23 @@ After you tested reporter locally add it to your CI pipeline.
 
 Bring this reporter on CI and never lose test results again!
 
-
 ## Documentation
 
-* 🛠️ [Frameworks](./docs/frameworks.md)
-* ⛲ [Pipes](./docs/pipes.md)
-* 📓 [JUnit](./docs/junit.md)
-* 🗄️ [Artifacts](./docs/artifacts.md)
-* 🔂 [Workflows](./docs/workflows.md)
-* 🔂 [Logger](./docs/logger.md)
+- 🛠️ [Frameworks](./docs/frameworks.md)
+- ⛲ [Pipes](./docs/pipes.md)
+  - [Testomat.io](./docs/pipes/testomatio.md)
+  - [GitHub](./docs/pipes/github.md)
+  - [Gitlab](./docs/pipes/gitlab.md)
+  - [CSV](./docs/pipes/csv.md)
+  - [HTML report](./docs/pipes/html.md)
+- 📓 [JUnit](./docs/junit.md)
+- 🗄️ [Artifacts](./docs/artifacts.md)
+- 🔂 [Workflows](./docs/workflows.md)
+- 🖊️ [Logger](./docs/logger.md)
 
 ## Development
 
-
-### REST API 
+### REST API
 
 Testomat.io App uses REST API to collect data from the reporter.
 
@@ -144,6 +150,7 @@ To print all reporter logs:
 ```
 DEBUG=@testomatio/reporter:*
 ```
+
 To print all reporter logs of a specific pipe:
 
 ```

package/lib/adapter/codecept.js

@@ -1,9 +1,10 @@
 const debug = require('debug')('@testomatio/reporter:adapter:codeceptjs');
 const chalk = require('chalk');
 const TestomatClient = require('../client');
-const { STATUS, APP_PREFIX } = require('../constants');
+const { STATUS, APP_PREFIX, TESTOMAT_TMP_STORAGE_DIR } = require('../constants');
 const upload = require('../fileUploader');
-const Output = require('../output');
+const { parseTest, fileSystem } = require('../utils/utils');
+const { services } = require('../services');
 
 if (!global.codeceptjs) {
   // eslint-disable-next-line global-require, import/no-extraneous-dependencies
@@ -16,9 +17,9 @@ let currentMetaStep = [];
 let error;
 let stepShift = 0;
 
-const output = new Output({
-  filterFn: stack => !stack.includes('codeceptjs/lib/output'), // output from codeceptjs
-});
+// const output = new Output({
+//   filterFn: stack => !stack.includes('codeceptjs/lib/output'), // output from codeceptjs
+// });
 
 let stepStart = new Date();
 
@@ -53,22 +54,73 @@ function CodeceptReporter(config) {
 
   // Listening to events
   event.dispatcher.on(event.all.before, () => {
-    recorder.add('Creating new run', () => client.createRun());
+    // clear tmp dir
+    fileSystem.clearDir(TESTOMAT_TMP_STORAGE_DIR);
+
+    // recorder.add('Creating new run', () => );
+    client.createRun();
     videos = [];
     traces = [];
+
+    if (!global.testomatioDataStore) global.testomatioDataStore = {};
+  });
+
+  let hookSteps = [];
+  let suiteHookRunning = false;
+
+  event.dispatcher.on(event.suite.before, suite => {
+    suiteHookRunning = true;
+    hookSteps = [];
+    global.testomatioDataStore.steps = [];
+
+    services.setContext(suite.fullTitle());
+  });
+
+  event.dispatcher.on(event.suite.after, () => {
+    services.setContext(null);
+  });
+
+  event.dispatcher.on(event.hook.started, () => {
+    // global.testomatioDataStore.steps = [];
+  });
+
+  event.dispatcher.on(event.hook.passed, () => {
+    if (suiteHookRunning) {
+      hookSteps.push(...global.testomatioDataStore.steps);
+      services.setContext(null);
+    }
+  });
+
+  event.dispatcher.on(event.hook.failed, () => {
+    if (suiteHookRunning) {
+      hookSteps.push(...global.testomatioDataStore.steps);
+      services.setContext(null);
+    }
   });
 
-  event.dispatcher.on(event.test.before, () => {
+  event.dispatcher.on(event.test.before, test => {
+    suiteHookRunning = false;
+    global.testomatioDataStore.steps = [];
+
     recorder.add(() => {
       currentMetaStep = [];
-      output.reset();
-      output.start();
+      // output.reset();
+      // output.start();
       stepShift = 0;
     });
+
+    if (!global.testomatioDataStore) global.testomatioDataStore = {};
+    // reset steps
+    global.testomatioDataStore.steps = [];
+
+    services.setContext(test.fullTitle());
   });
 
   event.dispatcher.on(event.test.started, test => {
+    services.setContext(test.fullTitle());
+
     testTimeMap[test.id] = Date.now();
+    // start logging
   });
 
   event.dispatcher.on(event.all.result, async () => {
@@ -92,15 +144,24 @@ function CodeceptReporter(config) {
     }
     const testId = parseTest(tags);
     const testObj = getTestAndMessage(title);
+
+    const logs = getTestLogs(test);
+    const manuallyAttachedArtifacts = services.artifacts.get(test.fullTitle());
+    const keyValues = services.keyValues.get(test.fullTitle());
+    services.setContext(null);
+
     client.addTestRun(STATUS.PASSED, {
       ...stripExampleFromTitle(title),
       suite_title: test.parent && test.parent.title,
       message: testObj.message,
       time: getDuration(test),
-      steps: output.text(),
+      steps: global.testomatioDataStore.steps.join('\n') || null,
       test_id: testId,
+      logs,
+      manuallyAttachedArtifacts,
+      meta: keyValues,
     });
-    output.stop();
+    // output.stop();
   });
 
   event.dispatcher.on(event.test.failed, (test, err) => {
@@ -125,7 +186,7 @@ function CodeceptReporter(config) {
         time: 0,
       });
     }
-    output.stop();
+    // output.stop();
   });
 
   event.dispatcher.on(event.test.after, test => {
@@ -135,38 +196,44 @@ function CodeceptReporter(config) {
     failedTests.push(id || title);
     let testId = parseTest(tags);
     const testObj = getTestAndMessage(title);
-    if (error && error.stack && test.steps && test.steps.length) {
-      error.stack = test.steps[test.steps.length - 1].line();
-    }
 
     const files = [];
     if (artifacts.screenshot) files.push({ path: artifacts.screenshot, type: 'image/png' });
-    // todo: video must be uploaded later....    
+    // todo: video must be uploaded later....
 
-    
-    const reportTestPromise = client.addTestRun(STATUS.FAILED, {
-      ...stripExampleFromTitle(title),
-      test_id: testId,
-      suite_title: test.parent && test.parent.title,
-      error,
-      message: testObj.message,
-      time: getDuration(test),
-      files,
-      steps: output.text(),
-    }).then(pipes => {
-      testId = pipes.filter(p => p.pipe.includes('Testomatio'))[0]?.result?.data?.test_id;
-
-      debug("artifacts", artifacts);
-      
-      for (const aid in artifacts) {
-        if (aid.startsWith('video')) videos.push({ testId, title, path: artifacts[aid], type: 'video/webm' });
-        if (aid.startsWith('trace')) traces.push({ testId, title, path: artifacts[aid], type: 'application/zip' });
-      }
-    });
+    const logs = getTestLogs(test);
+    const manuallyAttachedArtifacts = services.artifacts.get(test.fullTitle());
+    const keyValues = services.keyValues.get(test.fullTitle());
+    services.setContext(null);
+
+    const reportTestPromise = client
+      .addTestRun(STATUS.FAILED, {
+        ...stripExampleFromTitle(title),
+        test_id: testId,
+        suite_title: test.parent && test.parent.title,
+        error,
+        message: testObj.message,
+        time: getDuration(test),
+        files,
+        steps: global.testomatioDataStore?.steps?.join('\n') || null,
+        logs,
+        manuallyAttachedArtifacts,
+        meta: keyValues,
+      })
+      .then(pipes => {
+        testId = pipes.filter(p => p.pipe.includes('Testomatio'))[0]?.result?.data?.test_id;
+
+        debug('artifacts', artifacts);
+
+        for (const aid in artifacts) {
+          if (aid.startsWith('video')) videos.push({ testId, title, path: artifacts[aid], type: 'video/webm' });
+          if (aid.startsWith('trace')) traces.push({ testId, title, path: artifacts[aid], type: 'application/zip' });
+        }
+      });
 
     reportTestPromises.push(reportTestPromise);
 
-    output.stop();
+    // output.stop();
   });
 
   event.dispatcher.on(event.test.skipped, test => {
@@ -182,7 +249,7 @@ function CodeceptReporter(config) {
       message: testObj.message,
       time: getDuration(test),
     });
-    output.stop();
+    // output.stop();
   });
 
   event.dispatcher.on(event.step.started, step => {
@@ -207,30 +274,37 @@ function CodeceptReporter(config) {
         // eslint-disable-next-line no-continue
         if (!metaSteps[i]) continue;
         if (metaSteps[i].isBDD()) {
-          output.push(repeat(stepShift) + chalk.bold(metaSteps[i].toString()) + metaSteps[i].comment);
+          // output.push(repeat(stepShift) + chalk.bold(metaSteps[i].toString()) + metaSteps[i].comment);
+          global.testomatioDataStore?.steps?.push(
+            repeat(stepShift) + chalk.bold(metaSteps[i].toString()) + metaSteps[i].comment,
+          );
         } else {
-          output.push(repeat(stepShift) + chalk.green.bold(metaSteps[i].toString()));
+          // output.push(repeat(stepShift) + chalk.green.bold(metaSteps[i].toString()));
+          global.testomatioDataStore?.steps?.push(repeat(stepShift) + chalk.green.bold(metaSteps[i].toString()));
         }
       }
     }
     currentMetaStep = metaSteps;
     stepShift = 2 * shift;
 
-    const durationMs = (+new Date() - (+stepStart));
+    const durationMs = +new Date() - +stepStart;
     let duration = '';
     if (durationMs) {
       duration = repeat(1) + chalk.grey(`(${durationMs}ms)`);
     }
 
     if (step.status === STATUS.FAILED) {
-      output.push(repeat(stepShift) + chalk.red(step.toString()) + duration);
+      // output.push(repeat(stepShift) + chalk.red(step.toString()) + duration);
+      global.testomatioDataStore?.steps?.push(repeat(stepShift) + chalk.red(step.toString()) + duration);
     } else {
-      output.push(repeat(stepShift) + step.toString() + duration);
+      // output.push(repeat(stepShift) + step.toString() + duration);
+      global.testomatioDataStore?.steps?.push(repeat(stepShift) + step.toString() + duration);
     }
   });
 
   event.dispatcher.on(event.step.comment, step => {
-    output.push(chalk.cyan.bold(step.toString()));
+    // output.push(chalk.cyan.bold(step.toString()));
+    global.testomatioDataStore?.steps?.push(chalk.cyan.bold(step.toString()));
   });
 }
 
@@ -238,7 +312,7 @@ async function uploadAttachments(client, attachments, messagePrefix, attachmentT
   if (attachments.length > 0) {
     console.log(APP_PREFIX, `Attachments: ${messagePrefix} ${attachments.length} ${attachmentType}/-s ...`);
 
-    const promises = attachments.map(async (attachment) => {
+    const promises = attachments.map(async attachment => {
       const { testId, title, path, type } = attachment;
       const file = { path, type, title };
       return client.addTestRun(undefined, {
@@ -252,18 +326,6 @@ async function uploadAttachments(client, attachments, messagePrefix, attachmentT
   }
 }
 
-function parseTest(tags) {
-  if (tags) {
-    for (const tag of tags) {
-      if (tag.startsWith('@T')) {
-        return tag.substring(2, tag.length);
-      }
-    }
-  }
-
-  return null;
-}
-
 function getTestAndMessage(title) {
   const testObj = { message: '' };
   const testArr = title.split(/\s(\|\s\{.*?\})/);
@@ -286,4 +348,21 @@ function repeat(num) {
   return ''.padStart(num, ' ');
 }
 
+// TODO: think about moving to some common utils
+function getTestLogs(test) {
+  const suiteLogsArr = services.logger.getLogs(test.parent.fullTitle());
+  const suiteLogs = suiteLogsArr ? suiteLogsArr.join('\n').trim() : '';
+  const testLogsArr = services.logger.getLogs(test.fullTitle());
+  const testLogs = testLogsArr ? testLogsArr.join('\n').trim() : '';
+
+  let logs = '';
+  if (suiteLogs) {
+    logs += `${chalk.bold('\t--- BeforeSuite ---')}\n${suiteLogs}`;
+  }
+  if (testLogs) {
+    logs += `\n${chalk.bold('\t--- Test ---')}\n${testLogs}`;
+  }
+  return logs;
+}
+
 module.exports = CodeceptReporter;