reporting-labs 0.5.0 → 0.6.0

package/README.md

@@ -8,6 +8,10 @@ reportingLabs turns a test run into a single HTML file. No server. No upload. No
 
 Today it ships with a **Playwright** reporter. WebdriverIO, Cypress and Jest/Vitest are on the roadmap.
 
+<p align="center"><img src="https://raw.githubusercontent.com/naveenautomationlabs/reporting-labs/main/docs/quickstart.gif" alt="reportingLabs in three steps: install, tag your tests, open one HTML file" width="900"></p>
+
+<p align="center"><em>Install, tag your tests, open one HTML file.</em></p>
+
 <p align="center"><img src="https://raw.githubusercontent.com/naveenautomationlabs/reporting-labs/main/docs/overview.png" alt="Overview page of a reportingLabs report" width="900"></p>
 
 ## Quick start (2 minutes)
@@ -298,6 +302,8 @@ Every option is optional. `npx reporting-labs init` writes them all, with commen
 | `embedAttachments` | `true` | Screenshots inside the HTML (one file) |
 | `embedLimit` | 2 MB | Bigger attachments are copied to `./assets` |
 | `embedVideos` | `false` | Videos inside the HTML too (bigger file, no folder issues) |
+| `emitJson` | `true` | Also write `report.json` alongside `index.html` (used by `merge`) |
+| `jsonFile` | `'report.json'` | File name of the JSON blob |
 | `embedFonts` | `true` | Bundle the fonts (~140 KB) so it looks the same offline |
 | `announce` | `true` | Print the report path after the run |
 | `open` | `'on-failure'` | Open the report in the browser after the run: `'on-failure'`, `'always'` or `'never'`. Never opens in CI |
@@ -326,7 +332,7 @@ Two things to set up:
 1. **Publish the report.** Upload the `reporting-labs/` folder as a build artifact (or archive it in Jenkins). Videos and large files sit in `reporting-labs/assets/`, so keep the folder together.
 2. **Keep the history.** `reporting-labs.history.json` powers the trend and the new vs known failures. On GitHub Actions save it with `actions/cache`. On Jenkins the workspace usually keeps it on its own.
 
-Ready-to-copy samples: [docs/ci/github-actions.yml](https://github.com/naveenautomationlabs/reporting-labs/blob/main/docs/ci/github-actions.yml) and [docs/ci/Jenkinsfile](https://github.com/naveenautomationlabs/reporting-labs/blob/main/docs/ci/Jenkinsfile).
+Ready-to-copy samples: [github-actions.yml](https://github.com/naveenautomationlabs/reporting-labs/blob/main/docs/ci/github-actions.yml), [Jenkinsfile](https://github.com/naveenautomationlabs/reporting-labs/blob/main/docs/ci/Jenkinsfile) and [gitlab-ci.yml](https://github.com/naveenautomationlabs/reporting-labs/blob/main/docs/ci/gitlab-ci.yml). Each one includes an optional Slack (and, for Jenkins, email) step you can uncomment.
 
 ```yaml
 # GitHub Actions, the two steps that matter
@@ -344,6 +350,94 @@ Ready-to-copy samples: [docs/ci/github-actions.yml](https://github.com/naveenaut
 
 **Jenkins note.** Jenkins blocks inline JavaScript by default, so a single-file report shows up blank inside the Jenkins HTML Publisher (Playwright's own HTML report has the same issue). Download the archived artifact and open it locally, or ask an admin to relax the policy in the script console: `System.setProperty("hudson.model.DirectoryBrowserSupport.CSP", "")`.
 
+### Split your run across shards, then merge into one report
+
+Big test suite? Run it faster by splitting it into shards. Each shard is a separate CI job, all running at the same time. When they finish, join them into one report.
+
+**How it works:**
+
+1. Each shard runs your tests. Each one writes its own `reporting-labs/index.html` and a small `report.json` next to it. Save each shard's folder as a CI artifact.
+2. In one last job, download all the shard folders and run one command:
+
+```bash
+npx reporting-labs merge ./all-shards -o merged/
+```
+
+You get one report with:
+
+- All tests from every shard in one list, sorted by priority
+- Combined pass / fail / flaky numbers on top
+- One Trend chart, one Environment card, one Failure Clusters view
+- Screenshots and videos kept in `merged/assets/shard-1-of-4/`, `merged/assets/shard-2-of-4/`, so nothing overwrites
+
+**GitHub Actions example** (replace `npx playwright test` with your own command):
+
+```yaml
+jobs:
+  test:
+    strategy:
+      matrix: { shard: [1, 2, 3, 4] }
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with: { node-version: 20, cache: npm }
+      - run: npm ci
+      - run: npx playwright install --with-deps
+      - run: npx playwright test --shard=${{ matrix.shard }}/4
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: report-shard-${{ matrix.shard }}
+          path: reporting-labs/
+          retention-days: 30
+
+  merge:
+    if: always()
+    needs: test
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with: { node-version: 20, cache: npm }
+      - run: npm ci
+      - uses: actions/download-artifact@v4
+        with: { path: all-shards, pattern: 'report-shard-*' }
+      - run: npx reporting-labs merge all-shards -o merged/
+      - uses: actions/upload-artifact@v4
+        with: { name: merged-report, path: merged/, retention-days: 30 }
+```
+
+**Notes:**
+
+- The reporter always writes `report.json` alongside `index.html`, so `merge` just works. Turn it off with `emitJson: false` if you do not want it.
+- Not using shards? Ignore this section. The single-run report keeps working exactly as before.
+
+### Slack, email, Teams: use your CI's own integration
+
+reportingLabs deliberately does not build its own Slack or email sender. Every CI already has a first-class integration you can lean on, and it stays out of the way of the report itself:
+
+- **GitHub Actions:** `slackapi/slack-github-action` posts a message with the run URL and artifact link. `dawidd6/action-send-mail` handles email. Both are one YAML block, both take a repo secret. See the [github-actions.yml sample](https://github.com/naveenautomationlabs/reporting-labs/blob/main/docs/ci/github-actions.yml).
+- **Jenkins:** the Slack Notification plugin (`slackSend`) and the Email Extension plugin (`emailext`) do the same, plus they can attach `reporting-labs/index.html`. See the [Jenkinsfile sample](https://github.com/naveenautomationlabs/reporting-labs/blob/main/docs/ci/Jenkinsfile).
+- **GitLab CI:** the Slack integration in Project Settings posts pipeline results with no code at all. For a rich message, add a `notify` job with `curl` to your webhook.
+- **CircleCI, Bitbucket, Azure Pipelines:** each has a native Slack orb / task; a plain `curl` to the webhook also works from any shell step.
+
+Two reasons this stays outside the reporter:
+
+1. Your admin most likely already set up notifications for build and deploy. The same channel serves test results with no new moving parts.
+2. Notification delivery (auth, TLS, retries, corporate relays) is a full topic on its own. CI integrations handle it, this reporter stays a single HTML file.
+
+If you need a richer machine-readable summary in the same message, add Playwright's own JSON reporter next to reportingLabs and pipe it to your Slack step:
+
+```ts
+reporter: [
+  ['reporting-labs', require('./reporting-labs.config').default],
+  ['json', { outputFile: 'results.json' }],
+]
+```
+
+Your Slack step can then read `results.json` for pass / fail counts and top failures.
+
 ## Good to know
 
 - **One file.** Screenshots and fonts are inside `index.html`, so it works from an email or a CI artifact. Videos and large files go to `./assets` next to it. Keep the folder together when you share it.

package/bin/cli.js

@@ -5,7 +5,11 @@ const args = process.argv.slice(2);
 const cmd = args[0];
 const has = f => args.includes(f);
 
-if (cmd === 'init') {
+if (cmd === 'merge') {
+  const { merge, parseArgs } = require('./merge');
+  const { dirs, opts } = parseArgs(args.slice(1));
+  merge(dirs, opts);
+} else if (cmd === 'init') {
   const js = has('--js');
   const file = path.resolve(js ? 'reporting-labs.config.js' : 'reporting-labs.config.ts');
   if (fs.existsSync(file) && !has('--force')) {
@@ -37,6 +41,7 @@ if (cmd === 'init') {
   npx reporting-labs init            write reporting-labs.config.ts with every option, commented
   npx reporting-labs init --js       same, as reporting-labs.config.js
   npx reporting-labs init --force    overwrite an existing config file
+  npx reporting-labs merge <dirs>    combine several shard runs into one report (see 'merge --help')
 
 Then in playwright.config.ts:
   import reportingLabs from './reporting-labs.config';

package/bin/config-template.txt

@@ -31,6 +31,7 @@ const config: ReportingLabsOptions = {
   // ── Output ───────────────────────────────────────────────────────────────────
   // outputFolder: 'reporting-labs',                  // where index.html and copied attachments go
   // outputFile: 'index.html',
+  // emitJson: true,                                // also write report.json (used by `reporting-labs merge` for sharded runs)
   // embedAttachments: true,                          // inline screenshots as base64: one file, opens anywhere
   // embedLimit: 2 * 1024 * 1024,                     // attachments bigger than this (bytes) are copied as files
   // embedVideos: false,                              // true = videos inside the HTML too (bigger file, no folder issues)

package/bin/merge.js

@@ -0,0 +1,173 @@
+// Merge several shard runs into one report.
+// Usage: node bin/merge.js <shardDir1> <shardDir2> ... [-o <outDir>]
+//        (invoked by bin/cli.js as `npx reporting-labs merge ...`)
+
+'use strict';
+const fs = require('fs');
+const path = require('path');
+
+function usage(exit = 0) {
+  console.log(
+`reporting-labs merge
+
+  Combine several shard runs into one report.
+
+Usage:
+  npx reporting-labs merge <shard-dir> [<shard-dir> ...] [--out <dir>] [--file <name>]
+
+Arguments:
+  <shard-dir>          A folder that contains report.json (the shard's reporting-labs output).
+                       You can also pass a parent folder that contains many shards, e.g.
+                       "npx reporting-labs merge ./all-shards" where each subfolder is a shard.
+
+Options:
+  --out, -o <dir>      Where the merged report is written. Default: reporting-labs-merged
+  --file <name>        Output HTML file name. Default: index.html
+  --title <text>       Override the report title. Default: the first shard's title
+  --help, -h           Show this message
+
+Example (GitHub Actions):
+  # each shard uploads its own reporting-labs/ folder as an artifact,
+  # a final job downloads them all under ./all-shards/, then:
+  npx reporting-labs merge ./all-shards -o merged/`);
+  process.exit(exit);
+}
+
+function parseArgs(argv) {
+  const dirs = [];
+  const opts = { out: 'reporting-labs-merged', file: 'index.html', title: undefined };
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i];
+    if (a === '--help' || a === '-h') usage(0);
+    else if (a === '--out' || a === '-o') opts.out = argv[++i];
+    else if (a === '--file') opts.file = argv[++i];
+    else if (a === '--title') opts.title = argv[++i];
+    else if (a.startsWith('-')) { console.error('unknown flag: ' + a); usage(1); }
+    else dirs.push(a);
+  }
+  if (!dirs.length) usage(1);
+  return { dirs, opts };
+}
+
+/** Expand each arg: if the arg is a shard dir (has report.json) keep it; otherwise treat as a parent and expand its immediate subdirs that have report.json. */
+function resolveShardDirs(inputs) {
+  const out = [];
+  for (const p of inputs) {
+    const abs = path.resolve(p);
+    if (!fs.existsSync(abs)) { console.error('missing:', abs); process.exit(1); }
+    if (fs.existsSync(path.join(abs, 'report.json'))) out.push(abs);
+    else {
+      const kids = fs.readdirSync(abs).map(k => path.join(abs, k)).filter(k => fs.statSync(k).isDirectory() && fs.existsSync(path.join(k, 'report.json')));
+      if (!kids.length) { console.error(`no report.json under ${abs}, nor in any subfolder`); process.exit(1); }
+      out.push(...kids);
+    }
+  }
+  return out;
+}
+
+function rewriteAttachmentPath(a, prefix) {
+  if (!a) return a;
+  // The reporter emits 'src' pointing at 'assets/<name>' for file-backed attachments;
+  // namespace it under the shard's folder so multiple shards can share one merged assets/.
+  const src = a.src;
+  if (typeof src === 'string' && src.startsWith('assets/')) return { ...a, src: 'assets/' + prefix + '/' + src.slice('assets/'.length) };
+  return a;
+}
+
+function copyDir(from, to) {
+  if (!fs.existsSync(from)) return;
+  fs.mkdirSync(to, { recursive: true });
+  for (const entry of fs.readdirSync(from, { withFileTypes: true })) {
+    const src = path.join(from, entry.name), dst = path.join(to, entry.name);
+    if (entry.isDirectory()) copyDir(src, dst);
+    else fs.copyFileSync(src, dst);
+  }
+}
+
+function shardLabel(dir, i, data) {
+  if (data && data.shard && data.shard.total) return `shard-${data.shard.current}-of-${data.shard.total}`;
+  return `shard-${i + 1}`;
+}
+
+function merge(inputs, opts) {
+  const dirs = resolveShardDirs(inputs);
+  console.log(`Merging ${dirs.length} shard${dirs.length === 1 ? '' : 's'} into ${opts.out}`);
+
+  const shards = dirs.map((d, i) => {
+    const data = JSON.parse(fs.readFileSync(path.join(d, 'report.json'), 'utf8'));
+    return { dir: d, label: shardLabel(d, i, data), data };
+  });
+
+  const first = shards[0].data;
+  // Compose combined data. Start from the first shard's options/env, then merge in test rows.
+  const combined = {
+    ...first,
+    title: opts.title || first.title,
+    startTime: Math.min(...shards.map(s => s.data.startTime || 0)),
+    duration: Math.max(...shards.map(s => s.data.duration || 0)),
+    generatedAt: Date.now(),
+    stats: { passed: 0, failed: 0, skipped: 0, flaky: 0, timedOut: 0, interrupted: 0, total: 0 },
+    tests: [],
+    globalErrors: [],
+    globalOutput: [],
+    projects: [],
+    workers: shards.reduce((n, s) => n + (s.data.workers || 0), 0),
+    shard: undefined,   // no longer sharded
+    env: [...(first.env || [])],
+    history: first.history || [],   // all shards share the same history file
+  };
+
+  const projSet = new Set();
+  for (const s of shards) {
+    const stats = s.data.stats || {};
+    for (const k of Object.keys(combined.stats)) combined.stats[k] += stats[k] || 0;
+    (s.data.projects || []).forEach(p => projSet.add(p));
+
+    for (const t of (s.data.tests || [])) {
+      const nt = {
+        ...t,
+        results: (t.results || []).map(r => ({
+          ...r,
+          attachments: (r.attachments || []).map(a => rewriteAttachmentPath(a, s.label)),
+        })),
+      };
+      combined.tests.push(nt);
+    }
+    for (const e of (s.data.globalErrors || [])) combined.globalErrors.push(e);
+    for (const o of (s.data.globalOutput || [])) combined.globalOutput.push(o);
+  }
+  combined.projects = [...projSet].sort();
+  combined.env.push({ k: 'Shards', v: shards.map(s => s.label).join(', ') });
+  if (combined.runStatus === undefined || combined.runStatus === 'passed') {
+    const anyFail = shards.some(s => s.data.runStatus && s.data.runStatus !== 'passed');
+    if (anyFail) combined.runStatus = 'failed';
+  }
+
+  // Prepare output folder
+  fs.mkdirSync(opts.out, { recursive: true });
+  const assetsRoot = path.join(opts.out, 'assets');
+  fs.rmSync(assetsRoot, { recursive: true, force: true });
+
+  for (const s of shards) {
+    const src = path.join(s.dir, 'assets');
+    if (fs.existsSync(src)) copyDir(src, path.join(assetsRoot, s.label));
+  }
+
+  // Render combined HTML using the reporter's own template.
+  const { renderHtml } = require(path.resolve(__dirname, '..', 'dist', 'template.js'));
+  const outFile = path.join(opts.out, opts.file);
+  fs.writeFileSync(outFile, renderHtml(combined), 'utf8');
+  fs.writeFileSync(path.join(opts.out, 'report.json'), JSON.stringify(combined), 'utf8');
+
+  const st = combined.stats;
+  const rel = path.relative(process.cwd(), outFile);
+  console.log(`  merged: ${st.total} tests · ${st.passed} passed · ${st.failed + st.timedOut + st.interrupted} failed · ${st.flaky} flaky · ${st.skipped} skipped`);
+  console.log(`  written to ${rel}`);
+}
+
+module.exports = { merge, parseArgs };
+
+if (require.main === module) {
+  const { dirs, opts } = parseArgs(process.argv.slice(2));
+  merge(dirs, opts);
+}

package/dist/reporter.d.ts

@@ -24,7 +24,8 @@ export default class ReportingLabsReporter implements Reporter {
     private printMissingMeta;
     /** Open the report in the default browser, like Playwright's HTML reporter. Never in CI. */
     private maybeOpen;
-    /** Videos and traces are finalized asynchronously by the runner; wait until every file-backed attachment stops growing. */
+    /** Videos and traces are finalized asynchronously by the runner; wait until every file-backed attachment stops growing.
+     *  All files are polled in parallel so total wait is bounded by the slowest single file, not the sum. */
     private settleAttachmentFiles;
     /** Runtime facts for the Environment card: Playwright, Node, OS, browsers, CI job, git commit. */
     private collectEnv;

package/dist/reporter.js

@@ -202,6 +202,10 @@ class ReportingLabsReporter {
         };
         const file = path.join(this.outDir, this.options.outputFile ?? 'index.html');
         fs.writeFileSync(file, (0, template_1.renderHtml)(data), 'utf8');
+        if (this.options.emitJson !== false) {
+            const jsonFile = path.join(this.outDir, this.options.jsonFile ?? 'report.json');
+            fs.writeFileSync(jsonFile, JSON.stringify(data), 'utf8');
+        }
         if (this.options.announce !== false) {
             const rel = path.relative(process.cwd(), file);
             console.log(`\n  reporting-labs: report written to ${rel}`);
@@ -246,7 +250,8 @@ class ReportingLabsReporter {
         catch { /* ignore */ }
     }
     // ---- helpers -------------------------------------------------------------
-    /** Videos and traces are finalized asynchronously by the runner; wait until every file-backed attachment stops growing. */
+    /** Videos and traces are finalized asynchronously by the runner; wait until every file-backed attachment stops growing.
+     *  All files are polled in parallel so total wait is bounded by the slowest single file, not the sum. */
     async settleAttachmentFiles() {
         const paths = new Set();
         for (const test of this.suite.allTests())
@@ -254,6 +259,8 @@ class ReportingLabsReporter {
                 for (const a of r.attachments)
                     if (a.path)
                         paths.add(a.path);
+        if (!paths.size)
+            return;
         const size = (p) => { try {
             return fs.statSync(p).size;
         }
@@ -261,18 +268,19 @@ class ReportingLabsReporter {
             return -1;
         } };
         const sleep = (ms) => new Promise(r => setTimeout(r, ms));
-        for (const p of paths) {
+        const wait = async (p) => {
             let last = size(p);
+            if (last < 0)
+                return; // never appeared; nothing to wait for
             for (let i = 0; i < 20; i++) { // up to ~5 s per file
-                if (last < 0)
-                    break; // not on disk (yet); nothing to wait for
                 await sleep(250);
                 const now = size(p);
                 if (now === last && now > 0)
-                    break;
+                    return; // stable and non-empty: settled
                 last = now;
             }
-        }
+        };
+        await Promise.all([...paths].map(wait));
     }
     /** Runtime facts for the Environment card: Playwright, Node, OS, browsers, CI job, git commit. */
     collectEnv(base) {

package/dist/types.d.ts

@@ -13,6 +13,10 @@ export interface ReportingLabsOptions {
     outputFolder?: string;
     /** Output HTML file name inside outputFolder. Default: "index.html" */
     outputFile?: string;
+    /** Also write a machine-readable `report.json` alongside `index.html`. Used by `reporting-labs merge` to combine shard runs. Default: true. */
+    emitJson?: boolean;
+    /** File name of the JSON blob inside outputFolder. Default: "report.json" */
+    jsonFile?: string;
     /** Inline screenshots as base64 (single file, opens anywhere). Default: true */
     embedAttachments?: boolean;
     /** Max size (bytes) of a single attachment to embed. Larger ones are copied as files. Default: 2 MB */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "reporting-labs",
-  "version": "0.5.0",
+  "version": "0.6.0",
   "description": "reportingLabs – beautiful, customizable single-file HTML test reports. One line of config. Playwright adapter included, more runners on the way.",
   "keywords": [
     "test-report",