reporting-labs 0.4.1 → 0.4.3

package/README.md

@@ -53,6 +53,17 @@ Done. Everything below is optional.
 
 > **In a hurry?** Skip the config file and pass options inline: `reporter: [['reporting-labs', { title: 'My app' }]]`.
 
+## See everything the report can do
+
+Every card and tab is switched on. What you see depends on what your tests give it:
+
+1. **Add `meta()` to your tests** and the report ranks failures by priority, groups them by owner and feature, and links to your stories. See [the helpers](#add-details-to-your-tests-3-small-helpers).
+2. **Run the suite twice** and the Trend chart, "new vs known" failures, flaky dots and "Got slower" appear. History is kept in `reporting-labs.history.json`.
+3. **Turn on Playwright's `screenshot`, `video` and `trace`** in `playwright.config.ts` and they show up on every failed test.
+4. **Have more than one Playwright project** (say chromium and webkit) and you get a feature × project heatmap.
+
+The report tells you about these itself: a card that has nothing to show yet explains what to do.
+
 ## Add details to your tests (3 small helpers)
 
 The report already shows steps, screenshots, videos, traces and errors on its own. Three small helpers add the rest. Import them from `reporting-labs` and call them inside a test.
@@ -73,6 +84,7 @@ One line per test. With this the report can rank failures by priority, group the
 - Known keys: `priority`, `severity`, `owner`, `feature`, `epic`, `story`, `issue`, `component`, `team`. Any other key you pass is shown too.
 - Your Playwright tags like `@sanity` or `@regression` stay as they are and still show on the test.
 - To make story and epic keys clickable, set `links` in the config: `links: { story: 'https://yourteam.atlassian.net/browse/{id}' }`.
+- `priority`, `severity`, `feature` and `owner` each get a tab in the Breakdown chart and a filter on the Tests page. Want the same for your own key, say `meta({ team: 'web' })`? Add it to `dimensions` in the config: `dimensions: ['priority', 'severity', 'feature', 'owner', 'team']`.
 
 ### `log()`: a line in the report
 
@@ -219,6 +231,42 @@ Every outcome Playwright can produce, not just pass and fail:
 - interrupted runs show a banner with how many tests did not finish
 - the Environment card shows Playwright and Node versions, OS, browsers, workers, shard, the CI job link and the git commit
 
+## How it compares
+
+Legend: ✅ built in · 🟡 possible with manual setup or extra config · ❌ not in the official docs
+
+| Feature | Playwright HTML | Allure | reportingLabs |
+|---|:---:|:---:|:---:|
+| Single HTML file, opens without a server | ❌ folder, served by `show-report` | 🟡 single-file mode (2.24+, Allure 3 plugin), otherwise `allure open` | ✅ |
+| No extra tooling | ✅ | ❌ CLI needed; Allure 2 needs Java | ✅ npm package only |
+| Setup | ✅ built in | 🟡 reporter + generate step | ✅ one line in `reporter:` |
+| Steps, screenshots, videos, traces | ✅ | ✅ | ✅ |
+| Visual comparison viewer (slider, side by side, diff) | ❌ | ❌ | ✅ |
+| Tags and annotations shown | ✅ | ✅ labels via runtime API | ✅ `meta()` + tags |
+| Priority / severity / owner / feature / epic / story | 🟡 custom annotations | ✅ labels | ✅ |
+| Links to Jira, TMS | 🟡 annotation with a URL | ✅ `issue()`, `tms()` | ✅ `links` templates |
+| Failures ranked by priority and severity | ❌ | ❌ | ✅ Needs attention |
+| Failures by owner | ❌ | 🟡 owner label, no rollup documented | ✅ |
+| History across runs | ❌ | 🟡 History Trend when historical data is accumulated | ✅ history file, zero setup |
+| New vs known failures ("failing since #1840") | ❌ | ❌ | ✅ |
+| Flaky detection | ✅ `flaky` outcome per run | 🟡 `@Flaky` annotation (Java) | ✅ outcome + flakiest over last runs |
+| Got slower vs last run | ❌ | ❌ | ✅ |
+| Trend chart | ❌ | ✅ | ✅ |
+| Failures grouped by root cause | ❌ | 🟡 `categories.json` with regex | ✅ automatic clusters |
+| Failures explained in plain words | ❌ | ❌ | ✅ 18 kinds |
+| Bug report in one click | ❌ | ❌ | ✅ Markdown, Jira, text |
+| Environment info | ❌ | 🟡 `environment.properties` | ✅ automatic + `env` option |
+| API calls with request/response | ❌ | 🟡 manual attachments | ✅ automatic, Copy as cURL |
+| Logs and test data blocks | 🟡 attachments | 🟡 attachments, parameters | ✅ `log()`, `testData()` |
+| Secrets masked | ❌ | 🟡 parameter masking | ✅ automatic |
+| CSV / JSON export of failures, Slack summary | ❌ | ❌ | ✅ |
+| Timeline by worker | ❌ | ✅ | ✅ |
+| Combine shards / several runs | ✅ blob + `merge-reports` | ✅ Launches | ❌ on the roadmap |
+| Frameworks beyond Playwright | ❌ | ✅ many languages | ❌ on the roadmap |
+| Free and open source | ✅ | ✅ (Allure TestOps is a separate paid product) | ✅ MIT |
+
+Based on the official docs as of September 2026. ❌ means the feature is not described in the tool's documentation, not that it is impossible. Sources: Playwright [reporters](https://github.com/microsoft/playwright/blob/main/docs/src/test-reporters-js.md), [annotations](https://github.com/microsoft/playwright/blob/main/docs/src/test-annotations-js.md), [retries](https://github.com/microsoft/playwright/blob/main/docs/src/test-retries-js.md), [sharding](https://github.com/microsoft/playwright/blob/main/docs/src/test-sharding-js.md), [trace viewer](https://github.com/microsoft/playwright/blob/main/docs/src/trace-viewer-intro-js.md); Allure [allure-playwright](https://github.com/allure-framework/allure-js/blob/main/packages/allure-playwright/README.md), [Allure 3](https://github.com/allure-framework/allure3), [Allure 2.24.0 release](https://github.com/allure-framework/allure2/releases/tag/2.24.0), [Allure docs: command line](https://github.com/allure-framework/allure-docs/blob/main/content/reporting/commandline.md), [features](https://github.com/allure-framework/allure-docs/blob/main/content/gettingstarted/features.md), [report structure](https://github.com/allure-framework/allure-docs/blob/main/content/gettingstarted/report-structure.md). If something here is out of date, open an issue and it will be fixed.
+
 ## All options
 
 Every option is optional. `npx reporting-labs init` writes them all, with comments, into `reporting-labs.config.ts` (`--js` for JavaScript, `--force` to overwrite).
@@ -232,8 +280,8 @@ Every option is optional. `npx reporting-labs init` writes them all, with commen
 | `env` | – | Extra rows on the Environment card |
 | `links` | `{}` | Turn meta keys into links. `{id}` is replaced by the value |
 | `maskKeys` | `[]` | Extra keys to mask as `****` |
-| `dimensions` | `['priority','severity','feature','owner']` | Meta keys that get charts and filters |
-| `dimensionOrder` | P0…P4, blocker…trivial | Sort order per dimension |
+| `dimensions` | `['priority','severity','feature','owner']` | Which `meta()` keys get a tab in the Breakdown chart and a dropdown filter on the Tests page. Add your own key, e.g. `'team'`, to get a chart for it |
+| `dimensionOrder` | P0…P4, blocker…trivial | The order values appear in those charts and filters. Only needed for your own values, e.g. `{ severity: ['high','medium','low'] }` |
 | `widgets` | all on | Hide cards: `{ tags: false, timeline: false, ... }`. Failure clusters and the Trend chart are always shown |
 | `sections` | `[]` | Extra HTML below the summary, e.g. release notes |
 | `history` | `{ enabled: true, keep: 30 }` | Run history file; `file` sets a custom path |

package/bin/config-template.txt

@@ -3,6 +3,9 @@ import type { ReportingLabsOptions } from 'reporting-labs';
 
 // reportingLabs configuration.
 // Every option is optional. Lines that start with // show the default; uncomment and change what you need.
+// All cards and tabs are on by default. What the report shows depends on your tests: add meta() for
+// priorities and owners, run the suite twice for trends and flaky detection, and keep Playwright's
+// screenshot / video / trace settings on. A card with nothing to show yet tells you what to do.
 // Docs: https://github.com/naveenautomationlabs/reporting-labs#options
 
 const config: ReportingLabsOptions = {
@@ -35,8 +38,17 @@ const config: ReportingLabsOptions = {
 
   // ── Test details ─────────────────────────────────────────────────────────────
   // Values come from meta({ priority, severity, owner, feature, epic, story, ... }) in your tests.
-  // dimensions: ['priority', 'severity', 'feature', 'owner'],   // which meta keys get charts and filters
-  // dimensionOrder: { severity: ['blocker', 'critical', 'major', 'minor', 'trivial'] },
+  //
+  // dimensions: which meta keys get their own tab in the Breakdown chart and a dropdown filter on the
+  // Tests page. Default: priority, severity, feature, owner. Add your own key when you want a chart
+  // for it, e.g. meta({ team: 'web' }) + 'team' here = a "Team" tab showing pass/fail per team.
+  // Keys like epic and story do not need to be here: they show as chips and links on every test anyway.
+  // dimensions: ['priority', 'severity', 'feature', 'owner', 'team'],
+  //
+  // dimensionOrder: the order values appear in those charts and filters. Priority (P0..P4) and
+  // severity (blocker..trivial) are already ordered. Set this only for your own values,
+  // e.g. { severity: ['high', 'medium', 'low'] } or { team: ['web', 'api', 'mobile'] }.
+  // dimensionOrder: { severity: ['high', 'medium', 'low'] },
   // links: {                                         // turn meta keys into links; {id} is the value
   //   story: 'https://acme.atlassian.net/browse/{id}',
   //   epic: 'https://acme.atlassian.net/browse/{id}',

package/dist/template.js

@@ -268,6 +268,7 @@ a{color:var(--accent)}
 .item .tt .n{font-size:13px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
 .item .d{font:11px var(--mono);color:var(--ink-3);margin-top:3px}
 .empty{padding:40px 20px;text-align:center;color:var(--ink-3)}
+.nudge{font-size:var(--fs-s);color:var(--ink-2);background:var(--surface-2);border:1px dashed var(--line);border-radius:var(--radius);padding:10px 12px;margin-bottom:10px;line-height:1.5} .nudge code{font-family:var(--mono);font-size:12px} .nudge b{color:var(--ink)}
 
 /* ---------- detail ---------- */
 .detail{background:var(--surface);border:1px solid var(--line);border-radius:var(--radius);padding:22px;min-width:0}
@@ -872,6 +873,7 @@ function summary(){
   if(W.skipped!==false) { const sc=skippedCard(); if(sc) g.push(sc); }
   if(W.environment!==false) { const ec=envCard(); if(ec) g.push(ec); }
   if(data.history.length>1) g.push(h('div',{class:'card w12'}, h('h2',{},'Trend', h('span',{class:'hint'},'last '+data.history.length+' runs')), trend()));
+  else if(data.history.length===1) g.push(h('div',{class:'card w12'}, h('h2',{},'Trend', h('span',{class:'hint'},'from the second run')), h('div',{class:'nudge'}, h('b',{},'First run recorded.'),' Run the suite once more and this card shows the pass-rate trend, and every failure gets "new this run" or "failing since #…", last-10-runs dots, a Flakiest tests card and a Got slower tab. History lives in ',h('code',{},data.historyFile||'reporting-labs.history.json'),', commit it or cache it in CI.')));
   const ge=data.globalErrors||[], go=data.globalOutput||[];
   if(ge.length||go.some(x=>x.stream==='err')) g.unshift(h('div',{class:'card w12 fail-rail'}, h('h2',{}, ge.length? ge.length+' error'+(ge.length>1?'s':'')+' outside tests' : 'Output outside tests', h('span',{class:'hint'},'spec files that failed to load, global setup, worker crashes')),
     ...ge.map(errorView), go.length? h('details',{class:'errfull'}, h('summary',{},'Console output outside tests ('+go.length+' chunks)'), h('pre',{class:'txt'}, go.map(x=>(x.stream==='err'?'[stderr] ':'')+x.text).join(''))) : null));
@@ -1137,10 +1139,11 @@ function breakdown(){
   if(data.options.widgets.tags&&data.tests.some(t=>t.tags.length)) tabs.push(['tags','Tags',tagsChart]);
   if(data.projects.length>1&&DIMS.length) tabs.push(['heat','Heatmap',heatmap]);
   if(!tabs.length) return h('div',{class:'empty'},'Add meta({ priority, severity, owner, feature }) to your tests to see breakdowns here.');
+  const nudge=!DIMS.length? h('div',{class:'nudge'}, h('b',{},'Tip:'),' add ',h('code',{},"meta({ priority: 'P1', severity: 'major', owner: 'priya', feature: 'checkout' })"),' at the top of a test and this card gains Priority, Severity, Feature and Owner tabs, the failures get ranked, and owners get their own rollup.') : null;
   let cur=tabs[0][0]; const bar=h('div',{class:'bk-tabs'}), body=h('div',{});
   const render=()=>{ body.innerHTML=''; body.append(tabs.find(t=>t[0]===cur)[2]()); bar.querySelectorAll('button').forEach(b=>b.setAttribute('aria-selected',b.dataset.k===cur)); };
   for(const [k,l] of tabs) bar.append(h('button',{'data-k':k,onclick:()=>{cur=k;render();}},l));
-  render(); return h('div',{}, bar, body);
+  render(); return h('div',{}, nudge, bar, body);
 }
 function errorSignature(msg){
   return (msg||'').split('\n')[0].replace(/\d+(\.\d+)?(ms|s)\b/g,'N').replace(/\b\d{2,}\b/g,'N').replace(/["'][^"']{0,60}["']/g,'"…"').replace(/\s+/g,' ').trim().slice(0,160);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "reporting-labs",
-  "version": "0.4.1",
+  "version": "0.4.3",
   "description": "reportingLabs – beautiful, customizable single-file HTML test reports. One line of config. Playwright adapter included, more runners on the way.",
   "keywords": [
     "test-report",