overall-quality-dashboard 0.0.0-stage → 1.3.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Mazen Thabet
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,227 @@
-# Temporary Holding Version
+# Overall Quality Report
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+One page for every QA signal of a project: the latest results and trends of each automated suite (UI, API, …), the automation bugs in
+**Jira**, automation coverage from **Notion**, and **JMeter** performance runs. It shows a health score with a written summary, what needs
+attention, failure clusters, flaky tests, tests back to passing (quarantine), slowest tests, endpoints and trends.
+
+You feed it data, it does the rest. No server, no database, no installs: one Python script (standard library, Python 3.9+) turns CI results
+into a static page you open in a browser or send as a single HTML file. Jira, Notion and performance are optional: without them their
+cards say how to connect them and everything else works as usual.
+
+```
+ CI runs (GitHub, GitLab, Jenkins, Azure DevOps, files)   ┐
+ Jira bugs (token or CSV export)                           ├─► python3 qa_dash.py update ─► data/ ─► index.html, dist/qa-dashboard.html
+ Notion coverage (token or CSV export)                     │
+ JMeter results (.csv / .jtl / statistics.json)            ┘
+```
+
+---
+
+## Quick start
+
+Using Playwright? Add the reporter instead (next section): it sets everything up on the first run. Without Playwright, from a copy of the tool:
+
+```bash
+git clone <this repo> qa-dashboard && cd qa-dashboard
+cp config.example.json config.json        # your project: suites, targets, SLA, Jira and Notion mapping
+python3 qa_dash.py serve                    # opens the dashboard at http://127.0.0.1:8765
+```
+
+Then open **Settings** in the dashboard, add your data sources and click **Save and run the update**.
+Without `serve`, open `index.html` directly (after `python3 qa_dash.py build`): Settings then saves into the folder you pick, and you run
+`python3 qa_dash.py update` in a terminal.
+Settings → **Download the user guide** gives the people who read the report a one-page guide (`docs/user-guide.html`).
+
+To look around first, `python3 qa_dash.py demo` loads 45 days of sample data. It replaces the history: run it before adding real data, or with `QA_DASH_DATA=/tmp/qa-demo` in front.
+
+---
+
+## Use it as a Playwright reporter
+
+Add it to any Playwright project and the dashboard is refreshed after every run. It never opens by itself: open it when you need it.
+
+```bash
+npm i -D overall-quality-dashboard
+```
+```ts
+// playwright.config.ts
+reporter: [
+  ['list'],
+  ['overall-quality-dashboard', { suite: { API: 'api', Chrome: 'ui' } }],   // or suite: 'ui' for every test
+],
+```
+
+After each run it prints one line, and the dashboard is in `qa-dashboard/` next to the config:
+
+```bash
+npx qa-dashboard                 # open it at http://127.0.0.1:8765 (Settings, Sync and Run the update work there)
+npx qa-dashboard update          # any qa_dash.py command: update (Jira, Notion, CI links), build, …
+```
+or open `qa-dashboard/dist/qa-dashboard.html`, the single file to share.
+
+| Option | Default | |
+|---|---|---|
+| `suite` | the project's name | the dashboard suite of every test (`'ui'`), or per project (`{ API: 'api', Chrome: 'ui' }`) |
+| `outputDir` | `qa-dashboard` | where the dashboard lives, relative to the Playwright config |
+| `python` | `python3` (or `QA_DASH_PYTHON`) | Python 3.9+ is needed; without it the run still passes and the reporter says so |
+| `branch` | from the CI, else git | the branch the run is filed under (only `baselineBranch` runs count) |
+
+- The run is recorded with retries, flaky results and every error, keyed like a `results.json` from CI, so both can feed the same dashboard.
+- Sharded runs are skipped shard by shard: add the merged run once with `npx playwright merge-reports --config playwright.config.ts <blob-report folder>`.
+- In CI the history must survive between runs, or every run starts a new dashboard. With GitHub Actions:
+
+```yaml
+- uses: actions/cache@v4
+  with: { path: qa-dashboard/data, key: "qa-dashboard-${{ github.run_id }}", restore-keys: qa-dashboard- }
+- run: npx playwright test
+- uses: actions/upload-artifact@v4
+  with: { name: qa-dashboard, path: qa-dashboard/dist/qa-dashboard.html }
+```
+
+Locally, add `qa-dashboard/` to `.gitignore`, or commit `qa-dashboard/data/` on purpose to share the history.
+
+## 1. Add data: Settings + `update`
+
+1. Open **Settings** (left menu) and fill in what you have:
+   - **Test runs: CI links**: one line per suite (a name such as `api` or `ui`, and a link). See the table below.
+   - **Test run files**: upload a run's results for any CI: `results.json`, JUnit `.xml`, a Playwright report's `index.html`, or the job log.
+   - **Jira**: the search or filter link of the automation bugs with your Jira email and API token, or a Jira CSV export.
+   - **Notion**: the page or database link (or its id) with a Notion token, or a Notion CSV export (zip or CSV).
+   - Logins and tokens (Jira, Notion, GitHub and the other CIs) are saved in `.env` when the dashboard is opened with `serve`; token
+     fields show dots, and the page never receives a saved token back. Opened as a file, edit `.env` instead (see `.env.example`).
+   - **Performance**: JMeter results (`.csv` / `.jtl`, aggregate CSV or `statistics.json`) and the load profile's name.
+   - **Run history**: 7, 14, 30 or 60 days, and the branch that counts.
+   - **How it is counted**: the report's name, each suite's label and manual minutes, targets, which parts count in the health score,
+     Jira squads (where the squad lives, offered from the last export's fields), the Notion rules (which rows count and which columns
+     hold the name, group and status, offered from the last export's columns) and the performance SLA. These are written to `config.json`.
+2. **Save and run the update** (dashboard opened with `python3 qa_dash.py serve`, any browser): writes `sources.json`, the changed values into
+   `config.json` (only those: the rest of the file, including edits made by hand, stays as it is) and the uploads into `inbox/`, runs the update
+   with its output shown on the page, and reloads the dashboard when it finishes.
+3. Opened as a file instead: **Save to the qa-dashboard folder** (Chrome / Edge; pick the folder once), then run `python3 qa_dash.py update`
+   and reload. Other browsers: *Download sources.json* (and *config.json*), move them into the folder, and copy uploads into `inbox/<suite>/`,
+   `inbox/perf/`, `inbox/jira/` or `inbox/notion/`.
+
+`serve` listens on 127.0.0.1 only and answers only its own page: every call carries a token put into the page as it is served, and calls from
+other sites or host names are refused. `--port` picks another port (the next free one is used when it's taken), `--no-open` skips the browser.
+
+| CI link | What `update` adds | Login it needs (`.env`) |
+|---|---|---|
+| GitHub Actions workflow `…/actions/workflows/<file>.yml` | every scheduled run in the window not in the dashboard yet | `gh auth login` once, or `GITHUB_TOKEN` |
+| GitHub Actions run `…/actions/runs/<id>` | that run | same |
+| GitLab CI `…/-/pipelines`, `…/-/pipelines/<id>`, `…/-/jobs/<id>` | scheduled pipelines in the window (results artifact, else the job log) | `GITLAB_TOKEN` |
+| Jenkins `…/job/<name>` or `…/job/<name>/<number>` | timer-started builds in the window (results artifact, else the console log) | `JENKINS_USER` + `JENKINS_TOKEN` |
+| Azure DevOps `…/_build?definitionId=<id>` or `…/_build/results?buildId=<id>` | scheduled builds in the window (results artifact, else the logs) | `AZURE_DEVOPS_TOKEN` |
+| A link straight to a results or log file | that file | `CI_TOKEN` (sent as a bearer token), if any |
+
+For each run it takes the best file it finds: a `test-results-*` artifact (Playwright JSON + JUnit), then a Playwright HTML report, then the log.
+Only scheduled runs count unless *Also add manually started runs* is ticked. Re-running `update` is safe: runs and files already in the
+dashboard are recognised and skipped. Without a Jira or Notion token, `update` says so and skips them: import their CSV exports instead.
+The last export of each is kept in `data/exports/` and read again on every build, so a change to the Jira squads or the Notion rules applies
+without importing it again. A download that GitHub cuts off is tried again (four tries); a run that still fails is picked up by the next update.
+
+GitHub Actions has been verified end to end. The GitLab, Jenkins and Azure DevOps fetchers follow those tools' APIs but have not been
+run against a live server yet; if one does not work for you, upload the run's files under *Test run files*.
+
+The Executive Summary has two buttons:
+- **Sync** rebuilds the dashboard from the data and settings already in the folder (`python3 qa_dash.py build`, no fetching), then reloads.
+- **Export as PDF** saves that page as a PDF report (the browser's print dialog → *Save as PDF*): A4 landscape, light theme, without the
+  menu and buttons, named after the report and its date.
+
+Buttons that run the tool (*Run the update*, *Sync*) need the dashboard opened with `serve`; opened as a file, they say how.
+The browser keeps unsaved Settings changes and each viewer's *Score settings* per site, and a page opened as a file is another site than
+the served one: Settings has *Discard changes*, and the score card says when this browser uses its own score settings (*Reset*).
+
+## 2. Add files by hand
+
+```bash
+python3 qa_dash.py ingest <file, folder or zip> --module api     # the suite the run belongs to
+python3 qa_dash.py build
+```
+
+| Format | Notes |
+|---|---|
+| Playwright JSON reporter (`results.json`) | retries, flaky outcomes, every error; the best input |
+| Playwright HTML report (`playwright-report/index.html`) | the results embedded in the page; same test ids as the JSON reporter |
+| JUnit XML | Playwright, pytest, Newman, Surefire, Karate, Appium, Cypress… (no retries, so no flaky detection) |
+| CI job logs (`.txt`, `.log`, log-archive `.zip`) | Playwright `list` output and list-style custom reporters (statuses, retries, failure headlines, run time), `pytest -v`, Jest/Mocha/Newman ticks, Java TestNG through SHAFT (each test timed from its start and finish lines, Maven's total time). Secret masking (`***`) in titles is undone from the suite's history |
+| Jest / Vitest JSON | `--json --outputFile=results.json` |
+| JMeter per-request CSV / JTL, aggregate CSV, `statistics.json`, or a whole run folder | one run per folder, from the richest file; `--scenario "Target 30 RPS"` names the load profile |
+| k6 summary JSON | |
+| Plain JSON | `{"tests": [{"name", "suite", "status", "duration", "error", "endpoint"}]}` or `{"transactions": [...]}`; see `samples/results-generic.json` |
+| Jira CSV export | replaces the Jira data |
+| Notion CSV export | replaces the coverage data, mapped through `notion.rows` |
+
+`python3 qa_dash.py ingest samples/` shows one of each (use a scratch folder: `QA_DASH_DATA=/tmp/qa-samples python3 qa_dash.py ingest samples/`).
+
+Jira and Notion can also be refreshed on their own: `python3 qa_dash.py sync jira` / `sync notion` (with the tokens in `.env`, which is read automatically).
+
+## 3. Configure (`config.json`)
+
+Most of these can be set on the Settings page. `moduleRules`, `testModuleRules`, `jira.moduleMapping`, `sources` and extra Notion rules
+(more than one, or rules with `exclude` / `progress`) are edited in the file.
+
+| Setting | What it does |
+|---|---|
+| `project.name`, `environment` | header of the report |
+| `windowDays`, `baselineBranch` | history window (Settings can override it) and the branch whose runs count |
+| `targets` | pass rate, coverage and flakiness targets |
+| `healthScore.exclude` | components left out of the health score by default (`perf`, `coverage`, `bugs`, `pass:<suite>`); viewers can change it in *Score settings* |
+| `quarantineDays` | how recent a recovery must be to appear in *Quarantine: recovered tests* |
+| `modules.<suite>` | label and `manualMinutes` (how long one test takes by hand, for *QA hours saved*) |
+| `moduleRules` / `testModuleRules` | file a file under a suite by its path / file each test of a combined run under its own suite by its spec path |
+| `perf` | the SLA: `targetRps`, `maxErrorPct`, optional p95 limits (`defaultSlaP95Ms`, `sla` per transaction), `regressionPct` |
+| `jira` | site, project, squads, where squads live (`labels`, `components` or a custom field), extra JQL, priorities that count as severe, label → suite mapping |
+| `notion` | database or page link; optional `rows` rules and `layers` (see below) |
+| `sources` | how the sidebar names each source and where it links |
+
+**Notion without rules** (the default): every row is one case, named by the first column (the page name), automated when a column such as
+*Automated*, *Automation status* or *Is automated* holds Yes / Automated / Done (`automatedValues`), grouped by *Module*, *Feature* or *Area*.
+**Rules** are for databases that mix kinds of rows or count several items per row; Settings → Coverage: Notion → *Advanced* edits the first one:
+
+```json
+"rows": [
+  {"match": {"Type": "UI feature"}, "layer": "ui", "title": "Name", "feature": "Module", "status": "Automated"},
+  {"match": {"Type": "API service"}, "layer": "api", "title": "Name", "feature": "Module", "count": "Endpoints", "automatedCount": "Endpoints automated"}
+],
+"layers": {"ui": {"label": "UI", "unit": "features"}, "api": {"label": "API", "unit": "endpoints"}}
+```
+
+## 4. How the numbers are calculated
+
+- **Latest run first**: pass rate (vs the previous run), failures, run duration, failure clusters, endpoints, slowest tests, CI efficiency and the health
+  score describe each suite's latest run. Trend charts (one point per run), Recent Runs, flakiness, regressions and quarantine look back over the window.
+- **Pass rate** = (passed + passed on retry) ÷ executed, skipped excluded.
+- **Test status** from each test's runs in the window: *broken*, *regression* (failed the last 2+ runs after passing), *failing*, *flaky* (instability ≥ `targets.flakyRate`), *stable*.
+  Tests missing from the latest run (renamed or removed) are left out.
+- **Failure clusters** group the latest run's errors by their first line into *product*, *test / locator* (selectors, strict mode, setup) and
+  *infrastructure* (connection errors, crashes, any 5xx status or "Internal Server Error", any timeout). **Signal reliability** = product ÷ all failures.
+- **Quarantine: recovered tests** = failed in the window for a reason other than infrastructure, passing again in the last `quarantineDays`, and in every run since.
+- **QA hours saved** = Σ this year's runs of (tests executed × `manualMinutes` ÷ 60 − the run's duration), never below 0.
+- **Bugs raised** = Jira bugs matching the query created in the last 90 days; *still open* = those not done.
+- **Coverage** per layer in its own unit (e.g. UI features, API endpoints); layers with different units are never added up.
+- **Performance SLA** (latest run): throughput ≥ `targetRps` and error rate < `maxErrorPct`, plus p95 limits when set; a transaction is over the SLA when its own error rate reaches the limit.
+- **Health score (0–100)** over the components switched on: latest-run pass rates 50 (shared; a pass rate scores from 80 % up), performance SLA 20,
+  coverage 15 (shared by the layers), open critical/high bugs 15 (−3 each). Missing sources are left out and the rest rescaled.
+
+## 5. Files
+
+```
+qa_dash.py            the pipeline: update, ingest, sync jira | notion, build, demo, serve
+reporter.js           the Playwright reporter (npm package overall-quality-dashboard); bin/qa-dashboard.js opens the dashboard
+index.html            the dashboard (reads data/data.js)
+docs/user-guide.html  the user guide (Settings → Download the user guide)
+config.example.json   every setting, with examples (copy to config.json)
+samples/              an example of every supported input
+ci/github-actions.yml example workflow for publishing it from CI
+.env.example          the tokens `update` can use
+
+Created on your machine and kept out of git:
+config.json, sources.json  your settings and data sources
+data/                      history (one line per run, per suite), Jira and Notion snapshots, the last CSV exports, data.js
+dist/qa-dashboard.html     the single file to share
+inbox/                     files saved by Settings, moved to inbox/done/ by `update`
+```
+
+---
+
+Created by **Mazen Thabet**: [GitHub](https://github.com/mazenathabet) · [LinkedIn](https://www.linkedin.com/in/mazenahmedthabet/) · MIT license

package/bin/qa-dashboard.js

@@ -0,0 +1,20 @@
+#!/usr/bin/env node
+// Open the QA dashboard the reporter keeps up to date: `npx qa-dashboard` starts it at http://127.0.0.1:8765 (Settings can save,
+// Sync and run the update there). Any other arguments go to qa_dash.py: `npx qa-dashboard update`, `npx qa-dashboard build`.
+// --dir <folder> when the reporter's outputDir isn't ./qa-dashboard.
+const fs = require('fs');
+const path = require('path');
+const { spawnSync } = require('child_process');
+
+const args = process.argv.slice(2);
+const at = args.indexOf('--dir');
+const dir = path.resolve(at >= 0 ? args.splice(at, 2)[1] || 'qa-dashboard' : 'qa-dashboard');
+const tool = path.join(dir, 'qa_dash.py');
+if (!fs.existsSync(tool)) {
+  console.error(`No dashboard in ${dir} yet: run your Playwright tests once with the overall-quality-dashboard reporter (or pass --dir <its outputDir>).`);
+  process.exit(1);
+}
+const python = process.env.QA_DASH_PYTHON || 'python3';
+const r = spawnSync(python, [tool, ...(args.length ? args : ['serve'])], {cwd: dir, stdio: 'inherit'});
+if (r.error) console.error(r.error.code === 'ENOENT' ? `${python} not found: the dashboard needs Python 3.9+ (or set QA_DASH_PYTHON to its path).` : r.error.message);
+process.exit(r.status == null ? 1 : r.status);

package/config.example.json

@@ -0,0 +1,57 @@
+{
+  "project": { "name": "Overall Quality Report", "environment": "staging" },
+  "windowDays": 30,
+  "baselineBranch": "main",
+  "targets": { "passRate": 95, "coverage": 80, "flakyRate": 0.1 },
+  "healthScore": { "exclude": [] },
+  "quarantineDays": 7,
+
+  "modules": {
+    "ui":   { "label": "UI",          "manualMinutes": 6 },
+    "api":  { "label": "API",         "manualMinutes": 2 },
+    "perf": { "label": "Performance" }
+  },
+  "moduleRules": [
+    { "match": "test-results-api", "module": "api" },
+    { "match": "test-results-ui",  "module": "ui" }
+  ],
+  "testModuleRules": [],
+
+  "perf": {
+    "targetRps": 30,
+    "maxErrorPct": 10,
+    "defaultSlaP95Ms": null,
+    "regressionPct": 15,
+    "sla": {}
+  },
+
+  "jira": {
+    "baseUrl": "https://your-company.atlassian.net",
+    "project": "PAY",
+    "squads": ["Payments Squad"],
+    "squadField": "labels",
+    "issueTypes": ["Bug"],
+    "extraJql": "labels = AUTOMATION",
+    "severePriorities": ["Blocker", "Highest", "Critical", "High"],
+    "moduleMapping": { "ui": ["ui", "frontend"], "api": ["api", "backend"], "perf": ["performance", "load"] }
+  },
+
+  "notion": {
+    "databaseId": "",
+    "url": "",
+    "name": "Automation coverage",
+    "properties": {},
+    "rows": [],
+    "layers": {},
+    "layerMapping": {},
+    "automatedValues": ["Automated", "Yes", "Done"],
+    "inProgressValues": ["In Progress", "Automating"],
+    "excludedValues": ["Not applicable", "Not Automatable", "Deprecated"]
+  },
+
+  "sources": {
+    "api": { "via": "GitHub Actions", "name": "API tests, scheduled runs", "url": "https://github.com/<org>/<repo>/actions/workflows/api-tests.yml?query=event%3Aschedule" }
+  },
+
+  "links": {}
+}

package/docs/user-guide.html

@@ -0,0 +1,177 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Overall Quality Report: user guide</title>
+<style>
+  :root { --text:#1b2233; --muted:#5b6478; --border:#e1e4ee; --accent:#2f3fd0; --soft:#eef0fb; --bg:#ffffff; --code:#f4f5f9; }
+  * { box-sizing: border-box; }
+  body { margin: 0; background: var(--bg); color: var(--text); font: 15px/1.6 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; }
+  main { max-width: 860px; margin: 0 auto; padding: 40px 24px 64px; }
+  h1 { font-size: 30px; margin: 0 0 6px; color: #141c6b; }
+  h2 { font-size: 20px; margin: 36px 0 10px; color: #141c6b; border-bottom: 1px solid var(--border); padding-bottom: 6px; }
+  h3 { font-size: 16px; margin: 22px 0 6px; }
+  p.lead { color: var(--muted); font-size: 16px; margin-top: 0; }
+  code { background: var(--code); border-radius: 4px; padding: 1px 5px; font: 13px ui-monospace, Menlo, monospace; }
+  pre { background: var(--code); border-radius: 8px; padding: 12px 14px; overflow-x: auto; font: 13px/1.5 ui-monospace, Menlo, monospace; }
+  table { border-collapse: collapse; width: 100%; margin: 10px 0 16px; font-size: 14px; }
+  th, td { border-bottom: 1px solid var(--border); padding: 7px 8px; text-align: left; vertical-align: top; }
+  th { color: var(--muted); font-weight: 600; font-size: 13px; }
+  .steps { counter-reset: s; list-style: none; padding: 0; }
+  .steps li { counter-increment: s; position: relative; padding: 2px 0 12px 40px; }
+  .steps li::before { content: counter(s); position: absolute; left: 0; top: 0; width: 26px; height: 26px; border-radius: 50%; background: var(--accent); color: #fff; font-weight: 700; text-align: center; line-height: 26px; font-size: 13px; }
+  .note { background: var(--soft); border-left: 4px solid var(--accent); border-radius: 6px; padding: 10px 14px; margin: 12px 0; }
+  .toc a { color: var(--accent); text-decoration: none; } .toc { columns: 2; font-size: 14px; }
+  @media print { main { padding: 0; } h2 { break-after: avoid; } pre, table { break-inside: avoid; } }
+</style>
+</head>
+<body>
+<main>
+<h1>Overall Quality Report</h1>
+<p class="lead">How to read the dashboard, and how to give it the data it needs: test runs from GitHub, automation bugs from Jira,
+coverage from Notion and performance runs from JMeter.</p>
+
+<div class="toc">
+  <a href="#open">1. Open the dashboard</a><br><a href="#read">2. What each page tells you</a><br><a href="#data">3. Give it data (Settings)</a><br>
+  <a href="#update">4. Run the update</a><br><a href="#files">5. Files you can add by hand</a><br><a href="#score">6. The health score</a><br>
+  <a href="#rules">7. Rules that keep the numbers right</a><br><a href="#faq">8. Questions</a>
+</div>
+
+<h2 id="open">1. Open the dashboard</h2>
+<ul>
+  <li><b>On your machine:</b> open <code>qa-dashboard/index.html</code> in a browser (Chrome or Edge recommended). After every update, reload the page.</li>
+  <li><b>To share it:</b> send <code>qa-dashboard/dist/qa-dashboard.html</code>. It is one file with the data inside and opens anywhere, offline.</li>
+</ul>
+<p>Nothing to install besides Python 3.9 or newer (already on macOS). The dashboard needs no server and no internet connection to view.</p>
+
+<h2 id="read">2. What each page tells you</h2>
+<table>
+  <tr><th>Page</th><th>What it shows</th></tr>
+  <tr><td>Executive Summary</td><td>The quality health score with a short written summary, what needs attention, what is going well, the failure clusters of the latest runs, hours saved, bugs caught, test-suite speed and trends.</td></tr>
+  <tr><td>Quality Trends</td><td>Every suite run by run: pass rate, failures and flaky tests. One point per run, so a day without a run leaves no gap.</td></tr>
+  <tr><td>Bugs</td><td>The automation bugs from Jira: raised, open, critical/high, time to resolve, created vs resolved per week, the open list.</td></tr>
+  <tr><td>Coverage</td><td>UI coverage (features of the automation plan) and API coverage (swagger endpoints), each in its own unit.</td></tr>
+  <tr><td>Each suite (UI, API, …)</td><td><b>Overview</b> (latest run, trend, regressions, flaky tests, tests back to passing that can leave quarantine), <b>Recent Runs</b>, <b>Test Health</b>, <b>Endpoints</b>, <b>Failure Clusters</b>, <b>Slowest Tests</b>.</td></tr>
+  <tr><td>Performance</td><td>JMeter runs against the SLA (throughput and error rate), latency, throughput, transactions, SLA breaches.</td></tr>
+  <tr><td>Settings</td><td>Where the data comes from, how it is counted (suites, targets, Jira squads, Notion rules, SLA), file uploads, and this guide.</td></tr>
+</table>
+<p><b>Export as PDF</b> (top right of the Executive Summary) saves the summary as a PDF report: in the print dialog that opens, choose
+<i>Save as PDF</i>. It is laid out on A4 landscape, in the light theme, without the menu and buttons.
+<b>Sync</b>, beside it, rebuilds the dashboard from the data and settings already in the folder and reloads the page; it needs the dashboard
+opened with <code>python3 qa_dash.py serve</code> (to fetch new runs, Jira and Notion, use <b>Run the update</b> in Settings).</p>
+<div class="note">Every figure describes the <b>latest run</b> of each suite. Only trends, recent runs, flakiness, regressions and the quarantine list look back over the last 30 days. Every number has a <b>?</b> next to it that explains how it is calculated.</div>
+
+<h2 id="data">3. Give it data: the Settings page</h2>
+<p>Open <b>Settings</b> in the left menu. Fill in what you have; every part is optional.</p>
+<h3>Project</h3>
+<p>The report's name (the title on every page) and the environment it tests.</p>
+<h3>Run history</h3>
+<p>Choose how far back the dashboard looks: 7, 14, 30 or 60 days. It sets the trends, flakiness, quarantine and Recent Runs, and how far back the update fetches runs.
+<i>Branch that counts</i> is the branch the scheduled runs use (usually <code>main</code>); runs from other branches are kept but left out of the numbers.</p>
+<h3>Test runs: CI links</h3>
+<p>One line per suite: type the suite's name (for example <code>api</code> or <code>ui</code>) and paste a link.</p>
+<ul>
+  <li><b>GitHub Actions</b>: a workflow link (<code>https://github.com/&lt;org&gt;/&lt;repo&gt;/actions/workflows/api-tests.yml</code>) adds every scheduled run in the history window that is not in the dashboard yet; a run link (<code>…/actions/runs/&lt;run id&gt;</code>) adds that one run.</li>
+  <li><b>GitLab CI</b> (<code>…/-/pipelines</code>), <b>Jenkins</b> (<code>…/job/&lt;name&gt;</code>) and <b>Azure DevOps</b> (<code>…/_build?definitionId=…</code>) work the same way once their token is set (section 4).</li>
+  <li><b>A link straight to a results or log file</b> works for any CI.</li>
+</ul>
+<p>Only scheduled (nightly) runs are pulled; manually started runs are left out unless you tick the option to include them.</p>
+<h3>Test run files</h3>
+<p>For any CI, or when a link can't be used: name the suite and upload the run's <code>results.json</code>, JUnit <code>.xml</code>, the Playwright report's <code>index.html</code>, or the CI job's log (<code>.txt</code>, <code>.log</code> or the log archive <code>.zip</code>). One file per run.</p>
+<h3>Suites</h3>
+<p>For each suite: the name shown on every page, and how many minutes one of its tests takes a person to run by hand (for <i>QA hours saved</i>).</p>
+<h3>Targets and health score</h3>
+<p>The pass rate and coverage targets, from what share of unstable runs a test counts as flaky, how recent a recovery must be to show under quarantine,
+and which parts count in the health score by default.</p>
+<h3>Automation bugs (Jira)</h3>
+<p>Paste the link of a Jira search that lists the automation bugs (for example a search for <code>labels = AUTOMATION</code> in your project and squad), or a saved filter's link,
+with your Jira email and an API token (id.atlassian.com → Security → API tokens). No API token? Export that search instead (<i>Export → Export CSV</i>) and upload the CSV here.</p>
+<p>Under <i>Advanced</i>: where the squad is set on a bug (labels, components, or a custom field such as <i>Squad</i>; the fields of your last export are offered),
+the squads, and which priorities count as critical/high. The Jira site is taken from the link unless you set it.</p>
+<h3>Coverage (Notion)</h3>
+<p>Paste the link of the Notion page that holds the coverage database (or the database's link or id), with a Notion token. Without a Notion token, export the database
+(<i>⋯ → Export → Markdown &amp; CSV</i>) and upload the zip or the CSV here. That is usually all: the dashboard finds the name, the automated status and the group by itself.</p>
+<p>Tokens and logins are saved in <code>.env</code> on your computer when the dashboard is opened with <code>python3 qa_dash.py serve</code>. Their fields show dots; leave a
+token field empty to keep the saved one.</p>
+<p>Only if the coverage looks wrong, open <i>Advanced: how rows count</i> and tell the dashboard how to read the database: which rows count (a column and its value, e.g. <i>Source</i> is <i>Jira story</i>),
+the columns holding the name, the group (e.g. <i>Module</i>) and the automated status, which suite the coverage belongs to, and what one row is (features, stories…).
+After the first upload the export's columns are offered to pick from. A change here applies on the next update, without uploading the export again.</p>
+<h3>Performance runs (JMeter)</h3>
+<p>Upload the run's results: the per-request <code>.csv</code> / <code>.jtl</code> (best), the aggregate <code>.csv</code>, or the HTML report's <code>statistics.json</code>.
+Give the load profile a name (e.g. <i>Target 30 RPS</i>). Under <i>SLA</i>: the throughput a run must reach, the error rate it must stay under,
+and an optional p95 response time limit.</p>
+<h3>Save</h3>
+<ul>
+  <li>Opened with <code>python3 qa_dash.py serve</code>: <b>Save</b> (or <b>Save and run the update</b>) writes straight into the folder, in any browser.</li>
+  <li>Opened as a file: <b>Save to the qa-dashboard folder</b> (Chrome / Edge): the first time, the browser asks you to pick the <code>qa-dashboard</code> folder. The data sources go to <code>sources.json</code>,
+    everything else to <code>config.json</code> (only the values you changed; the rest of the file stays as it is), and the uploaded files to <code>qa-dashboard/inbox/</code>.</li>
+  <li>Other browsers: <b>Download sources.json</b> (and <b>config.json</b> when you changed it), move them into <code>qa-dashboard</code>, and copy the files you wanted to upload into <code>qa-dashboard/inbox/perf/</code>, <code>inbox/jira/</code> or <code>inbox/notion/</code>.</li>
+  <li>Until you save, a note at the top of Settings says you have changes that aren't saved yet: the update only sees saved settings.
+    <b>Discard changes</b> goes back to the saved settings. Unsaved changes are kept in this browser only, and the page opened as a file
+    and the page opened through <code>serve</code> count as two different sites, so each has its own.</li>
+</ul>
+
+<h2 id="update">4. Run the update</h2>
+<p>The update adds the new runs from GitHub, refreshes Jira and Notion, takes the files from <code>inbox/</code>, and rebuilds the dashboard.</p>
+<p><b>From the page:</b> start the dashboard with <code>python3 qa_dash.py serve</code> (it opens at <code>http://127.0.0.1:8765</code>). Settings then has
+<b>Save and run the update</b>: it saves, runs the update with its output on the page, and reloads the dashboard when it is done. Keep the terminal open while you use it.</p>
+<p><b>From a terminal</b>, in the <code>qa-dashboard</code> folder (when you opened <code>index.html</code> as a file):</p>
+<pre>python3 qa_dash.py update</pre>
+<p>Then reload the page.</p>
+<table>
+  <tr><th>Source</th><th>What the update needs</th></tr>
+  <tr><td>GitHub</td><td>Your GitHub login: run <code>gh auth login</code> once (GitHub CLI), or put <code>GITHUB_TOKEN=…</code> in <code>qa-dashboard/.env</code>.</td></tr>
+  <tr><td>Jira</td><td><code>JIRA_EMAIL</code> and <code>JIRA_API_TOKEN</code> in <code>.env</code> (create a token at id.atlassian.com → Security → API tokens). Without them: upload a Jira CSV export.</td></tr>
+  <tr><td>Notion</td><td><code>NOTION_TOKEN</code> in <code>.env</code>, from a Notion integration that the database page is shared with. Without it: upload a Notion export.</td></tr>
+  <tr><td>GitLab / Jenkins / Azure DevOps</td><td><code>GITLAB_TOKEN</code>; <code>JENKINS_USER</code> and <code>JENKINS_TOKEN</code>; <code>AZURE_DEVOPS_TOKEN</code>. If a CI can't be reached, upload its run files in Settings instead.</td></tr>
+</table>
+<p>Running it again is safe: runs and files already in the dashboard are recognised and skipped.</p>
+
+<h2 id="files">5. Files you can add by hand</h2>
+<pre>python3 qa_dash.py ingest &lt;file, folder or zip&gt; --module api      # or screening, ui, …
+python3 qa_dash.py build</pre>
+<table>
+  <tr><th>File</th><th>Where it comes from</th></tr>
+  <tr><td><code>results.json</code> / <code>junit-results.xml</code></td><td>A run's <code>test-results-…</code> artifact (Playwright JSON and JUnit reports)</td></tr>
+  <tr><td><code>index.html</code> of a Playwright report</td><td>A run's <code>playwright-report-…</code> artifact</td></tr>
+  <tr><td>A run's log archive or <code>.txt</code></td><td>Run page → ⋯ → Download log archive</td></tr>
+  <tr><td>JMeter <code>.csv</code> / <code>.jtl</code>, aggregate CSV, <code>statistics.json</code>, or a whole run folder</td><td>Performance runs (add <code>--scenario "Target 30 RPS"</code>)</td></tr>
+  <tr><td>Jira CSV export</td><td>Replaces the Jira data</td></tr>
+  <tr><td>Notion CSV export (or its zip)</td><td>Replaces the coverage data</td></tr>
+</table>
+
+<h2 id="score">6. The health score</h2>
+<p>A score from 0 to 100, from the latest run of each suite and the open critical/high bugs. Performance and coverage can count too:
+use <b>Score settings</b> on the score card to switch any part on or off. Your choice is kept in your browser, and the score card says so when it
+differs from the team default (set in Settings → Targets and health score); <i>Reset</i> goes back to the team default.
+Parts that are switched off also disappear from "What needs attention".</p>
+
+<h2 id="rules">7. Rules that keep the numbers right</h2>
+<ul>
+  <li>Use scheduled (nightly) runs. Manually started runs are partial or experimental.</li>
+  <li>Give one file per run: <code>results.json</code> <i>or</i> the JUnit file <i>or</i> the HTML report <i>or</i> the log. (A <code>test-results-…</code> zip with both JSON and JUnit is handled.)</li>
+  <li>Always say which suite a file belongs to (<code>--module</code>), except for <code>test-results-&lt;suite&gt;-…</code> zips.</li>
+  <li>Artifacts expire on GitHub (HTML report 10 days, test results 30 days): keep the update running, or add a missed run from its log archive.</li>
+  <li>Never run <code>python3 qa_dash.py demo</code> in the real folder: it replaces the history with sample data.</li>
+</ul>
+
+<h2 id="faq">8. Questions</h2>
+<h3>A suite shows a low pass rate. Is the product broken?</h3>
+<p>Open the suite's <b>Failure Clusters</b>: each failure is labelled <i>product</i>, <i>test / locator</i> or <i>infrastructure</i>.
+Environment errors, any 5xx answer and every timeout count as infrastructure, not product defects.</p>
+<h3>Why is a test listed under "Quarantine: recovered tests"?</h3>
+<p>It failed (for a reason other than infrastructure) in the last 30 days, started passing again in the last 7 days, and has passed in every run since. A short streak means it may still be unstable.</p>
+<h3>The update says "no token" for Jira or Notion.</h3>
+<p>Add the token to <code>.env</code> (section 4), or upload a CSV export in Settings and run the update again.</p>
+<h3>Our project has no Jira or Notion.</h3>
+<p>Leave them empty. The Bugs and Coverage pages and cards stay empty with a note on how to connect them; the test runs, performance and everything else work as usual,
+and the health score only counts what is connected.</p>
+<h3>Can another project use this?</h3>
+<p>Yes. Take a fresh copy of the tool (<code>cp config.example.json config.json</code>, then <code>python3 qa_dash.py build</code>), open the page and fill in Settings:
+the data sources, the suites, targets and SLA, and how Jira and Notion are read.</p>
+<h3>Who made it?</h3>
+<p>Mazen Thabet. Every page's footer shows the version and how to reach him.</p>
+</main>
+</body>
+</html>