kwikqa 0.1.1 → 0.2.0

package/README.md

@@ -40,6 +40,9 @@ Using pnpm, Yarn, or Bun? See [Package managers and monorepos](#package-managers
   pnpm, Yarn, or Bun. KwikQA runs your project's own copy and never installs packages or browsers.
 - A git repository
 
+For repairs, KwikQA needs one of those three frameworks. It also reads and diagnoses runs from
+[other test frameworks](#other-test-frameworks).
+
 ## Setup
 
 ```sh
@@ -94,6 +97,64 @@ failing specs and records each failure's page (see below).
 
 `login` signs you in with GitHub; the free plan needs no payment details.
 
+### Other test frameworks
+
+KwikQA also reads the reports these write, and diagnoses the run:
+
+- Vitest (`--reporter=junit`);
+- Jest (`jest-junit`);
+- Mocha (`mocha-junit-reporter`);
+- pytest (`--junitxml`);
+- Playwright's own `junit` reporter;
+- JUnit and TestNG through Maven Surefire or Failsafe (`target/surefire-reports`), including
+  Selenium tests;
+- JUnit and TestNG through Gradle (`build/test-results/test`), including the test-retry plugin's
+  retries;
+- TestNG's own `testng-results.xml`;
+- Allure results (`allure-results`) from TestNG, JUnit, Playwright, and pytest;
+- CTRF reports, from Playwright, Jest, pytest, and other tools that write them;
+- Cucumber's JSON report, from cucumber-js (`--format json:<file>`) and Cucumber-JVM (the `json`
+  plugin).
+
+Point `--report` at the file or its directory:
+
+```sh
+npx kwikqa analyze --report reports/junit.xml --html
+```
+
+You get failure groups, a diagnosis, and the [failure report](#failure-report). KwikQA doesn't
+capture, repair, or validate these runs, and `fix` says so before it sends anything. For Playwright
+repairs, use its JSON reporter as shown above.
+
+Some details depend on the reporter:
+
+- For TestNG, `testng-results.xml` keeps retried attempts and failed configuration methods, which
+  Surefire's XML of the same run doesn't correlate. Under Maven, add TestNG's XML reporter to
+  Surefire's configuration, then point `--report` at `test-output`:
+
+  ```xml
+  <properties>
+    <property>
+      <name>listener</name>
+      <value>org.testng.reporters.XMLReporter</value>
+    </property>
+  </properties>
+  ```
+- Allure adds each run's results to its directory, and KwikQA reads the directory as one run, so
+  clean it between runs: `mvn clean`, or pytest's `--clean-alluredir`. Otherwise an earlier run's
+  results read as retries.
+- Cucumber's JSON keeps only each scenario's final attempt, so a scenario that passed on a retry
+  reads as passed. Cucumber-JVM 8 needs a JSON library on the test class path to write it, such as
+  `tools.jackson.core:jackson-databind`.
+- Gradle names a JUnit parameterized test by its display name, which includes its arguments. To
+  keep test data out of KwikQA's titles, set
+  `junit.jupiter.params.displayname.default = {displayName}[{index}]` in
+  `junit-platform.properties`.
+- jest-junit records no test file unless you set `addFileAttribute: true`.
+- pytest records each test's line only with `-o junit_family=xunit1`.
+- JUnit XML from a tool KwikQA doesn't recognize is still read and diagnosed, as an unidentified
+  framework.
+
 ### Package managers and monorepos
 
 This page uses npm's commands. With another package manager, install and run KwikQA the same way:
@@ -119,6 +180,9 @@ WebdriverIO wherever your package manager installed it inside the repository, in
 | --- | --- |
 | `npx kwikqa analyze` | Diagnose the failures in the latest report. `--report <path>` picks a report file or a directory of one run's reports; `--framework` names the framework of a JUnit report; `--local` builds the evidence summary without sending anything; `--no-changes` leaves out recent code changes (see below). |
 | `npx kwikqa analyze --capture` | First re-run the failing tests so every failure has full evidence. Playwright runs with full tracing; with Playwright 1.59+ and [Playwright CLI](https://www.npmjs.com/package/@playwright/cli) installed, it also pauses locator failures at the failing line and asks Playwright for locators of the closest elements (`--no-probe` skips this). Cypress and WebdriverIO re-run the failing specs with your own config plus KwikQA's reporter and failure capture, which records each failed attempt's DOM, command log, console, and network. Asks before running anything; `--yes` approves in CI. |
+| `npx kwikqa analyze --html` | Also write the [failure report](#failure-report), `kwikqa-report.html`; `--output <path>` writes it elsewhere. |
+| `npx kwikqa report` | Write the failure report of the latest analysis without analyzing again or sending anything. `--run <id>` picks an earlier run under `.kwikqa/runs/`; `--output <path>` picks the file. |
+| `npx kwikqa ci` | In a CI job: analyze as `analyze --yes --html` does, then publish a summary. On GitHub Actions that is the job summary and one pull-request comment; with your webhooks, also a Slack or Teams message. See [CI](#ci). Without a KwikQA token it analyzes locally. It takes `--report`, `--framework`, `--local`, `--no-changes`, and `--output` as `analyze` does, and `--notify always` to post to chat after every run, not only failed ones. |
 | `npx kwikqa fix` | Propose a repair for a test-side cause, show the diff and validation commands, and apply it if you approve. First it offers to re-run the failing tests once: a test that passes now is flaky, so KwikQA reports it and requests no repair. It starts with the failure group whose test-side diagnosis is strongest; if the model declines, or its proposal fails KwikQA's checks, it asks about the next such group, at most three, each with its own preview. One `fix` counts as one fix however many groups it tries. `--group <n>` repairs only group n, as `analyze` numbered it. `--propose-only` stops after the proposal. |
 | `npx kwikqa validate` | Run the approved validation. The repaired test must pass three times in a row, with retries off. Passing keeps the patch; failing rolls it back. When other tests failed with the same cause, it then offers to re-run them once and names any that still fail, so you can run `fix` again for them; the patch stays either way. |
 | `npx kwikqa fix --retry` | After a failed validation, re-diagnose from the new evidence and propose one revised patch. |
@@ -130,6 +194,27 @@ WebdriverIO wherever your package manager installed it inside the repository, in
 Exit codes: `0` success or an informative result, `1` the repair or validation did not succeed, `2`
 usage error, `3` sign-in, plan, or usage-limit problem.
 
+## Failure report
+
+`npx kwikqa analyze --html`, or `npx kwikqa report` after an analysis, writes one HTML file,
+`kwikqa-report.html`. It shows:
+
+- the run's counts;
+- each failure group, numbered as `analyze` numbered it, with its tests, its owner (test code,
+  application, or environment), the diagnosis, and the evidence the diagnosis cites;
+- flaky tests and failures no rule could classify;
+- what KwikQA could see for each evidence channel, and every request that was sent.
+
+A local analysis (`--local`) gets the same report without owners. Writing a report is free on
+every plan and needs no sign-in. `kwikqa report` rebuilds the evidence from the same test report and
+artifacts. If those have changed since the analysis, it says so instead of showing evidence the
+diagnosis did not see.
+
+The file is self-contained: it loads nothing from the network, and it stays readable when a viewer
+blocks its script and styles, as Jenkins does for archived HTML by default. It holds the same
+redacted evidence a request preview shows, so share it the way you share CI logs. KwikQA only ever
+replaces a file it wrote itself, and `init` offers to add `kwikqa-report.html` to `.gitignore`.
+
 ## Plans
 
 | Plan | Monthly | Annual | Analyses / month | Fixes / month |
@@ -217,14 +302,81 @@ Run records, transaction journals, captures, and validation output stay on your
 
 ## CI
 
-`analyze` can run unattended; `fix` cannot, because applying a patch always needs a person.
+`analyze` and `ci` run unattended; `fix` doesn't, because applying a patch always needs a person.
+Create a CI token at https://kwikqa.com/dashboard/account/ and store it as the secret
+`KWIKQA_TOKEN`. Running KwikQA with it approves its diagnosis requests, and each request's preview
+is printed to the job's log.
+
+### GitHub Actions
+
+`npx kwikqa ci` analyzes the failed run, writes `kwikqa-report.html`, and publishes a summary: one
+row per failure group with its owner and evidence grade, the flaky tests, and a pointer to the
+report. It holds no error text or evidence; the report does.
+
+```yaml
+permissions:
+  contents: read
+  pull-requests: write # only for the comment
+steps:
+  - uses: actions/checkout@v7
+  - run: npx playwright test
+  - if: failure()
+    run: npx kwikqa ci
+    env:
+      KWIKQA_TOKEN: ${{ secrets.KWIKQA_TOKEN }}
+      GITHUB_TOKEN: ${{ github.token }}
+      # Optional, see "Slack and Microsoft Teams" below
+      KWIKQA_SLACK_WEBHOOK_URL: ${{ secrets.KWIKQA_SLACK_WEBHOOK_URL }}
+      KWIKQA_TEAMS_WEBHOOK_URL: ${{ secrets.KWIKQA_TEAMS_WEBHOOK_URL }}
+  - if: failure()
+    uses: actions/upload-artifact@v7
+    with:
+      name: kwikqa-report
+      path: kwikqa-report.html
+```
+
+- **Job summary.** The summary is shown on the run's page.
+- **Pull-request comment.** On a pull request, the summary is also one comment. A re-run of the
+  same workflow job updates that comment instead of adding another, and a later run with no
+  failures updates it to say so. Each job of a workflow keeps its own comment; jobs of one matrix
+  share one, so analyze a sharded run once, after merging its reports.
+- **Pull requests from forks.** GitHub gives a fork's workflows no secrets and a read-only token.
+  KwikQA analyzes such a run locally, writes the report and the summary, and skips the comment.
+  Don't switch to `pull_request_target` to get the token: it runs the fork's code with your
+  repository's secrets.
+- **Never the cause of a failed build.** A comment or summary that can't be published is reported
+  in the log and skipped; the exit code is the analysis's. Without a KwikQA token, or past your
+  plan's limit, the run is analyzed locally and the summary says why it wasn't diagnosed.
+
+### Slack and Microsoft Teams
+
+`kwikqa ci` posts the run to the incoming webhooks you store as CI secrets. KwikQA keeps no webhook
+anywhere, and never prints one.
+
+- **Slack.** Create an incoming webhook for the channel, and store its URL as
+  `KWIKQA_SLACK_WEBHOOK_URL`.
+- **Teams.** In the channel, add the Workflows template "Post to a channel when a webhook request is
+  received", and store its URL as `KWIKQA_TEAMS_WEBHOOK_URL`. (Office 365 connector webhooks were
+  retired in May 2026.)
+
+The message names the repository and branch, links the pull request or commit and the run, and gives
+the counts. Then it has one line per failure group, with the group's category, where it fails, its
+size, and its owner, and the flaky count. It holds no error text, evidence, or source: a channel is
+a wider audience than a build log. KwikQA posts only when tests failed, unless you pass
+`--notify always`. A failed post is retried once, printed, and never fails the build.
+
+### Other CI systems
+
+`npx kwikqa ci` writes the report there too, and posts to Slack and Teams, but writes no summary or
+comment yet. Or run `analyze` directly:
 
 ```sh
-export KWIKQA_TOKEN=...   # create a CI token at https://kwikqa.com/dashboard/account/
-npx playwright test || npx kwikqa analyze --yes
-npx cypress run || npx kwikqa analyze --yes
+npx playwright test || npx kwikqa analyze --yes --html
+npx cypress run || npx kwikqa analyze --yes --html
 ```
 
+Then keep `kwikqa-report.html` as a build artifact.
+
 ## Account and billing
 
 The dashboard at https://kwikqa.com/dashboard/ shows your plan, usage, and recent requests. There