@plune-ai/cli 0.30.2 → 0.32.0

package/README.md

@@ -122,13 +122,13 @@ the last resort, used only when `PLUNE_TOKEN` is set nowhere else. Nothing write
 | `plune login` | Save a [Plune platform](https://plune.ai/platform) API token so `sync` and `ingest` can reach it. The token is **checked against the API before it is saved**, so a wrong one fails here rather than two commands later. Get one at `https://beta.plune.ai` → Account → API tokens. Flags: `--token <token>` (omit to paste it or pipe it via stdin), `--skip-verify` (save without checking, for offline setup). In CI, skip this step: every platform command reads `PLUNE_TOKEN` from the environment first, and the saved login only when it is not set. |
 | `plune logout` | Remove the saved token. |
 | `plune sync` | Upload the latest local run to the platform. Flags: `--file <path>` to send a specific run JSON, `--fail-on-failures` to read the run's summary back once after the upload and exit with [the gate](#a-release-gate---fail-on-failures)'s code. Each upload carries an `Idempotency-Key` (SHA-256 of the body), so sending the same run again returns the same run rather than creating a second one. Against a platform with `POST /v1/jobs` the upload is also reported as a job (kind `sync`, counting the run's rows) that ends completed, or failed with the error text if the sync fails; an older platform without the route is not told, and a job report never fails the sync: when the sync goes through, one the platform did not take is said in one line on stderr. |
-| `plune run import <file>` | Turn a **JUnit XML** or **Playwright JSON** report, or an **Allure `allure-results` folder**, into a run in Plune. Needs no provider key — nothing is generated. Flags: `--format junit\|playwright-json\|allure` (detected from the file when omitted; a folder is Allure), `--name <text>` to call the run something other than the [folder-based default](#what-the-run-is-called), `--key <externalKey>` to land several reports in one run (with `PLUNE_SHARED_RUN=1` — see below), `--create` to offer unmatched tests to the review queue, `--fail-on-missing-attachments` to exit 1 when a file a JUnit report names with `[[ATTACHMENT\|path]]`, or an Allure result names as a `source`, is not there (the results are sent either way), `--upload-concurrency <n>` for files uploaded at once ([1 to 32, default 16](#many-screenshots-upload-concurrency-and-retries)), `--fail-on-failures` to exit with [the gate](#a-release-gate---fail-on-failures)'s code on the summary the platform froze when this import finished the run. |
+| `plune run import <file \| folder>` | Turn a **JUnit XML** or **Playwright JSON** report, a **folder of JUnit XML files** (Maven Surefire's `target/surefire-reports`, Gradle's `build/test-results/test`), or an **Allure `allure-results` folder**, into a run in Plune. Needs no provider key — nothing is generated. Flags: `--format junit\|playwright-json\|allure` (detected when omitted: a folder with a `*-result.json` is Allure, any other folder is read as the JUnit `*.xml` files in it), `--name <text>` to call the run something other than the [folder-based default](#what-the-run-is-called), `--key <externalKey>` to land several reports in one run (with `PLUNE_SHARED_RUN=1` — see below), `--create` to offer unmatched tests to the review queue, `--fail-on-missing-attachments` to exit 1 when a file a JUnit report names with `[[ATTACHMENT\|path]]`, or an Allure result names as a `source`, is not there (the results are sent either way), `--upload-concurrency <n>` for files uploaded at once ([1 to 32, default 16](#many-screenshots-upload-concurrency-and-retries)), `--fail-on-failures` to exit with [the gate](#a-release-gate---fail-on-failures)'s code on the summary the platform froze when this import finished the run. |
 | `plune run start` | Open a platform run — or join the one already carrying `--key` — and print its id. Flags: `--key <externalKey>` (generated when absent), `--name <text>` to call the run something other than the [default](#what-the-run-is-called), `--json` for one machine-readable line. |
 | `plune run finish <id>` | Close a platform run. This is what the reporter tells you to do for a run it had to leave open. Flags: `--terminate` to record it as cut short, `--reason <text>`, `--fail-on-failures` to exit with [the gate](#a-release-gate---fail-on-failures)'s code on the summary the platform froze. |
 | `plune run exec -- <command>` | Open a run, run the command inside it, close the run — and exit with whatever the command returned. Sets `PLUNE_SHARED_RUN` for you, so anything reporting inside joins that run. Flags: `--key <externalKey>`, `--name <text>` (as on `run start`), `--fail-on-failures` to exit with [the gate](#a-release-gate---fail-on-failures)'s code instead — or with the command's own when no result reached the run. |
 | `plune run delete <id>` | Delete a run and everything it produced — its results and the review-queue entries it raised. Approved test cases and the audit log stay. **Recoverable for 30 days** (an owner of the project can put it back from Settings → Bin), then gone for good. No prompt: it is your data, and this command belongs in scripts. |
 | `plune run report` | Replay `.plune/pending-results.jsonl` — send what the reporter could not. The lines of one reporter run go back as one run, with the commit and branch the reporter recorded. Once every result has gone the file is renamed to `pending-results.<time>.sent.jsonl`; if anything did not go, it stays as it was. Flags: `--file <path>`. |
-| `plune ingest [dir]` | Record a [Cairn](https://github.com/plune-ai/cairn) run in Plune. Omit `[dir]` for the newest run under `./runs`, or name the directory holding `report.json`. Generated cases arrive as **review proposals** — nothing is created until a person approves it. |
+| `plune ingest [dir]` | Record a [Cairn](https://github.com/plune-ai/cairn) run in Plune. Omit `[dir]` for the newest run under `./runs`, or name the directory holding `report.json`. Generated cases arrive as **review proposals** — nothing is created until a person approves it. Against a platform with `POST /v1/jobs` the ingest is also reported as a job (kind `ingest`, counting the cases sent, linked to the run it landed in) that ends completed, or failed with the error text if the ingest fails; as with `sync`, an older platform is not told and a job report never fails the ingest or changes its exit code. |
 | `plune pull [file]` | Write the project's test cases as **one Markdown document** — Testomat's classical format, plus Plune's own columns — to `plune/cases.md` (or `[file]`). Refuses to overwrite a file git sees as modified; `--force` overrides, `--suite <id>` takes one suite or folder and what is under it. |
 | `plune push [file] [--dry-run]` | Send the document back. Cases are matched by `id`; a block without one becomes a draft; an unknown id is refused by line; **nothing is deleted**. Prints the report — created, updated, unchanged, refused, warnings — and with `--dry-run` writes nothing. Exit `4` when anything was refused. |
 | `plune requirements import <file.md>` | Import requirements from a **Markdown file** (an SRS, a PRD, a page exported from a wiki). With no mapping flags it lists the tables and sections the file has and exits `0`; with `--table` or `--sections` it is a **dry run** that writes nothing until you add `--yes`. See [Requirements from a Markdown file](#requirements-from-a-markdown-file). Needs a Plune platform that serves `POST /v1/projects/:id/requirements/import` (plune-ai/plune#1621). |
@@ -239,6 +239,7 @@ costs nothing beyond a Plune token.
 ```bash
 plune login                           # once, on your machine; in CI export PLUNE_TOKEN instead
 plune run import ./junit.xml          # Jest, Vitest, pytest, PHPUnit, Surefire, Cypress, …
+plune run import target/surefire-reports   # or a folder of JUnit files: Maven, Gradle — see Java below
 plune run import ./playwright.json    # or Playwright's own JSON report
 plune run import ./allure-results     # or the folder an Allure adapter wrote — see below
 ```
@@ -301,13 +302,15 @@ pytest --junitxml=report.xml -o junit_logging=system-out
 plune run import report.xml
 ```
 
-Java, JUnit 5 (Maven Surefire writes the test's output into its testcase):
+Java, JUnit 5 (Maven Surefire writes the test's output into its testcase; Gradle only with
+`reports.junitXml.outputPerTestCase = true`). The path counts from the folder the XML file is in
+(`target/surefire-reports`, `build/test-results/test`), so the file has to be below it:
 
 ```java
 @Test
 void checkout() {
-    // ...
-    System.out.println("[[ATTACHMENT|target/artifacts/checkout.png]]");
+    // ... save the screenshot as target/surefire-reports/artifacts/checkout.png
+    System.out.println("[[ATTACHMENT|artifacts/checkout.png]]");
 }
 ```
 
@@ -330,7 +333,7 @@ For Jest, use Allure below.
 
 Every Allure adapter (pytest, JUnit 5, TestNG, Playwright, Cypress, Jest, Vitest, WebdriverIO, .NET) writes
 the same folder: a `*-result.json` per test, a `*-container.json` per suite, and the files the tests attached.
-Point `plune run import` at the folder (a folder is read as Allure, no flag needed). Each test arrives with its
+Point `plune run import` at the folder (a folder with a `*-result.json` in it is read as Allure, no flag needed). Each test arrives with its
 name, its status as Allure shows it (`passed`, `failed`, `broken`, `skipped`), its time, the failure message and
 trace, and its steps; a test run again (same `historyId`) is one test with attempts, the last one counting.
 The files come from the test, from its steps and from its fixtures (`befores`, `afters`), each with the
@@ -354,11 +357,36 @@ def test_checkout(page):
         allure.attach.file("artifacts/checkout.png", name="checkout", attachment_type=allure.attachment_type.PNG)
 ```
 
-Java, JUnit 5 (`allure-junit5`, with the AspectJ agent from Allure's setup; the results folder is
-`target/allure-results` unless `allure.results.directory` says otherwise):
+Java. Maven Surefire and Gradle write JUnit XML on their own, a `TEST-<class>.xml` per class, so a Java suite
+needs no Allure: point the import at the folder.
 
 ```bash
 mvn test
+plune run import target/surefire-reports
+
+./gradlew test
+plune run import build/test-results/test
+```
+
+Every `*.xml` in the folder that is a JUnit report is read, in the order of the file names, and together they make
+one run. The folder's other files and its subfolders (Gradle's `binary/`) are not opened; an `*.xml` that is not a
+JUnit report is skipped with a line in the output. Each test arrives with its class and name as the key
+(`cart.CartTest#addsAnItem` from Surefire; Gradle keeps the brackets, `cart.CartTest#addsAnItem()`), its status
+(`pass`, `failure`, `error`, `skipped`), the failure's message, type and stack trace, and its duration when the suite
+has a `timestamp`. Gradle writes one; Surefire's report has none, so Maven tests arrive without a duration for now
+(plune-ai/plune#2310). A file reaches a test only
+through a `[[ATTACHMENT|path]]` line in its own output, [as above](#files-from-a-junit-report), the path counted from
+the folder the XML file is in. Gradle puts that output in the test's testcase only with
+`reports.junitXml.outputPerTestCase = true` (by default it writes it once per suite).
+
+Allure adds the steps, and files attached without printing a line. The JUnit 5 adapter is
+`io.qameta.allure:allure-jupiter` (from 2.35.1, `allure-junit5` is only a relocation to it, with no jar), with the
+AspectJ agent from Allure's setup. It writes `allure-results` in the working folder; to have it in the build's
+output, set `allure.results.directory` (in `src/test/resources/allure.properties`, or as a system property) to
+`target/allure-results` for Maven or `build/allure-results` for Gradle:
+
+```bash
+mvn test                                  # with allure.results.directory=target/allure-results
 plune run import target/allure-results
 ```
 
@@ -372,11 +400,13 @@ void checkout() {
 Jest has no per-test output in JUnit; `allure-jest` gives it one (`allure.attachment(...)` in a test or a step).
 Tests are matched to Plune cases by the same key as a JUnit report's, `classname#name`, where the adapter's
 names allow it (pytest, JUnit 5, Playwright); for Vitest, Jest and Cypress the key is the runner, the file and the
-test's full name (`vt:test/cart.test.js#cart adds an item`, `jest:…`, `cy:…`). A file whose name starts with `_`
+test's full name (`vt:test/cart.test.js#cart adds an item`, `jest:…`, `cy:…`). For JUnit 5 the name carries the
+method's parentheses in Allure and in Gradle's JUnit XML (`shop.CartTest#coupon()`) but not in Surefire's
+(`shop.CartTest#coupon`), so a project that moves between Allure and Surefire meets its cases under other keys. A file whose name starts with `_`
 is not sent: the core hides it as Playwright's own reports do, and Allure has no such convention, so name
 attachments without a leading underscore.
 
-Check that the output reaches the report: Gradle writes it only with `reports.junitXml.outputPerTestCase = true`,
+For a JUnit report, check that the output reaches the report: Gradle writes it only with `reports.junitXml.outputPerTestCase = true`,
 and Maven Surefire moves it out of the XML when `redirectTestOutputToFile` is `true`. As in the Playwright
 reporter, a file whose name starts with an underscore is skipped, and a file several tests name is uploaded once per test.
 
@@ -424,6 +454,37 @@ The shard that is the Nth also prints `plune: run closed` just above that line,
   and leaves the run open, as `PLUNE_SHARED_RUN` does: close it with `plune run finish`.
 - With `--fail-on-failures`, the shard that closes the run gates on its summary; the others have no verdict yet.
 
+#### A combination per job of a matrix: `PLUNE_CONFIGURATION`
+
+The jobs of a matrix of browsers and systems report into one run, each under its own combination of
+configuration values (plune-ai/plune#1251):
+
+```yaml
+strategy:
+  matrix:
+    browser: [Chrome, Firefox]
+    os: [Windows, macOS]
+env:
+  PLUNE_RUN: ${{ github.run_id }}-${{ github.run_attempt }}
+  PLUNE_SHARED_RUN: 1
+  PLUNE_CONFIGURATION: "Browser=${{ matrix.browser }};Operating system=${{ matrix.os }}"
+```
+
+`PLUNE_CONFIGURATION` is `Group=Value` pairs split by `;`, with the names of the groups and values as in
+Settings → Configurations. Case does not matter, spaces around a name or a value are dropped, and so is an
+empty pair (`;;`, a `;` at the end); a value may hold `=`, since a pair splits at its first one. Every results
+batch of `plune run import` and of the Playwright reporter carries it, and Plune files the results under that
+combination of the run: one the run already has (a run launched from the dashboard with configurations, joined
+by its key), or one it adds. A value that is not in Settings is taken as sent, marked "not in the list". Unset
+or blank, the batch is sent as before. The run is closed once at the end, as above.
+
+- A pair without `=`, without a group or without a value, or one group named twice, is a configuration error:
+  nothing is sent and the command fails.
+- Every job of a run names the same groups. Plune refuses with 422 a batch that names other groups, or one that
+  would take the run past its limits (25 combinations, 40 000 results to expect); its message is printed as it
+  came, and the batch goes to the fallback file like any other refused batch.
+- The fallback file keeps the combination, and `plune run report` sends the batch back into it.
+
 ### A time limit for the import
 
 The whole `plune run import` stops after 600 seconds by default, so a platform that stops answering cannot hold a CI job until its own timeout kills it. Raise it for a very large report, or turn it off with `0`: