@itmetsam/samurai-framework 0.12.1 → 0.15.1

package/README.md

@@ -19,12 +19,12 @@ test("Visitor can sign in", async ({ page, env, secrets }) => {
 - **Auto-waiting.** Actions wait until the element is attached, visible, stable, enabled and not covered. Assertions retry until they pass or time out.
 - **Semantic locators.** By role, label, text, test id, CSS or XPath, with ordered fallbacks.
 - **Environments and secrets.** One test file runs against `dev`, `staging` or `production`; secrets come from the environment or a `.env.<name>` file and are masked in logs and reports.
-- **Parallel runs.** Tests run side by side in worker processes (`--workers`, `workers` in the config; half the CPUs by default), and a flow's Parallel node runs its branches at the same time (see [Running tests and reports](docs/running-and-reports.md#parallel-runs)).
+- **Parallel runs.** Tests run side by side in worker processes (`--workers`, `workers` in the config; half the CPUs by default), and a flow's Parallel node runs its branches at the same time (see [Running tests and reports](https://github.com/degrootsam/samurai-framework/wiki/Running-tests-and-reports#parallel-runs)).
 - **Network control.** Mock, block or modify requests; wait for requests, responses and network idle; read response bodies.
-- **Flows.** Chain tests and groups into a graph with conditions, variables, waits and HTTP requests, and run it from the command line or CI (see [Flows](docs/flows.md)).
-- **Recorder building blocks.** Specs are parsed into steps and edited back with minimal diffs, and an engine records what a person does in the browser. A toolbar in the recorded page adds assertions, variables and secrets, waits, screenshots and pause (see [Steps and the recorder](docs/recorder.md)). The Electron app that ties them together lives in a separate repository (see the [roadmap](docs/roadmap.md)).
+- **Flows.** Chain tests and groups into a graph with conditions, variables, waits and HTTP requests, and run it from the command line or CI (see [Flows](https://github.com/degrootsam/samurai-framework/wiki/Flows)).
+- **Recorder building blocks.** Specs are parsed into steps and edited back with minimal diffs, and an engine records what a person does in the browser. A toolbar in the recorded page adds assertions, variables and secrets, waits, screenshots and pause (see [Steps and the recorder](https://github.com/degrootsam/samurai-framework/wiki/Steps-and-the-recorder)). The Electron app that ties them together lives in a separate repository (see the [roadmap](https://github.com/degrootsam/samurai-framework/wiki/Roadmap)).
 
-> **Status:** early. Firefox only. See [what is still open](docs/roadmap.md).
+> **Status:** early. Firefox only. See [what is still open](https://github.com/degrootsam/samurai-framework/wiki/Roadmap).
 
 ## Getting started
 
@@ -39,14 +39,14 @@ npx samurai browsers install firefox             # once per machine
 npx samurai run
 ```
 
-Specs and the config then import from `@itmetsam/samurai-framework` instead of the relative paths used below. See [Command line](docs/cli.md).
+Specs and the config then import from `@itmetsam/samurai-framework` instead of the relative paths used below. See [Command line](https://github.com/degrootsam/samurai-framework/wiki/Command-line).
 
 The rest of this section works in a checkout of this repository.
 
 ### 1. Requirements
 
 - [Node.js](https://nodejs.org) 24 or newer (`engines` in `package.json`; CI uses 26) and, to work on the framework, [Bun](https://bun.sh) 1.3 or newer
-- Firefox, downloaded by the framework: run `bun run dev browsers install firefox` once in this checkout (`npx samurai browsers install firefox` in a project; see [Which browser runs](docs/pages-and-browsers.md#which-browser-runs)). The Firefox installed on the machine is not used.
+- Firefox, downloaded by the framework: run `bun run dev browsers install firefox` once in this checkout (`npx samurai browsers install firefox` in a project; see [Which browser runs](https://github.com/degrootsam/samurai-framework/wiki/Pages-and-browsers#which-browser-runs)). The Firefox installed on the machine is not used.
 
 ### 2. Install
 
@@ -103,13 +103,13 @@ bun run dev --env production   # another environment (or SAMURAI_ENV=production)
 bun run dev --timeout 60000    # per-test timeout in ms; also --expect-timeout
 ```
 
-`bun run dev` is `samurai run` straight from the sources, so it takes the same options (`--grep`, `--file`, `--group`, `--headless`, `--json`, …) and the other commands too (`bun run dev list`, `bun run dev browsers install firefox`). See [Command line](docs/cli.md).
+`bun run dev` is `samurai run` straight from the sources, so it takes the same options (`--grep`, `--file`, `--group`, `--headless`, `--json`, …) and the other commands too (`bun run dev list`, `bun run dev browsers install firefox`). See [Command line](https://github.com/degrootsam/samurai-framework/wiki/Command-line).
 
-A browser window opens for each test and closes when the test ends (`--headless` hides it; on CI it is the default, see [Configuration](docs/configuration.md#headless)). Each test's result is printed as it finishes, followed by a summary, and the full report is written to `result/report.json`. The process exits with code 0 when every test passed, 1 otherwise and 2 for a usage or configuration error. Framework logs go to stderr and `logs/`.
+A browser window opens for each test and closes when the test ends (`--headless` hides it; on CI it is the default, see [Configuration](https://github.com/degrootsam/samurai-framework/wiki/Configuration#headless)). Each test's result is printed as it finishes, followed by a summary, and the full report is written to `result/report.json`. The process exits with code 0 when every test passed, 1 otherwise and 2 for a usage or configuration error. Framework logs go to stderr and `logs/`.
 
 ### Flows (optional)
 
-A project can also keep flows in `flows/*.flow.json` (edited in the Samurai app) that chain tests and groups with conditions and variables. `samurai flow check` validates them and `samurai flow run <flow>` runs them, with a report in `result/flows/`. See [Flows](docs/flows.md).
+A project can also keep flows in `flows/*.flow.json` (edited in the Samurai app) that chain tests and groups with conditions and variables. `samurai flow check` validates them and `samurai flow run <flow>` runs them, with a report in `result/flows/`. See [Flows](https://github.com/degrootsam/samurai-framework/wiki/Flows).
 
 ### 6. Secrets (optional)
 
@@ -119,29 +119,29 @@ cp .env.example .env.staging
 # SAMURAI_SECRET_TEST_PASSWORD=...
 ```
 
-Read it in a test as `secrets.TEST_PASSWORD`. Real environment variables win over the file, and `.env.*` files are git-ignored. See [Environments and secrets](docs/environments-and-secrets.md).
+Read it in a test as `secrets.TEST_PASSWORD`. Real environment variables win over the file, and `.env.*` files are git-ignored. See [Environments and secrets](https://github.com/degrootsam/samurai-framework/wiki/Environments-and-secrets).
 
 ## Documentation
 
-| Guide                                                        | What it covers                                                                                        |
-| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- |
-| [Writing tests](docs/writing-tests.md)                       | `test`, `describe`, fixtures, how a test passes or fails                                              |
-| [Locators](docs/locators.md)                                 | Finding elements, chaining, fallbacks, actions and their auto-waiting                                 |
-| [Assertions](docs/assertions.md)                             | `expect` for elements and for plain values                                                            |
-| [Pages and browsers](docs/pages-and-browsers.md)             | Navigation, waits, screenshots, PDF, viewport, emulation, dialogs, downloads, contexts, cookies       |
-| [Network](docs/network.md)                                   | Waiting for traffic, mocking, blocking, response bodies                                               |
-| [Configuration](docs/configuration.md)                       | Every option in `samurai.config.ts`, CLI flags and precedence                                         |
-| [Environments and secrets](docs/environments-and-secrets.md) | Per-environment settings, variables, secrets and masking                                              |
-| [Command line](docs/cli.md)                                  | `samurai run`, `list`, `record` and `init`, options, output, exit codes, `--json`                     |
-| [Flows](docs/flows.md)                                       | `samurai flow list`, `check` and `run`: graphs of tests, groups and steps, reports, CI                |
-| [Writing a step](docs/steps.md)                              | `defineStep`: flow steps of your own, the built-in steps                                              |
-| [Running tests and reports](docs/running-and-reports.md)     | What a run does, `report.json`, page logs, timeouts                                                   |
-| [Steps and the recorder](docs/recorder.md)                   | The step codec, the recording engine and the recorder toolbar                                         |
-| [Embedding the framework](docs/embedding.md)                 | Using the package from another app (such as the Electron UI): build, entry points, `runTests`, events |
-| [Contributing](docs/contributing.md)                         | Project layout, running the framework's own tests, conventions                                        |
-| [Roadmap](docs/roadmap.md)                                   | What is built and what is still open                                                                  |
-
-The [documentation index](docs/README.md) lists every guide by topic.
+| Guide                                                                                                       | What it covers                                                                                        |
+| ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
+| [Writing tests](https://github.com/degrootsam/samurai-framework/wiki/Writing-tests)                         | `test`, `describe`, fixtures, how a test passes or fails                                              |
+| [Locators](https://github.com/degrootsam/samurai-framework/wiki/Locators)                                   | Finding elements, chaining, fallbacks, actions and their auto-waiting                                 |
+| [Assertions](https://github.com/degrootsam/samurai-framework/wiki/Assertions)                               | `expect` for elements and for plain values                                                            |
+| [Pages and browsers](https://github.com/degrootsam/samurai-framework/wiki/Pages-and-browsers)               | Navigation, waits, screenshots, PDF, viewport, emulation, dialogs, downloads, contexts, cookies       |
+| [Network](https://github.com/degrootsam/samurai-framework/wiki/Network)                                     | Waiting for traffic, mocking, blocking, response bodies                                               |
+| [Configuration](https://github.com/degrootsam/samurai-framework/wiki/Configuration)                         | Every option in `samurai.config.ts`, CLI flags and precedence                                         |
+| [Environments and secrets](https://github.com/degrootsam/samurai-framework/wiki/Environments-and-secrets)   | Per-environment settings, variables, secrets and masking                                              |
+| [Command line](https://github.com/degrootsam/samurai-framework/wiki/Command-line)                           | `samurai run`, `list`, `record` and `init`, options, output, exit codes, `--json`                     |
+| [Flows](https://github.com/degrootsam/samurai-framework/wiki/Flows)                                         | `samurai flow list`, `check` and `run`: graphs of tests, groups and steps, reports, CI                |
+| [Writing a step](https://github.com/degrootsam/samurai-framework/wiki/Writing-a-step)                       | `defineStep`: flow steps of your own, the built-in steps                                              |
+| [Running tests and reports](https://github.com/degrootsam/samurai-framework/wiki/Running-tests-and-reports) | What a run does, `report.json`, page logs, timeouts                                                   |
+| [Steps and the recorder](https://github.com/degrootsam/samurai-framework/wiki/Steps-and-the-recorder)       | The step codec, the recording engine and the recorder toolbar                                         |
+| [Embedding the framework](https://github.com/degrootsam/samurai-framework/wiki/Embedding-the-framework)     | Using the package from another app (such as the Electron UI): build, entry points, `runTests`, events |
+| [Contributing](docs/contributing.md)                                                                        | Project layout, running the framework's own tests, conventions                                        |
+| [Roadmap](https://github.com/degrootsam/samurai-framework/wiki/Roadmap)                                     | What is built and what is still open                                                                  |
+
+The [documentation index](https://github.com/degrootsam/samurai-framework/wiki/Home) lists every guide by topic.
 
 ## Scripts
 

package/dist/api.d.ts

@@ -1,6 +1,6 @@
-export { test, describe } from "./runner/registry.js";
+export { test, describe, beforeEach, afterEach, beforeAll, afterAll, } from "./runner/registry.js";
 export { expect } from "./assert/expect.js";
 export { defineConfig } from "./config/config.js";
 export { defineStep, type StepDefinition, type StepResult, type StepRunInput, } from "./flows/steps/index.js";
-export type { TestFixtures } from "./types/test.js";
+export type { AllHookFixtures, TestFixtures } from "./types/test.js";
 export { generatedFile, type GeneratedFile, type FileInput, } from "./browser/generated-files.js";

package/dist/api.js

@@ -1,4 +1,4 @@
-export { test, describe } from "./runner/registry.js";
+export { test, describe, beforeEach, afterEach, beforeAll, afterAll, } from "./runner/registry.js";
 export { expect } from "./assert/expect.js";
 export { defineConfig } from "./config/config.js";
 export { defineStep, } from "./flows/steps/index.js";

package/dist/cli/args.d.ts

@@ -12,6 +12,7 @@ export declare function parseCommandLine(argv: string[]): {
         headless?: boolean | undefined;
         port?: string | undefined;
         workers?: string | undefined;
+        retries?: string | undefined;
         grep?: string | undefined;
         group?: string | undefined;
         browser?: string[] | undefined;
@@ -40,13 +41,15 @@ export interface RunnerFlags {
     port?: number;
     /** `--workers`: a whole number or a percentage like `50%`, as typed */
     workers?: string;
+    /** `--retries`: a whole number from 0, as typed */
+    retries?: string;
     grep?: string;
     group?: string;
     files?: string[];
     /** `--browser`, repeated for more than one; replaces the config's `browsers` for this run */
     browsers?: SupportedBrowser[];
 }
-/** Reads `--headless`, `--port <n>`, `--workers <n|n%>`, `--grep <text>`, `--group <name>`, `--file <spec>` and `--browser <name>` (both repeatable) from `argv`, or from values that were parsed already */
+/** Reads `--headless`, `--port <n>`, `--workers <n|n%>`, `--retries <n>`, `--grep <text>`, `--group <name>`, `--file <spec>` and `--browser <name>` (both repeatable) from `argv`, or from values that were parsed already */
 export declare function parseRunnerFlags(source: string[] | CommandValues): RunnerFlags;
 /** Reads `--env`, `--timeout` and `--expect-timeout` from `argv` (or values parsed already); the environment falls back to SAMURAI_ENV */
 export declare function parseRunOverrides(source: string[] | CommandValues, env?: Partial<NodeJS.ProcessEnv>): EnvAndTimeouts;

package/dist/cli/args.js

@@ -1,5 +1,5 @@
 import { parseArgs } from "node:util";
-import { checkBrowserList, parseWorkers, } from "../runner/run.js";
+import { checkBrowserList, parseRetries, parseWorkers, } from "../runner/run.js";
 const FLAGS = {
     env: { type: "string" },
     timeout: { type: "string" },
@@ -7,6 +7,7 @@ const FLAGS = {
     headless: { type: "boolean" },
     port: { type: "string" },
     workers: { type: "string" },
+    retries: { type: "string" },
     grep: { type: "string" },
     group: { type: "string" },
     browser: { type: "string", multiple: true },
@@ -34,7 +35,7 @@ export function parseCommandLine(argv) {
     });
 }
 const valuesOf = (source) => Array.isArray(source) ? parseCommandLine(source).values : source;
-/** Reads `--headless`, `--port <n>`, `--workers <n|n%>`, `--grep <text>`, `--group <name>`, `--file <spec>` and `--browser <name>` (both repeatable) from `argv`, or from values that were parsed already */
+/** Reads `--headless`, `--port <n>`, `--workers <n|n%>`, `--retries <n>`, `--grep <text>`, `--group <name>`, `--file <spec>` and `--browser <name>` (both repeatable) from `argv`, or from values that were parsed already */
 export function parseRunnerFlags(source) {
     const values = valuesOf(source);
     const flags = {};
@@ -51,6 +52,10 @@ export function parseRunnerFlags(source) {
         parseWorkers(values.workers, "--workers");
         flags.workers = values.workers;
     }
+    if (values.retries !== undefined) {
+        parseRetries(values.retries, "--retries");
+        flags.retries = values.retries;
+    }
     if (values.grep !== undefined)
         flags.grep = values.grep;
     if (values.group !== undefined)

package/dist/cli/console-reporter.js

@@ -1,5 +1,13 @@
 import { painter } from "./paint.js";
 const seconds = (ms) => `${(ms / 1000).toFixed(1)}s`;
+/** What a result says about its retries: `passed on retry 2`, `failed after 2 retries`, or nothing */
+function retried(result) {
+    if (!("retry" in result) || !result.retry)
+        return undefined;
+    if (result.status === "success")
+        return `passed on retry ${result.retry}`;
+    return `failed after ${result.retry} ${result.retry === 1 ? "retry" : "retries"}`;
+}
 /** Prints a run for a person: one line per test as it ends, then every failure in full and a count */
 export class ConsoleReporter {
     constructor({ write, color }) {
@@ -30,14 +38,17 @@ export class ConsoleReporter {
     }
     line(name, result, browser) {
         const took = "duration" in result
-            ? ` ${this.paint(90, `(${seconds(result.duration)})`)}`
+            ? ` ${this.paint(90, `(${[seconds(result.duration), retried(result)].filter(Boolean).join(", ")})`)}`
             : "";
+        if (result.status === "skipped")
+            return this.paint(90, `- ${name}${this.tag(browser)} (${result.annotation === "fixme" ? "fixme" : "skipped"}${result.reason ? `: ${result.reason}` : ""})`);
         return result.status === "success"
             ? `${this.paint(32, "✔")} ${name}${this.tag(browser)}${took}`
             : `${this.paint(31, "✖")} ${name}${this.tag(browser)}${took}`;
     }
     summary(summary) {
-        const failed = summary.tests.filter((test) => test.status !== "success");
+        const skipped = summary.tests.filter((test) => test.status === "skipped");
+        const failed = summary.tests.filter((test) => test.status !== "success" && test.status !== "skipped");
         for (const test of failed) {
             if (!("name" in test))
                 continue;
@@ -50,10 +61,12 @@ export class ConsoleReporter {
             if ("file" in test)
                 this.write(`    ${this.paint(90, test.file)}\n`);
         }
-        const passed = summary.tests.length - failed.length;
+        const passed = summary.tests.length - failed.length - skipped.length;
+        const flaky = summary.tests.filter((test) => "flaky" in test && test.flaky).length;
         const parts = [
             failed.length > 0 && this.paint(31, `${failed.length} failed`),
-            this.paint(32, `${passed} passed`),
+            this.paint(32, `${passed} passed${flaky > 0 ? ` (${flaky} flaky)` : ""}`),
+            skipped.length > 0 && this.paint(90, `${skipped.length} skipped`),
         ].filter(Boolean);
         this.write(`\n${parts.join(", ")} ${this.paint(90, `(${seconds(summary.duration)})`)}\n`);
     }

package/dist/cli/flow.js

@@ -25,6 +25,7 @@ Options for run:
   --headless, --no-headless   Run the browser without a window, or with one
   --port <n>                  Browser debugging port (default ${DEFAULT_PORT})
   --workers <n|n%>            Worker processes the tests share (or SAMURAI_WORKERS; default half the CPUs)
+  --retries <n>               Run a failing test again up to n times, each with a fresh browser (or SAMURAI_RETRIES; default 0)
 
 Each run writes result/flows/<id>.json.
 Exit code: 0 when every node that should run passed, 1 when a node failed or the run was cancelled,
@@ -228,6 +229,10 @@ export async function flowCommand(argv, io) {
                     ...(io.env.SAMURAI_WORKERS && {
                         workersEnv: io.env.SAMURAI_WORKERS,
                     }),
+                    ...(flags.retries !== undefined && { retries: flags.retries }),
+                    ...(io.env.SAMURAI_RETRIES && {
+                        retriesEnv: io.env.SAMURAI_RETRIES,
+                    }),
                 },
             });
         }

package/dist/cli/main.js

@@ -48,6 +48,7 @@ export async function main(argv, io) {
             ...(flags.headless !== undefined && { headless: flags.headless }),
             ...(flags.port !== undefined && { port: flags.port }),
             ...(flags.workers !== undefined && { workers: flags.workers }),
+            ...(flags.retries !== undefined && { retries: flags.retries }),
             ...(flags.grep !== undefined && { grep: flags.grep }),
             ...(flags.group !== undefined && { group: flags.group }),
             ...(flags.files && { files: flags.files }),
@@ -59,7 +60,7 @@ export async function main(argv, io) {
                 io.out(`${JSON.stringify(tests, null, 2)}\n`);
             else
                 for (const found of tests)
-                    io.out(`${found.name}\n    ${found.file}\n`);
+                    io.out(`${found.name}${found.modifier ? ` [${found.modifier}]` : ""}\n    ${found.file}\n`);
             return 0;
         }
         const human = values.json

package/dist/cli/usage.d.ts

@@ -1 +1 @@
-export declare const USAGE = "samurai: browser tests over WebDriver BiDi\n\nUsage:\n  samurai run [options]       Run the tests (the default command)\n  samurai list [options]      List the tests without running them\n  samurai record <spec>       Open a browser and record what you do into a spec file\n  samurai init [folder]       Scaffold a project: config, an example test, .env.example\n  samurai flow <command>      List, check and run the flows in ./flows (see below)\n  samurai browsers [command]  List, install and remove the browsers SAMURAI launches\n\nOptions for run and list:\n  --env <name>                Environment to run against (or SAMURAI_ENV)\n  --file <spec>               Only this spec file; repeatable\n  --grep <text>               Only tests whose full name contains the text\n  --group <name>              Only the tests of this group of the config's groups\n  --browser <name>            Run in this browser (firefox or chrome) instead of the config's\n                              browsers; repeat it for more than one\n  --json                      Machine-readable output: JSON lines for run, one array for list\n\nOptions for run:\n  --timeout <ms>              Time one test may take\n  --expect-timeout <ms>       Time actions and assertions retry\n  --headless, --no-headless   Run the browser without a window, or with one.\n                              Default: the config's use.headless, else headless when CI is set\n  --port <n>                  Browser debugging port (default 9223)\n  --workers <n|n%>            Tests that run at the same time, each in a worker process: a number\n                              from 1, or a share of the CPUs like 50%. Default: the config's\n                              workers (or SAMURAI_WORKERS), else half the CPUs\n\nOptions for record (also --env, --port, --json, --headless, --no-headless):\n  --browser <name>            Record in this browser (default: the first of the config's browsers)\n  --new <title>               Add a test with this title (creates the spec file if needed)\n  --test <name|index>         Record into this test; not needed when the file has one\n  --at <n>                    Insert from step n (default: after the last step)\n  --url <url>                 Page to start on (default: the environment's baseURL)\n  Stop with Ctrl+C or by closing the browser window.\n\n  -h, --help                  Show this help\n  -v, --version               Show the version\n\nExit code: 0 when every test passed, 1 when a test failed, 2 for a usage or configuration error.\n\nsamurai flow:\n  list                        The flows in ./flows\n  check [<flow>...]           Check flows without running them\n  run <flow>... | --all       Run flows; each writes result/flows/<id>.json\n  Options for run: --env, --headless, --timeout, --expect-timeout, --port, --json, --allow-unsupported\n  Run `samurai flow` alone for the details.\n\nsamurai browsers:\n  list                        The pinned browsers and every build in the cache (the default)\n  install [firefox|chrome|all]\n                              Download the pinned builds (default: all)\n  uninstall <name> [--build <id>] | --unused\n\nRun it in the project folder, the one holding samurai.config.ts.\n";
+export declare const USAGE = "samurai: browser tests over WebDriver BiDi\n\nUsage:\n  samurai run [options]       Run the tests (the default command)\n  samurai list [options]      List the tests without running them\n  samurai record <spec>       Open a browser and record what you do into a spec file\n  samurai init [folder]       Scaffold a project: config, an example test, .env.example\n  samurai flow <command>      List, check and run the flows in ./flows (see below)\n  samurai browsers [command]  List, install and remove the browsers SAMURAI launches\n\nOptions for run and list:\n  --env <name>                Environment to run against (or SAMURAI_ENV)\n  --file <spec>               Only this spec file; repeatable\n  --grep <text>               Only tests whose full name contains the text\n  --group <name>              Only the tests of this group of the config's groups\n  --browser <name>            Run in this browser (firefox or chrome) instead of the config's\n                              browsers; repeat it for more than one\n  --json                      Machine-readable output: JSON lines for run, one array for list\n\nOptions for run:\n  --timeout <ms>              Time one test may take\n  --expect-timeout <ms>       Time actions and assertions retry\n  --headless, --no-headless   Run the browser without a window, or with one.\n                              Default: the config's use.headless, else headless when CI is set\n  --port <n>                  Browser debugging port (default 9223)\n  --workers <n|n%>            Tests that run at the same time, each in a worker process: a number\n                              from 1, or a share of the CPUs like 50%. Default: the config's\n                              workers (or SAMURAI_WORKERS), else half the CPUs\n  --retries <n>               Run a failing test again up to n times, each with a fresh browser\n                              (or SAMURAI_RETRIES; default 0)\n\nOptions for record (also --env, --port, --json, --headless, --no-headless):\n  --browser <name>            Record in this browser (default: the first of the config's browsers)\n  --new <title>               Add a test with this title (creates the spec file if needed)\n  --test <name|index>         Record into this test; not needed when the file has one\n  --at <n>                    Insert from step n (default: after the last step)\n  --url <url>                 Page to start on (default: the environment's baseURL)\n  Stop with Ctrl+C or by closing the browser window.\n\n  -h, --help                  Show this help\n  -v, --version               Show the version\n\nExit code: 0 when every test passed, 1 when a test failed, 2 for a usage or configuration error.\n\nsamurai flow:\n  list                        The flows in ./flows\n  check [<flow>...]           Check flows without running them\n  run <flow>... | --all       Run flows; each writes result/flows/<id>.json\n  Options for run: --env, --headless, --timeout, --expect-timeout, --port, --json, --allow-unsupported\n  Run `samurai flow` alone for the details.\n\nsamurai browsers:\n  list                        The pinned browsers and every build in the cache (the default)\n  install [firefox|chrome|all]\n                              Download the pinned builds (default: all)\n  uninstall <name> [--build <id>] | --unused\n\nRun it in the project folder, the one holding samurai.config.ts.\n";

package/dist/cli/usage.js

@@ -27,6 +27,8 @@ Options for run:
   --workers <n|n%>            Tests that run at the same time, each in a worker process: a number
                               from 1, or a share of the CPUs like 50%. Default: the config's
                               workers (or SAMURAI_WORKERS), else half the CPUs
+  --retries <n>               Run a failing test again up to n times, each with a fresh browser
+                              (or SAMURAI_RETRIES; default 0)
 
 Options for record (also --env, --port, --json, --headless, --no-headless):
   --browser <name>            Record in this browser (default: the first of the config's browsers)

package/dist/config/config.js

@@ -2,6 +2,8 @@ import path from "path";
 import logger from "../logger/index.js";
 import { pathToFileURL } from "url";
 import { validateBrowserConfig } from "./browsers.js";
+import { validateForbidOnlyConfig } from "./forbid-only.js";
+import { validateRetriesConfig } from "./retries.js";
 import { validateWorkersConfig } from "./workers.js";
 let activeProject;
 /** Makes `project` the one config, secrets, reports and downloads are resolved for; `undefined` ends it */
@@ -27,6 +29,8 @@ export async function loadConfig() {
     if (activeProject?.config) {
         validateBrowserConfig(activeProject.config);
         validateWorkersConfig(activeProject.config);
+        validateForbidOnlyConfig(activeProject.config);
+        validateRetriesConfig(activeProject.config);
         return activeProject.config;
     }
     const configUrl = path.join(projectDir(), "samurai.config.ts");
@@ -37,6 +41,8 @@ export async function loadConfig() {
     }
     validateBrowserConfig(config.default);
     validateWorkersConfig(config.default);
+    validateForbidOnlyConfig(config.default);
+    validateRetriesConfig(config.default);
     return config.default;
 }
 /** A project's config without keeping the project active (`withProject` is one at a time): the given one, else its samurai.config.ts */

package/dist/config/forbid-only.d.ts

@@ -0,0 +1,5 @@
+import type { SamuraiTestConfig } from "../types/config.js";
+/** Checks the `forbidOnly` key of a config. Throws an Error that names the key */
+export declare function validateForbidOnlyConfig(config: SamuraiTestConfig): void;
+/** Whether `test.only` is refused: the config's `forbidOnly`, else on when `CI` is set */
+export declare function forbidsOnly(config: SamuraiTestConfig, env?: Partial<NodeJS.ProcessEnv>): boolean;

package/dist/config/forbid-only.js

@@ -0,0 +1,9 @@
+/** Checks the `forbidOnly` key of a config. Throws an Error that names the key */
+export function validateForbidOnlyConfig(config) {
+    if (config.forbidOnly !== undefined && typeof config.forbidOnly !== "boolean")
+        throw new Error(`forbidOnly must be true or false, got "${String(config.forbidOnly)}"`);
+}
+/** Whether `test.only` is refused: the config's `forbidOnly`, else on when `CI` is set */
+export function forbidsOnly(config, env = process.env) {
+    return config.forbidOnly ?? Boolean(env.CI);
+}

package/dist/config/retries.d.ts

@@ -0,0 +1,8 @@
+import type { SamuraiTestConfig } from "../types/config.js";
+/**
+ * Reads a `retries` value: a whole number from 0, as a number or as text. `where` names the source (`--retries`,
+ * `SAMURAI_RETRIES`, a config key) in the error.
+ */
+export declare function parseRetries(value: unknown, where: string): number;
+/** Checks the `retries` key of a config. Throws an Error that names the key */
+export declare function validateRetriesConfig(config: SamuraiTestConfig): void;

package/dist/config/retries.js

@@ -0,0 +1,19 @@
+/**
+ * Reads a `retries` value: a whole number from 0, as a number or as text. `where` names the source (`--retries`,
+ * `SAMURAI_RETRIES`, a config key) in the error.
+ */
+export function parseRetries(value, where) {
+    if (typeof value === "number" && Number.isInteger(value) && value >= 0)
+        return value;
+    if (typeof value === "string" && /^\s*\d+\s*$/.test(value))
+        return Number(value.trim());
+    throw new Error(`${where} must be a whole number from 0, got "${String(value)}"`);
+}
+/** Checks the `retries` key of a config. Throws an Error that names the key */
+export function validateRetriesConfig(config) {
+    if (config.retries === undefined)
+        return;
+    if (typeof config.retries !== "number")
+        throw new Error(`retries must be a whole number from 0, got "${String(config.retries)}"`);
+    parseRetries(config.retries, "retries");
+}

package/dist/flows/run/node-pool.d.ts

@@ -0,0 +1,9 @@
+import type { RunEvent } from "../../runner/reporter.js";
+import type { WorkerPool } from "../../runner/worker-pool.js";
+import type { WorkItem } from "../../runner/worker-protocol.js";
+import type { SupportedBrowser } from "../../types/browser.js";
+import type { NodeTest } from "./events.js";
+/** The work items of a node: each test in each browser, the browsers one after the other */
+export declare function itemsOf(tests: NodeTest[], browsers: readonly SupportedBrowser[]): WorkItem[];
+/** Runs the tests of a node on the pool and reports them as a run would */
+export declare function runOnPool(pool: Pick<WorkerPool, "run">, items: WorkItem[], onEvent: (event: RunEvent) => void, signal: AbortSignal): Promise<"passed" | "failed" | "cancelled">;

package/dist/flows/run/node-pool.js

@@ -0,0 +1,27 @@
+/** The work items of a node: each test in each browser, the browsers one after the other */
+export function itemsOf(tests, browsers) {
+    return browsers
+        .flatMap((browser) => tests.map((t) => ({ file: t.file, testName: t.name, browser })))
+        .map((item, index) => ({ ...item, index }));
+}
+/** Runs the tests of a node on the pool and reports them as a run would */
+export async function runOnPool(pool, items, onEvent, signal) {
+    const results = await pool.run(items, {
+        signal,
+        onEvent: (event) => onEvent({
+            type: event.type,
+            name: event.item.testName,
+            file: event.item.file,
+            browser: event.item.browser,
+            ...(event.type === "test-start" &&
+                event.retry && { retry: event.retry }),
+            ...(event.type === "test-end" && { result: event.result }),
+        }),
+    });
+    if (signal.aborted)
+        return "cancelled";
+    // A skipped test is not a failure
+    return results.every((r) => r.status === "success" || r.status === "skipped")
+        ? "passed"
+        : "failed";
+}

package/dist/flows/run/project-options.d.ts

@@ -0,0 +1,44 @@
+import type { ProjectOptions } from "../../runner/project.js";
+import type { PoolOptions, WorkerPool } from "../../runner/worker-pool.js";
+import type { runTests } from "../../runner/run.js";
+import type { FlowFile, FlowProblem } from "../core/index.js";
+import type { FlowEvent } from "./events.js";
+/** The flow has errors; nothing was run */
+export declare class FlowCheckError extends Error {
+    readonly problems: FlowProblem[];
+    constructor(problems: FlowProblem[]);
+}
+export interface RunFlowInProjectOptions extends ProjectOptions {
+    /** The flow to run... */
+    flow?: FlowFile;
+    /** ...or the id of one in `<project>/flows` */
+    flowId?: string;
+    /** Beats the flow's `envDefault`, the config's `defaultEnvironment` and the first environment */
+    environment?: string;
+    signal?: AbortSignal;
+    onEvent?: (event: FlowEvent) => void;
+    /** `"fail"` (default): a flow with a node that can't run is refused. `"skip"`: those nodes are skipped. */
+    unsupported?: "fail" | "skip";
+    port?: number;
+    /**
+     * How many worker processes the tests of the whole run share: a number, or a percentage like `"50%"`. Unset
+     * takes `SAMURAI_WORKERS` (`workersEnv`), then the config's `workers`, then half the CPUs. 1 runs each Test
+     * and Group node through `runTests` in this process; more starts one pool that every node, and every branch
+     * of a Parallel node, takes its workers from
+     */
+    workers?: number | string;
+    /** The value of `SAMURAI_WORKERS`, when set */
+    workersEnv?: string;
+    /** How many times a failing test is run again: a number, or numeric text like the command line's `--retries`. Unset takes `SAMURAI_RETRIES` (`retriesEnv`), then the config's `retries`, then 0 */
+    retries?: number | string;
+    /** The value of `SAMURAI_RETRIES`, when set */
+    retriesEnv?: string;
+    /** @internal Replaces the worker pool (tests) */
+    createPool?: (options: PoolOptions) => Pick<WorkerPool, "run" | "dispose">;
+    /** @internal Replaces the runner of each Test and Group node (tests) */
+    testRunner?: typeof runTests;
+    /** @internal */
+    sleep?: (ms: number, signal: AbortSignal) => Promise<void>;
+}
+/** What `runFlowInProject` takes when the flow itself is given separately: everything but `flow` and `flowId` */
+export type FlowRunOptions = Omit<RunFlowInProjectOptions, "flow" | "flowId">;

package/dist/flows/run/project-options.js

@@ -0,0 +1,8 @@
+/** The flow has errors; nothing was run */
+export class FlowCheckError extends Error {
+    constructor(problems) {
+        super(`This flow can't run:\n${problems.map((p) => `  - ${p.message}`).join("\n")}`);
+        this.problems = problems;
+        this.name = "FlowCheckError";
+    }
+}

package/dist/flows/run/project.d.ts

@@ -1,46 +1,11 @@
 import { type FlowFile, type FlowProblem } from "../core/index.js";
 import { type ResolvedGroup } from "../../runner/groups.js";
-import { type ProjectOptions } from "../../runner/project.js";
-import { runTests } from "../../runner/run.js";
-import { WorkerPool, type PoolOptions } from "../../runner/worker-pool.js";
 import type { SamuraiTestConfig } from "../../types/config.js";
-import type { FlowEvent, FlowSummary, NodeTest } from "./events.js";
+import type { FlowSummary, NodeTest } from "./events.js";
+import { FlowCheckError, type FlowRunOptions, type RunFlowInProjectOptions } from "./project-options.js";
 import type { SupportedBrowser } from "../../types/browser.js";
-/** The flow has errors; nothing was run */
-export declare class FlowCheckError extends Error {
-    readonly problems: FlowProblem[];
-    constructor(problems: FlowProblem[]);
-}
-export interface RunFlowInProjectOptions extends ProjectOptions {
-    /** The flow to run... */
-    flow?: FlowFile;
-    /** ...or the id of one in `<project>/flows` */
-    flowId?: string;
-    /** Beats the flow's `envDefault`, the config's `defaultEnvironment` and the first environment */
-    environment?: string;
-    signal?: AbortSignal;
-    onEvent?: (event: FlowEvent) => void;
-    /** `"fail"` (default): a flow with a node that can't run is refused. `"skip"`: those nodes are skipped. */
-    unsupported?: "fail" | "skip";
-    port?: number;
-    /**
-     * How many worker processes the tests of the whole run share: a number, or a percentage like `"50%"`. Unset
-     * takes `SAMURAI_WORKERS` (`workersEnv`), then the config's `workers`, then half the CPUs. 1 runs each Test
-     * and Group node through `runTests` in this process; more starts one pool that every node, and every branch
-     * of a Parallel node, takes its workers from
-     */
-    workers?: number | string;
-    /** The value of `SAMURAI_WORKERS`, when set */
-    workersEnv?: string;
-    /** @internal Replaces the worker pool (tests) */
-    createPool?: (options: PoolOptions) => Pick<WorkerPool, "run" | "dispose">;
-    /** @internal Replaces the runner of each Test and Group node (tests) */
-    testRunner?: typeof runTests;
-    /** @internal */
-    sleep?: (ms: number, signal: AbortSignal) => Promise<void>;
-}
-/** What `runFlowInProject` takes when the flow itself is given separately: everything but `flow` and `flowId` */
-export type FlowRunOptions = Omit<RunFlowInProjectOptions, "flow" | "flowId">;
+export { FlowCheckError };
+export type { FlowRunOptions, RunFlowInProjectOptions };
 /** The environment: the option, else the flow's `envDefault` (when the project has it), else the config's default, else the first, else "default" */
 export declare function chooseEnvironment(config: SamuraiTestConfig, flow: FlowFile, option?: string): string;
 /** A test of the project as flows see it */
@@ -71,4 +36,3 @@ export declare function checkFlowInProject(options: Pick<RunFlowInProjectOptions
  * One at a time per process.
  */
 export declare function runFlowInProject(options: RunFlowInProjectOptions): Promise<FlowSummary>;
-export {};