testaro 78.0.6 → 78.4.0

package/README.md

@@ -53,11 +53,13 @@ Testaro can perform tests of these _rule engines_:
 - [Editoria11y](https://github.com/itmaybejj/editoria11y) (Princeton University)
 - [HTML CodeSniffer](https://www.npmjs.com/package/html_codesniffer) (Squiz Labs)
 - [Nu Html Checker](https://github.com/validator/validator) (World Wide Web Consortium)
+- [Pour Engine](https://github.com/pourdev/pour-engine) (David Yarham and Geoffrey Crofte)
 - [QualWeb](https://www.npmjs.com/package/@qualweb/core) (University of Lisbon)
+- [SureA11y](https://github.com/SureA11y/core) (Jorge Rumoroso)
 - [Testaro](https://www.npmjs.com/package/testaro) (CVS Health)
 - [WAVE](https://wave.webaim.org/api/) (WebAIM)
 
-For the rule engines that are open-source, the identified organizations are their principal or original sponsors.
+For the rule engines that are open-source, the identified organizations or persons are their principal or original sponsors or authors.
 
 As shown, Testaro is not only an integrator but also one of the integrated rule engines. That is because it provides about 50 tests of its own, mostly to complement tests provided by the other rule engines. Some of those Testaro tests are designed to act as approximate alternatives to tests of vulnerable, restricted, or no longer available rule engines. In all such cases the Testaro tests are independently designed and implemented, without reference to the code of the tests that inspired them.
 
@@ -81,7 +83,7 @@ The main concepts of Testaro are:
 
 ### Operating system and Node.js version
 
-Testaro can be installed under a MacOS, Windows, Debian, or Ubuntu operating system with the latest long-term-support version of [Node.js](https://nodejs.org/en/).
+Testaro can be installed under a MacOS, Windows, Debian, or Ubuntu operating system with the latest long-term-support version of [Node.js](https://nodejs.org/en/). The minimum version is Node 22.12: Node 20 reached end of life in April 2026, and some dependencies (the Alfa packages and pixelmatch) are ES modules that Testaro loads with `require()`, which the Node 22 line supports from 22.12 onward. When Node 22 reaches end of life in April 2027, the minimum is expected to rise to Node 24.
 
 ### Browser security
 
@@ -115,7 +117,7 @@ This application implements option B.
 
 ## Installation as an independent application
 
-To install Testaro as an independent application, rather than a dependency, clone the [Testaro repository](https://github.com/jrpool/testaro). To ensure that the binary browsers of its Playwright dependency get installed, execute `(p)npx playwright install` after executing `(p)npm install`.
+To install Testaro as an independent application, rather than a dependency, clone the [Testaro repository](https://github.com/YRA-Tech/testaro). To ensure that the binary browsers of its Playwright dependency get installed, execute `(p)npx playwright install` after executing `(p)npm install`.
 
 To update Testaro when it is an independent application, execute:
 
@@ -129,6 +131,8 @@ git pull
 
 The `.env` file stores your decisions about the environment in which Testaro runs. The variables that can be defined there are documented in the `env.example` file.
 
+Deployment options let an operator set a fleet-wide policy that a job can override: the load state and time limit a navigation waits for and whether a 4xx response ends launch retries at once (`NAV_WAIT_UNTIL`, `NAV_TIMEOUT`, `NAV_FAIL_FAST_4XX`, or a job's `navigation` property); a scanner identity sent as the `X-YRA-Scanner` request header (`SCANNER_ID` or a job's `scannerId`); a full-page scroll after navigation so lazily loaded content is present (`PRESCAN_SCROLL` or a job's `scroll`); a branded Chromium channel (a job's `browserChannel`); launch retries per rule of the `testaro` tool (`TESTARO_RULE_RETRIES` or a `testaro` test act's `retries`); and whether the `qualWeb` tool's browser runs stealth evasions and blocks ads and trackers (`QUALWEB_STEALTH`, `QUALWEB_ADBLOCK`, or a `qualWeb` test act's `stealth` and `adBlock`). The defaults keep the behavior of earlier versions.
+
 ## Jobs
 
 Jobs tell Testaro what to do.
@@ -141,7 +145,7 @@ Here is a sample job, showing properties that you can set:
   what: 'monthly health check', // Job description
   strict: true, // Whether to reject redirections from the target URL
   standard: 'only', // Report native (no), standard (only), or both (also) results
-  imageColor: 0, // Color type (0, 2, 4, 6) of the page image, if one is to be created along with a catalog
+  imageColor: 0, // Color type (0 (grayscale), 2 (RGB), 4 (grayscale alpha), 6 (RGBA)) if initial page image required
   imageScale: 2, // Optional: also capture the page image at this device pixel density (see the images section)
   device: { // Device to emulate
     id: 'iPhone 8',
@@ -164,6 +168,20 @@ Here is a sample job, showing properties that you can set:
                  // Chromium-specific). Defaults to true. Set false to opt
                  // out — useful for sites whose anti-bot heuristics react
                  // badly to stealth's patches.
+  browserChannel: 'chrome', // Optional. Run an installed branded Chromium (chrome or msedge)
+                            // instead of the bundled build; bundled or absent keeps the default.
+  scannerId: 'MyScanner/1.0; +https://example.com/scanner', // Optional. Sent as the
+                            // X-YRA-Scanner request header (default SCANNER_ID, else no header).
+  scroll: false, // Optional. Scroll the full page after navigation so lazily loaded content
+                 // is present before tools run (default PRESCAN_SCROLL, else false).
+  navigation: { // Optional. Load state to wait for (networkidle, load, or domcontentloaded),
+                // time limit in ms, and whether a 4xx response ends launch retries at once
+                // (defaults NAV_WAIT_UNTIL, NAV_TIMEOUT, NAV_FAIL_FAST_4XX, else networkidle,
+                // 10000, false).
+    waitUntil: 'load',
+    timeout: 30000,
+    failFast4xx: true
+  },
   creationTimeStamp: '241229T0537', // When job was created
   executionTimeStamp: '250110T1200', // When job will be ready to be performed
   target: {
@@ -288,32 +306,34 @@ In both cases, the first argument of `dirWatch` tells Testaro whether to continu
 
 ### Server polling
 
-Testaro can poll a server for jobs to be performed. The server can act as the “controller” described in [How to run a thousand accessibility tests](https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3). The server is responsible for preparing Testaro jobs, assigning them to Testaro agents, receiving reports back from those agents, and performing any further processing of the reports, including enhancement, storage, and disclosure to audiences. It can be any server reachable with a URL. That includes a server running on the same host as Testaro, with a URL such as `localhost:3000`.
+Testaro can poll a server for jobs to be performed. The server can act as the “controller” described in [How to run a thousand accessibility tests](https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3). The server is responsible for preparing Testaro jobs, assigning them to Testaro workers, receiving reports back from those workers, and performing any further processing of the reports, including enhancement, storage, and disclosure to audiences. It can be any server reachable with a URL. That includes a server running on the same host as Testaro, with a URL such as `localhost:3000`.
+
+To allow Testaro to poll a server for jobs, define the environment variables documented under `netWatch variables` in the [env.example](env.example) file. The URL paths are determined by agreement between Testaro and the server. A single Testaro instance can watch one server.
 
-To allow Testaro to poll a server for jobs, define the following environment variables:
+`NETWATCH_AUTH_TYPE` selects how Testaro authenticates to the server:
 
-- `NETWATCH_URL_JOB`: which URL to poll (the URL must contain the value of the `AGENT` environment variable)
-- `NETWATCH_URL_REPORT`: which URL to send job reports to
-- `NETWATCH_URL_AUTH`: the password to supply to the server when polling and when delivering a report
+- `none`: no credentials are sent. `NETWATCH_WORKER_ID` and `NETWATCH_WORKER_SECRET` are not required.
+- `pathBody`: the password (`NETWATCH_WORKER_SECRET`) is transmitted in the request body as the value of an `agentPW` property. If the server requires the ID of the Testaro instance in the URL path, include it in `NETWATCH_URL_JOB` and `NETWATCH_URL_REPORT` yourself; Testaro does not insert it.
+- `header`: the request carries an `authorization` header whose value is `Basic`, followed by a space and the base64 encoding of `NETWATCH_WORKER_ID:NETWATCH_WORKER_SECRET`. The worker ID must not contain a colon.
 
-The job request sent to the server can be a `POST` request, in which the `agentPW` property of the payload will be the password. Or it can be a `GET` request with the URL containing the password.
+Testaro sends job requests and completed reports as `POST` requests. When Testaro sends a report to the server, the report is the value of a `report` property in the request body. If `NETWATCH_WORKER_ID` is defined, Testaro also records it as the `sources.agent` property of the report, so the server can attribute the report to this instance under any auth type.
 
-Testaro will send the report as a `POST` request whose payload is a JSON object with two properties: `agentPW` (the password) and `report` (the report). However, if the environment does not contain a password, the payload is a JSON object containing only the report.
+The `AGENT` and `NETWATCH_URL_AUTH` variables of earlier versions are deprecated. Testaro still honors them (as `NETWATCH_WORKER_ID` and as a `pathBody` password, respectively) but warns; rename them.
 
 An application can make Testaro poll a server for jobs with:
 
 ```javaScript
 const {netWatch} = require('testaro/netWatch');
-netWatch(true, 300, true);
+netWatch(true, 300);
 ```
 
 A user can make Testaro poll a server for jobs with:
 
 ```bash
-node call netWatch true 300 true
+node call netWatch true 300
 ```
 
-The first argument of `netWatch` tells Testaro whether to continue polling after performing the first job. The second argument tells Testaro how many seconds to wait after receiving a no-jobs response before polling again. The third argument tells Testaro whether to be certificate-tolerant, i.e. to accept SSL certificates that fail verification against a list of certificate authorities (the default is `true`).
+The first argument of `netWatch` tells Testaro whether to continue polling after performing the first job. The second argument tells Testaro how many seconds to wait after receiving a no-jobs response before polling again. The optional third argument tells Testaro whether to be certificate-tolerant, i.e. to accept SSL certificates that fail verification against a list of certificate authorities (the default is `false`). Certificate tolerance disables the protection of an `https` connection, exposing credentials and reports to interception, so use it only against servers you control, such as local test servers with self-signed certificates.
 
 ## Reports
 
@@ -361,6 +381,7 @@ Testaro uses the following techniques to make the rule engines calculate XPaths:
 
 - `alfa` and `aslint`: They report XPaths, so Testaro needs only to normalize them.
 - `ed11y`: Testaro adds it and a `window.getXPath` method to the page. When the rule engine reports an element, Testaro computes its XPath.
+- `pour` and `surea11y`: Testaro adds the vendored engine bundle and a `window.getXPath` method to the page. Each engine reports a CSS selector for each element; Testaro finds the element in the page via its selector and executes `window.getXPath` on it.
 - `wave`: It reports a selector for each element; Testaro finds each element in the page via its selector and executes `window.getXPath` on the element.
 - `htmlcs`, `ibm`, `nuVal`, `nuVnu`, `qualWeb`: Testaro adds `data-xpath` attributes to all elements. The rule engines include code excerpts, with the `data-xpath` attributes, in the reported violations.
 - `axe`: It reports a selector for each element, and Testaro adds `data-xpath` attributes to all elements. Testaro finds each element in the page via its selector and uses the `data-xpath` attribute. When this fails, Testaro uses the `data-xpath` attribute if its complete value is included in the reported `node.html` value.
@@ -405,24 +426,49 @@ Details about these expectation properties are documened in the `VALIDATION.md`
 
 #### Standard results
 
-If the job instructs Testaro to include standard results, then the `result.standardResult` property of each act of type `test` will have three properties:
+If the job instructs Testaro to include standard results, then the `result.standardResult` property of each act of type `test` will have four properties:
 
 - `prevented`: Whether the rule engine was prevented from performing the act
 - `totals`: An array of 4 integers, counting the rule violations at 4 severity levels
+- `outcomeTotals`: An object counting the rule violations by outcome: `{failed, cantTell}`
 - `instances`: An array of data about the violations reported by the rule engine
 
 More specifically:
 
 - The `totals` value is an array like this: `[3, 0, 87, 4]`. This example would mean that the rule engine reported 3 failures at severity 0 (the least severe level), none at severity 1, 87 at severity 2, and 4 at severity 3. These four severities are conceptually ordinal, not metric.
+- The `outcomeTotals` value is an object like this: `{failed: 87, cantTell: 7}`. It counts the violations (weighted by `count`) that the rule engine asserted versus those it flagged as uncertain.
 - The `instances` value is an array of objects, each having these properties:
-  - `ruleId`: The ID of the rule that was violated
+  - `ruleID`: The ID of the rule that was violated
   - `what`: A description of the rule or of the violation
   - `ordinalSeverity`: The severity of the violation
+  - `outcome`: `failed` if the rule engine asserted the violation, or `cantTell` if the rule engine reported that it could not determine whether the rule was violated (the vocabulary of the ACT Rules Format). Every instance has an outcome.
+  - `uncertainty` (only with `cantTell`, and only if the rule engine gave a reason): One of `not-computable`, `judgement-required`, `runtime-dependent`, `spec-only`, `equivalence-unknown`, `out-of-scope`
+  - `needed` (only with `cantTell`, and only if the rule engine said): What a reviewer must determine to resolve the uncertainty
   - `count`: How many violations of the rule this instance reports
   - `catalogIndex`: Key of the HTML element in the catalog
 
 If no catalog entry was found for the instance, then instead of a `catalogIndex` property Testaro tries to insert a `pathID` property, whose value is a normalized XPath of the offending HTML element.
 
+The `outcome` property is the authoritative certainty signal. In version 78, each tool's `ordinalSeverity` conventions are unchanged from earlier versions: most tools encode uncertainty as a low severity (for example, axe `incomplete` results have severities 0 and 1, and `violations` have severities 2 and 3), so `ordinalSeverity` still mixes certainty with impact, and it mixes them differently per tool. Consumers should read `outcome` for certainty and should not infer it from `ordinalSeverity`. A later major version will redefine `ordinalSeverity` as impact only (0 minor, 1 moderate, 2 serious, 3 critical). Design record: `docs/standard-result-outcome.md`.
+
+Testaro's own rules report `failed` unless the rule's entry in `allRules` (in `tests/testaro.ts`) specifies `outcome: 'cantTell'` (for example, `allCaps`, whose violations are AI estimates) or a violation description carries a prefix: `2:` sets severity 2, `2?:` sets severity 2 and outcome `cantTell`, and `?:` sets outcome `cantTell` at the rule's default severity.
+
+### Checkpoints
+
+A job's acts run in order: a `launch` act opens a page, interaction acts (`button`, `link`, `text`, `press`, `url`, and so on) act on it, and `test` acts run rule engines. A **checkpoint** is a named page state reached by that flow, snapshotted and tested. Checkpoint 0 is the job target as launched; a `checkpoint` act (`{type: 'checkpoint', which: 'name'}`) creates the next one from the live page. A `test` act tests the most recent checkpoint, and each of its standard instances carries the `checkpoint` index. `report.checkpoints[k]` describes each checkpoint: its name, URL, title, the page images (`imageIndexes` into `report.images`), the catalog entries it added (`catalogRange`; every catalog entry also carries `checkpoint`), an ARIA snapshot, and, for a state reached by interaction, the acts (`replay`) that a test act's browser re-enacts after navigating to `launchURL` before the rule engine runs. Each such test act records `data.replay` with the count of replayed acts and a `fidelity` of `exact` or `divergent`, comparing the replayed page's DOM with the snapshot.
+
+A job's `isolation` property (or the `ISOLATION` environment default) sets how test acts are isolated: `process` (the default, and the previous behavior) runs each test act in a child process with its own browser; `browser` runs test acts in the job's process, each in a fresh context of one browser shared by the job; `page` runs them in the job's process on the live page of the current checkpoint, with no replay, so tools may contaminate one another's page (contaminating testaro rules still get a fresh page). `jobData.isolation` records the level used. Only `process` isolation kills a tool that overruns its time limit; the others report the act as timed out and continue.
+
+When a job has two or more checkpoints, the report gains `flow`, the running list of issues across them. `flow.checkpoints[k]` summarizes each checkpoint (the tools that observed it and its issue count), and `flow.deltas[k - 1]` compares checkpoints `k - 1` and `k`: the issues `added` by the acts between them, those that `persisted`, and those `removed`, each identified by tool, rule ID, element XPath (`pathID`), and start tag, so the same element in two page states is one issue. Only tools that observed both checkpoints are compared (`tools`); the rest are listed as `notObserved`. An earlier issue that a tool did not re-test at the later checkpoint, because its acts there were all scoped to changed subtrees that exclude the issue's element, is listed as `notRetested`, not `removed`. Each delta also carries `structure`, the difference between the two checkpoints' catalogs as XPaths (`added`, `removed`, `changed` start tags, `textChanged`, and the outermost changed elements as `roots`), and `aria`, a line diff of the two ARIA snapshots.
+
+Interaction acts can identify their elements by a `selector` property (any Playwright selector, such as `#name`, `text="Continue"`, or `role=button[name="Continue"]`) instead of an element type and text substring; a `text` act with `clear: true` replaces the input's value. With these, a recorded user path (a start URL and actions of the kinds a Playwright recording yields: `click`, `fill`, `select`, `navigate`, `wait`, `checkpoint`) converts directly into a job: `require('testaro/procs/userPath').getUserPathActs({startUrl, actions, testActs})` returns the acts, one per action, with each `checkpoint` action becoming a `checkpoint` act followed by a copy of each test act in `testActs`.
+
+When a checkpoint is created, the catalog entries of the previous checkpoint that none of its test acts cited are pruned at once, since no later test act can cite them, so a long flow does not accumulate every page state's catalog in the report that each test act reads.
+
+A `test` act may have a `scope` property. The default, `page`, tests the whole page of the act's checkpoint. With `changed`, the act tests only the subtrees that changed since the previous checkpoint (the `roots` of the structure diff, as CSS selectors), for the rules and tools that can be so restricted: the `testaro` rules declared element-local (`local: true` in `allRules`), `axe` (by its `include` context), and `surea11y` (by its context selector, the nearest common ancestor of the roots). Page-level `testaro` rules (heading order, landmarks, duplicate IDs, focus order, hover, motion, and others) and all other tools test the whole page whatever the scope, because any change can alter their verdicts. The act records `data.scope` (`requested`, `applied`, `reason`, `roots`, `pathIDs`, and for `testaro` the `localRules` and `pageRules`), so a reader can see what was and was not re-tested. The act is not scoped, and `jobData.warnings` says why, when there is no previous checkpoint, nothing changed, more than 50 subtrees changed, or the tool cannot restrict itself.
+
+Checkpoint acts are optional. A job without one behaves as before: every test act tests checkpoint 0, and interaction acts before a test act only produce a warning in `jobData.warnings`. In a job with checkpoint acts, interaction acts followed by a test act without a checkpoint act produce an implicit checkpoint and a warning. A rule engine that tests a URL rather than a page (WAVE, and the Nu checkers with `withSource`) cannot test a state reached by interaction; such a test act is prevented with the reason. Details: `docs/checkpoint-scanning.md`.
+
 ## Rule-engine details
 
 The rule engines whose tests Testaro performs have particularities described below.
@@ -494,6 +540,10 @@ The `nuVal` and `nuVnu` rule engines perform the tests of the Nu Html Checker. T
 
 Its `rules` argument is **not** an array of rule IDs, but instead is an array of rule _specifications_. A rule specification for `nuVal` or `nuVnu` is a string with the format `=ruleID` or `~ruleID`. The `=` prefix indicates that the rule ID is invariable. The `~` prefix indicates that the rule ID is variable, in which case the `ruleID` part of the specification is a matching regular expression, rather than the exact text of a message. This `rules` format arises from the fact that `nuVal` and `nuVnu` generate customized messages and do not accompany them with rule identifiers.
 
+### Pour Engine
+
+The `pour` rule engine makes use of the `pour/pour.min.js` file, a bundle that Testaro builds from the upstream [pour-engine](https://github.com/pourdev/pour-engine) repository (MIT), because upstream publishes no distributable bundle. The `pour/README.md` file documents the build command and the pinned upstream version. Testaro injects the bundle into the page and runs every rule. Findings in the engine's `violations` bucket become standard instances with outcome `failed`; findings in its `incomplete` bucket become instances with outcome `cantTell`. The engine's `passes`, `inapplicable`, and `manualReview` buckets are tallied in the act's `data` property and never become instances. If the bundle is missing or fails to define its global, the act is reported as prevented.
+
 ### QualWeb
 
 The `qualWeb` rule engine performs the ACT rules, WCAG Techniques, and best-practices tests of QualWeb. Only failures and warnings are included in the report. The EARL report of QualWeb is not generated, because it is equivalent to the report of the ACT rules tests.
@@ -519,13 +569,17 @@ Thus, when the `rules` argument is omitted, QualWeb will test for all of the rul
 
 The target can be provided to QualWeb either as HTML or as a URL. Experience indicates that the results can differ between these methods, with each method reporting some rule violations or some instances that the other method does not report. For at least some cases, more rules are reported violated when HTML is provided (`withNewItems: false`).
 
-QualWeb creates sandboxed Puppeteer pages to perform its tests on. Therefore, the host must permit sandboxed browsers to be launched. See the discussion above about browser security.
+QualWeb creates sandboxed Playwright pages to perform its tests on. Therefore, the host must permit sandboxed browsers to be launched. See the discussion above about browser security.
+
+### SureA11y
+
+The `surea11y` rule engine makes use of the `surea11y/surea11y.browser.js` file, the standalone browser bundle that [`@surea11y/core`](https://github.com/SureA11y/core) publishes, vendored verbatim. That package is licensed under the Mozilla Public License 2.0, unlike the MIT-licensed or Apache-licensed rule engines; the vendored file must not be modified. The `surea11y/README.md` file documents the pinned upstream version and the result shape. Testaro injects the bundle into the page and runs every automatic rule. The engine reports an outcome per rule (`pass`, `fail`, `cantTell`, or `notApplicable`) and may grade individual occurrences of a `fail` rule as `cantTell`. Occurrences graded `fail` become standard instances with outcome `failed`; occurrences graded `cantTell` become instances with outcome `cantTell`, carrying the engine's `uncertainty` code and `needed` guidance when it supplies them. Rules of type `manual` and rules with outcome `pass` or `notApplicable` are tallied in the act's `data` property and never become instances.
 
 ### Testaro
 
 The rules that Testaro can test for are implemented in files within the `testaro` directory.
 
-The Testaro rules are classified by an `allRules` array defined in the `tests/testaro.js` file. Each item in that array is an object with these properties:
+The Testaro rules are classified by an `allRules` array defined in the `tests/testaro.ts` file. Each item in that array is an object with these properties:
 
 - `id`: the rule ID.
 - `what`: a description of the rule.
@@ -558,7 +612,7 @@ If you want the stand-alone API to perform the tests, you need to have that API
 
 You can define additional Testaro rules and functionality. Contributions are welcome.
 
-Please report any issues, including feature requests, at the [repository](https://github.com/jrpool/testaro/issues).
+Please report any issues, including feature requests, at the [repository](https://github.com/YRA-Tech/testaro/issues).
 
 ## Accessibility principles
 
@@ -654,7 +708,7 @@ From 12 February 2024 through 30 September 2025, contributors of code to Testaro
 
 ## Future work
 
-Future work contemplated for this project is described in its [issues](https://github.com/jrpool/testaro/issues) and also discussed in the [UPGRADES.md](UPGRADES.md) file.
+Future work contemplated for this project is described in its [issues](https://github.com/YRA-Tech/testaro/issues) and also discussed in the [UPGRADES.md](UPGRADES.md) file.
 
 ## Etymology
 
@@ -663,6 +717,7 @@ Future work contemplated for this project is described in its [issues](https://g
 ## License
 
 © 2021–2025 CVS Health and/or one of its affiliates. All rights reserved.
+© 2026 Jeff Witt.
 © 2025–2026 Jonathan Robert Pool.
 
 Licensed under the [MIT License](https://opensource.org/license/mit/). See [LICENSE](../../LICENSE) file at the project root for details.

package/UPGRADES.md

@@ -8720,3 +8720,7 @@ you'll get notified when `#334` merges and when the subsequent changesets "Versi
 - **Re-test**: run a job with a `qualWeb` act and confirm the `No DOM` error is gone before merging your `qwpw` branch.
 
 I'll be ready to help with the dependency bump and verification when that release lands.
+
+# Certainty in the standard result (2026-09-01)
+
+Decision: add a first-class `outcome` (`failed` | `cantTell`) to every standard instance, with optional `uncertainty` (six ACT-style codes borrowed from surea11y) and `needed` (reviewer guidance), plus `standardResult.outcomeTotals`. `ordinalSeverity` and `totals` are unchanged in 78.x; a later major version redefines `ordinalSeverity` as impact only. This follows the `allCaps` ruling above (do not overload `ordinalSeverity` with confidence) by giving certainty its own field rather than by hiding it in `what`. Shared shape helpers live in `procs/standard.js`; the alfa adapter no longer rewrites rule IDs to `cantTell` / `cantTellTextContrast`. Full design, including the recommended Testilo tic consolidation of the 57 `*Risk` issues and certainty-aware scoring: `docs/standard-result-outcome.md`.

package/actSpecs-doc.md

@@ -55,9 +55,13 @@ The validity criterion named in item 2 may be any of these:
 - `'isWaitable'`: is `'url'`, `'title'`, or `'body'`
 - `'areStrings'`: is an array of strings
 
+## qualWeb tool
+
+A `qualWeb` test act may have `stealth` and `adBlock` boolean properties, which set whether the tool's browser runs stealth evasions and blocks ads and trackers (defaults `QUALWEB_STEALTH` and `QUALWEB_ADBLOCK`, else true).
+
 ## testaro tool
 
-The `tools.testaro` object has an `args` property specifying that a `testaro` test act may include an `args` property with an object value.
+The `tools.testaro` object has a `retries` property (launch retries per rule; default `TESTARO_RULE_RETRIES`, else 2) and an `args` property specifying that a `testaro` test act may include an `args` property with an object value.
 
 If it does, the property names of the object value must be `testaro` rule IDs. Any property value must be an array of the positional arguments to be concatenated to the four default arguments (`page`, `report`, `actIndex`, and `withItems`) in the signature of the `reporter` function of each `testaro` rule.
 
@@ -84,3 +88,11 @@ Licensed under the [MIT License](https://opensource.org/license/mit/). See [LICE
 at the project root for details.
 
 SPDX-License-Identifier: MIT
+
+## Checkpoint acts
+
+The move acts (`button`, `checkbox`, `link`, `radio`, `search`, `select`, `text`) may have a `selector` property, any Playwright selector, which replaces the act type's element selector and the text matching of `which`; `index` still selects among the matches. A `checkbox` or `radio` act must have `which` or `selector`. A `text` act with `clear: true` replaces the input's value instead of appending to it. A `select` act whose `what` matches no option text falls back to Playwright's matching of `what` as an option value or label. `procs/userPath.js` builds acts of these kinds from a recorded user path.
+
+A `test` act may have a `scope` property: `page` (the default) tests the whole page of the act's checkpoint; `changed` tests only the subtrees that changed since the previous checkpoint, for the rules and tools that can be so restricted (the element-local `testaro` rules, `axe`, and `surea11y`), and the whole page for the rest. A job with a changed-scope test act must have a checkpoint act. The act records what it was given as `data.scope`. See `docs/checkpoint-scanning.md`.
+
+A `checkpoint` act snapshots the current page state as a checkpoint that subsequent `test` acts observe. Its `which` property names the checkpoint (unique within the job, and not `start`, the name of checkpoint 0). A checkpoint act must follow a `launch` act. See `docs/checkpoint-scanning.md`.

package/actSpecs.js

@@ -19,15 +19,24 @@ exports.actSpecs = {
     button: [
       'Click a button or submit input',
       {
+        selector: [false, 'string', 'hasLength', 'Playwright selector of the element, instead of an element type and text substring'],
         which: [false, 'string', 'hasLength', 'substring of button text'],
         index: [false, 'number', '', 'index among matches if not 0'],
         what: [false, 'string', 'hasLength', 'comment']
       }
     ],
+    checkpoint: [
+      'Snapshot the current page state (catalog, page image, ARIA snapshot) as a checkpoint that subsequent test acts observe',
+      {
+        which: [true, 'string', 'hasLength', 'checkpoint name, unique within the job'],
+        what: [false, 'string', 'hasLength', 'comment']
+      }
+    ],
     checkbox: [
       'Check a checkbox',
       {
-        which: [true, 'string', 'hasLength', 'substring of checkbox text'],
+        selector: [false, 'string', 'hasLength', 'Playwright selector of the element, instead of an element type and text substring'],
+        which: [false, 'string', 'hasLength', 'substring of checkbox text (required unless selector is given)'],
         index: [false, 'number', '', 'index among matches if not 0'],
         what: [false, 'string', 'hasLength', 'comment']
       }
@@ -52,6 +61,7 @@ exports.actSpecs = {
     link: [
       'Click a link and wait for the page to be idle or loaded',
       {
+        selector: [false, 'string', 'hasLength', 'Playwright selector of the element, instead of an element type and text substring'],
         which: [false, 'string', 'hasLength', 'substring of link text'],
         index: [false, 'number', '', 'index among matches if not 0'],
         what: [false, 'string', 'hasLength', 'comment']
@@ -96,7 +106,8 @@ exports.actSpecs = {
     radio: [
       'Check a radio button',
       {
-        which: [true, 'string', 'hasLength', 'substring of radio-button text'],
+        selector: [false, 'string', 'hasLength', 'Playwright selector of the element, instead of an element type and text substring'],
+        which: [false, 'string', 'hasLength', 'substring of radio-button text (required unless selector is given)'],
         index: [false, 'number', '', 'index among matches if not 0'],
         what: [false, 'string', 'hasLength', 'comment']
       }
@@ -110,6 +121,7 @@ exports.actSpecs = {
     search: [
       'Enter text into a search input, optionally with 1 placeholder for an all-caps literal environment variable',
       {
+        selector: [false, 'string', 'hasLength', 'Playwright selector of the element, instead of an element type and text substring'],
         which: [false, 'string', 'hasLength', 'substring of input text'],
         index: [false, 'number', '', 'index among matches if not 0'],
         what: [true, 'string', 'hasLength', 'text to enter, with optional __PLACEHOLDER__']
@@ -118,6 +130,7 @@ exports.actSpecs = {
     select: [
       'Select a select option',
       {
+        selector: [false, 'string', 'hasLength', 'Playwright selector of the element, instead of an element type and text substring'],
         which: [false, 'string', 'hasLength', 'substring of select-list text'],
         index: [false, 'number', '', 'index among matches if not 0'],
         what: [true, 'string', 'hasLength', 'substring of option text content']
@@ -144,15 +157,18 @@ exports.actSpecs = {
       {
         which: [true, 'string', 'isTest', 'tool name'],
         launch: [false, 'object', '', 'new target, browserID, and/or what, if any'],
-        rules: [false, 'array', 'areStrings', 'rule IDs or (for testaro, nuVal, or nuVnu) specifications, if not all']
+        rules: [false, 'array', 'areStrings', 'rule IDs or (for testaro, nuVal, or nuVnu) specifications, if not all'],
+        scope: [false, 'string', 'isScope', 'page (default) or changed (only the subtrees changed since the previous checkpoint, for rules and tools that can be so restricted)']
       }
     ],
     text: [
       'Enter text into a text input, optionally with 1 placeholder for an all-caps literal environment variable',
       {
+        selector: [false, 'string', 'hasLength', 'Playwright selector of the element, instead of an element type and text substring'],
         which: [false, 'string', 'hasLength', 'substring of input text'],
         index: [false, 'number', '', 'index among matches if not 0'],
-        what: [true, 'string', 'hasLength', 'text to enter, with optional __PLACEHOLDER__']
+        what: [true, 'string', 'hasLength', 'text to enter, with optional __PLACEHOLDER__'],
+        clear: [false, 'boolean', '', 'whether to replace any existing value instead of appending']
       }
     ],
     url: [
@@ -196,12 +212,20 @@ exports.actSpecs = {
         withSource: [false, 'boolean', '', 'whether to use the page source and not the browser page']
       }
     ],
+    qualWeb: [
+      'Perform QualWeb tests',
+      {
+        stealth: [false, 'boolean', '', 'whether the browser runs stealth evasions (default QUALWEB_STEALTH, else true)'],
+        adBlock: [false, 'boolean', '', 'whether the browser blocks ads and trackers (default QUALWEB_ADBLOCK, else true)']
+      }
+    ],
     testaro: [
       'Perform Testaro tests',
       {
         withItems: [true, 'boolean', '', 'itemize'],
         stopOnFail: [true, 'boolean', '', 'whether testing is to stop after first failure'],
-        args: [false, 'object', 'areArrays', 'extra arguments of rules taking any']
+        args: [false, 'object', 'areArrays', 'extra arguments of rules taking any'],
+        retries: [false, 'number', '', 'launch retries per rule (default TESTARO_RULE_RETRIES, else 2)']
       }
     ],
     wave: [

package/call.js

@@ -45,8 +45,8 @@ const callRun = async jobIDStart => {
   const jobDirFileNames = await fs.readdir(todoDir);
   const jobFileNames = jobDirFileNames.filter(fileName => fileName.endsWith('.json'));
   const specifiedJobFileNames = jobIDStart
-  ? jobFileNames.filter(fileName => fileName.startsWith(jobIDStart))
-  : jobFileNames;
+    ? jobFileNames.filter(fileName => fileName.startsWith(jobIDStart))
+    : jobFileNames;
   const jobFileName = specifiedJobFileNames[0];
   // If it exists:
   if (jobFileName) {
@@ -74,7 +74,7 @@ const callDirWatch = async (isForever, intervalInSeconds) => {
 };
 // Starts a network watch, converting the interval argument to a number.
 const callNetWatch = async (isForever, intervalInSeconds, isCertTolerant) => {
-  await netWatch(
+  return await netWatch(
     isForever === 'true',
     Number.parseInt(intervalInSeconds, 10),
     isCertTolerant ? isCertTolerant === 'true' : undefined
@@ -100,9 +100,10 @@ else if (fn === 'dirWatch' && fnArgs.length === 2) {
 }
 else if (fn === 'netWatch' && [2, 3].includes(fnArgs.length)) {
   callNetWatch(... fnArgs)
-  .then(() => {
+  .then(isOK => {
     console.log('Network watch ended');
-    process.exit(0);
+    // Exit with a failure code if the watch was misconfigured or aborted, so supervisors notice.
+    process.exit(isOK ? 0 : 1);
   });
 }
 else {

package/docker-compose.yml

@@ -26,10 +26,16 @@ services:
       # Send nuVal requests to the sidecar instead of the W3C service.
       TESTARO_NU_URL: "http://nu:8888/?parser=html&out=json"
       # JODBIR and REPORTDIR from Dockerfile override those from .env.
-      # Import other specific environment variables from .env file.
-      AGENT: ${AGENT}
+      # Import other specific environment variables from .env file. The
+      # NETWATCH_* variables default to empty so that directory-watch use
+      # needs no .env entries for them; set them in .env for server polling.
       WAVE_KEY: ${WAVE_KEY}
       ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
+      NETWATCH_URL_JOB: ${NETWATCH_URL_JOB:-}
+      NETWATCH_URL_REPORT: ${NETWATCH_URL_REPORT:-}
+      NETWATCH_AUTH_TYPE: ${NETWATCH_AUTH_TYPE:-}
+      NETWATCH_WORKER_ID: ${NETWATCH_WORKER_ID:-}
+      NETWATCH_WORKER_SECRET: ${NETWATCH_WORKER_SECRET:-}
     volumes:
       # Host directories for jobs and reports. The container runs as the
       # unprivileged pwuser; the host directories must be writable by it