testaro 68.0.0 → 69.0.0

package/UPGRADES.md

@@ -3259,3 +3259,7 @@ That will help you see whether future failures cluster around late DOM injection
 ## Status
 Your fix is a sensible incremental change; keep it for now, and if you see new mismatches, the next step is adding a DOM-stability heuristic or an overlay-handling step rather than abandoning the approach.
 ```
+
+### Automated accessibility testing at Slack
+
+[Automated accessibility testing at Slack](https://slack.engineering/automated-accessibility-testing-at-slack/) is based on Playwright, with Axe as a single tool.

package/VALIDATION.md

@@ -1,6 +1,67 @@
 # Validation
 
-Quick notes to run the Testaro validation tests locally (Windows PowerShell):
+## Introduction
+
+Testing the correctness of Testaro is named “validation” rather than “testing”, for confusion avoidance.
+
+The original strategy for validation is to permit any test act to contain an `expect` property, whose value specifies a fact about the result. The specification language is a custom Testaro-specific language. When an act in a job has an `expect` property, then the act in the corresponding report contains an `expectations` property that describes the success or failure of the result to conform to the specification.
+
+In practice, no use of the `expect`/`expectations` pattern is known except for test acts whose tool is `testaro`. The other tools are treated as black boxes with no contracts entitling Testaro to hold the tools accountable for compliance. Thus, the pattern has been used for the validation of the tests of the `testaro` tool.
+
+It may be appropriate to replace the pattern with a conventional testing approach that is widely practiced and understood, based on a platform such as Vitest and/or Playwright. While such a replacement is being considered, the documentation on the pattern that has been part of the `README.md` file is moved into this document. Here it is named “classic validation”.
+
+## Classic validation
+
+### Expectations
+
+Any `test` act can contain an `expect` property. If it does, the value of that property must be an array of arrays. Each array specifies expectations about the results of the operation of the tool.
+
+For example, a `test` act might have this `expect` property:
+
+```javaScript
+'expect': [
+  ['standardResult.totals.0', '=', 0],
+  ['standardResult.instances.length', '=', 0]
+]
+```
+
+That would state the expectations that the `standardResult` property of the act will report no rule violations at severity level 0 and no instances of rule violations.
+
+The first item in each array is an identifier of a property of the act. The item has the format of a string with `.` delimiters. Each `.`-delimited segment its the name of the next property in the hierarchy. If the current object is an array, the next segment must be a non-negative integer, representing the index of an element of the array.
+
+If there is only 1 item in an array, it states the expectation that the specified property does not exist. Otherwise, there are 3 items in the array.
+
+The second item in each array, if there are 3 items, is an operator, drawn from:
+
+- `<`: less than
+- `=`: equal to
+- `>`: greater than
+- `!`: unequal to
+- `i`: includes
+- `e`: equivalent to (parsed identically as JSON)
+
+The third item in each array, if there are 3 items in the array, is the criterion with which the value of the first property is compared.
+
+A typical use for an `expect` property is checking the correctness of a Testaro test. Thus, the validation jobs in the `validation/tests/jobs` directory all contain `test` acts with `expect` properties. See the “Validation” section below.
+
+### Validators
+
+Testaro and the tests of the Testaro tool can be validated with the _executors_ located in the `validation/executors` directory.
+
+The executor for a single test is `test`. To execute it for any test `xyz`, call it with the statement `npm test xyz`.
+
+The other executors are:
+
+- `run`: validates immediate test execution
+- `watchDir`: validates directory watching
+- `watchNet`: validates network watching
+- `tests`: validates all the Testaro tests
+
+To execute any executor `xyz` among these, call it with the statement `npm run xyz`.
+
+The `tests` executor makes use of the jobs in the `validation/tests/jobs` directory, and they, in turn, run tests on HTML files in the `validation/tests/targets` directory.
+
+### Validation in Windows PowerShell
 
 1. Install project dependencies
 

package/actSpecs-doc.md

@@ -0,0 +1,53 @@
+# actSpecs
+
+## Introduction to the `actSpecs` file
+
+The `actSpecs.js` file contains rules governing acts. The rules determine whether an act is valid.
+
+## Rule format
+
+The rules in `actSpecs.js` are organized into two objects, `etc` and `tools`. The `etc` object contains rules for acts of all types. The `tools` object contains additional rules that apply to some acts of type `test`, depending on the values of their `which` properties, namely which tools they perform tests of.
+
+Here is an example of an act:
+
+```json
+{
+  "type": "link",
+  "which": "warming",
+  "what": "article on climate change"
+}
+```
+
+And here is the applicable property of the `etc` object in `actSpecs.js`:
+
+```js
+link: [
+  'Click a link',
+  {
+    which: [true, 'string', 'hasLength', 'substring of the link text'],
+    what: [false, 'string', 'hasLength', 'comment']
+  }
+]
+```
+
+The rule is an array with two elements: a string ('Click a link') describing the act and an object containing requirements for any act of type `link`.
+
+The requirement `which: [true, 'string', 'hasLength', 'substring of the link text']` specifies what is required for the `which` property of a `link`-type act. The requirement is an array.
+
+In most cases, the array has length 4:
+
+- Item 0. Is the property (here `which`) required (`true` or `false`)? The value `true` here means that every `link`-type act **must** contain a `which` property.
+- Item 1. What format must the property value have (`'string'`, `'array'`, `'boolean'`, `'number'`, or `'object'`)?
+- Item 2. What other validity criterion applies (if any)? (Empty string if none.) The `hasLength` criterion means that the string must be at least 1 character long.
+- Item 3. Description of the property. In this example, the description says that the value of `which` must be a substring of the text content of the link that is to be clicked. Thus, a `link` act tells Testaro to find the first link whose text content has this substring and click it.
+
+The validity criterion named in item 2 may be any of these:
+
+- `'hasLength'`: is not a blank string
+- `'isURL`': is a string starting with `http`, `https`, or `file`, then `://`, then ending with 1 or more non-whitespace characters
+- `'isBrowserType'`: is `'chromium'`, `'firefox'`, or `'webkit'`
+- `'isFocusable'`: is `'a'`, `'button'`, `'input'`, `'select'`, or `'option'`
+- `'isState'`: is `'loaded'` or `'idle'`
+- `'isTest'`: is the name of a tool
+- `'isWaitable'`: is `'url'`, `'title'`, or `'body'`
+- `'areStrings'`: is an array of strings

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "68.0.0",
+  "version": "69.0.0",
   "description": "Run 1000 web accessibility tests from 11 tools and get a standardized report",
   "main": "index.js",
   "scripts": {

package/procs/catalog.js

@@ -74,7 +74,7 @@ exports.getCatalog = async report => {
           const tidySegments = segments?.map(segment => segment.trim().replace(/\s+/g, ' ')) ?? [];
           const neededSegments = tidySegments.filter(segment => segment.length);
           neededSegments.splice(1, neededSegments.length - 2);
-          const text = addToCatalog(index, cat, 'text', neededSegments.join('◢‖◤'));
+          const text = addToCatalog(index, cat, 'text', neededSegments.join('\n'));
           const domRect = element.getBoundingClientRect();
           const boxID = addToCatalog(
             index,
@@ -90,8 +90,8 @@ exports.getCatalog = async report => {
             tagName,
             id,
             startTag,
-            textLinkable: false,
             text,
+            textLinkable: false,
             boxID,
             pathID
           };

package/tests/htmlcs.js

@@ -129,7 +129,7 @@ exports.reporter = async (page, report, actIndex) => {
       // If standard results are to be reported:
       if (standard) {
         const instance = {
-          ruleID: parts[1],
+          ruleID: `${parts[0][0]}-${parts[1]}`,
           what: parts[4],
           ordinalSeverity: parts[0] === 'Warning' ? 0 : 2,
           count: 1

package/tests/testaro.js

@@ -20,10 +20,7 @@ const {launch} = require('../procs/launch');
 
 // CONSTANTS
 
-/*
-  Metadata of all rules in default execution order.
-  Property needsLaunch is true if the rule is first or the previous one contaminates the page.
-*/
+// Metadata of all rules in default execution order.
 const allRules = [
   {
     id: 'shoot0',