testaro 2.3.11 → 4.0.0

package/README.md

@@ -10,14 +10,10 @@ The purpose of Testaro is to provide programmatic access to over 600 accessibili
 
 Running Testaro requires telling it which operations (including tests) to perform and which URLs to perform them on, and giving Testaro an object to put its output into.
 
-Testaro outputs progress messages to the standard output. It populates the object with log information and test reports.
-
 ## System requirements
 
 Version 14 or later of [Node.js](https://nodejs.org/en/).
 
-A file system with a directory that the Testaro user has permission to read and write in.
-
 ## Dependencies
 
 Testaro uses:
@@ -29,6 +25,7 @@ Testaro includes some of its own accessibility tests. In addition, it performs t
 - [alfa](https://alfa.siteimprove.com/) (Siteimprove alfa)
 - [Automated Accessibility Testing Tool](https://www.npmjs.com/package/aatt) (Paypal AATT, running HTML CodeSniffer)
 - [axe-playwright](https://www.npmjs.com/package/axe-playwright) (Deque Axe-core)
+- [Tenon](https://tenon.io/documentation/what-tenon-tests.php)
 - [WAVE API](https://wave.webaim.org/api/) (WebAIM WAVE)
 
 As of this version, the counts of tests in the packages referenced above were:
@@ -36,10 +33,11 @@ As of this version, the counts of tests in the packages referenced above were:
 - Alfa: 103
 - Axe-core: 138
 - Equal Access: 163
+- Tenon: 433
 - WAVE: 110
 - subtotal: 612
 - Testaro tests: 16
-- grand total: 628
+- grand total: 1061
 
 ## Code organization
 
@@ -100,25 +98,25 @@ A script is a JSON file with the properties:
 
 Here is an example of a script:
 
-```json
+```javascript
 {
-  "what": "Test example.com with alfa",
-  "strict": true,
-  "commands": [
+  what: 'Test example.com with alfa',
+  strict: true,
+  commands: [
     {
-      "type": "launch",
-      "which": "chromium",
-      "what": "Chromium browser"
+      type: 'launch',
+      which: 'chromium',
+      what: 'Chromium browser'
     },
     {
-      "type": "url",
-      "which": "https://example.com/",
-      "what": "page with a few accessibility defects"
+      type: 'url',
+      which: 'https://example.com/',
+      what: 'page with a few accessibility defects'
     },
     {
-      "type": "test",
-      "which": "alfa",
-      "what": "Siteimprove alfa package"
+      type: 'test',
+      which: 'alfa',
+      what: 'Siteimprove alfa package'
     }
   ]
 }
@@ -256,6 +254,24 @@ An example of a **Testaro-defined** test is:
 
 In this case, Testaro runs the `motion` test with the specified parameters.
 
+###### Tenon
+
+The `tenon` test requires two commands:
+- A command of type `tenonRequest`.
+- A command of type `test` with `tenon` as the value of `which`.
+
+The reason for this is that the Tenon API operates asynchronously. You ask it to perform a test, and it puts your request into a queue. To learn whether Tenon has completed your test, you make a status request. You can continue making status requests until Tenon replies that your test has been completed. Then you submit a request for the test result, and Tenon replies with the result. (As of May 2022, status requests were observed to misreport still-running tests as completed. The `tenon` test works around that.)
+
+Tenon says that tests are typically completed in 3 to 6 seconds but that the latency can be longer, depending on demand.
+
+Therefore, you can include a `tenonRequest` command early in your script, and a `tenon` test late in your script. Tenon will move your request through its queue while Testaro is processing your script. When Testaro reaches your `tenon` test command, Tenon will most likely have completed your test. If not, the `tenon` test will wait and then make a second request before giving up.
+
+Thus, a `tenon` test actually does not perform any test; it merely collects the result. The page that was active when the `tenonRequest` command was performed is the one that Tenon tests.
+
+In case you want to perform more than one `tenon` test, you can do so. Just give each pair of commands a distinct `id` property, so each `tenon` test command will request the correct result.
+
+Tenon recommends giving it a public URL rather than giving it the content of a page, if possible. So, it is best to give the `withNewContent` property of the `tenonRequest` command the value `true`, unless the page is not public.
+
 ##### Scoring
 
 An example of a **scoring** command is:
@@ -383,70 +399,55 @@ A typical use for an `expect` property is checking the correctness of a Testaro
 
 ## Batches
 
-There are two ways to use a script to give instructions to Testaro:
-- The script can be the complete specification of the operations to perform and the URLs to perform them on.
-- The script can specify the operations to perform, and a _batch_ can specify which URLs to perform them on.
-
-A batch is a JSON file with this format:
-
-```json
-{
-  "what": "Accessibility standards",
-  "hosts": [
-    {
-      "which": "https://www.w3.org/TR/WCAG21/",
-      "what": "WCAG 2.1"
-    },
-    {
-      "which": "https://www.w3.org/WAI/standards-guidelines/aria/",
-      "what": "W3C WAI-ARIA"
-    }
-  ]
-}
-```
-
-When you combine a script with a batch, Testaro performs the script, replacing the `which` and `what` properties of all `url` commands with the values in the first object in the `hosts` array of the batch, then again with the values in the second `host` object, and so on. Those replacements also occur in the inserted extra `url` commands mentioned above.
+In some cases you may wish to repeatedly run Testaro with the same script, changing only its `url` commands. The purpose would be to perform the same set of tests on multiple web pages. Such a use would apply only to scripts whose `url` commands are all identical, not to a script that moves from one host to another.
 
-A batch offers an efficient way to perform a uniform set of operations on every host in a set of hosts. In this way you can run the same set of tests on multiple web pages.
-
-A no-batch script offers a way to carry out a complex operation, which can include navigating from one host to another, which is not possible with a batch. Any `url` commands are performed as-is, without changes to their URLs.
+Testaro does not support such batch processing, but Testilo does. See its `README.md` file for instructions.
 
 ## Execution
 
 ### Invocation
 
-To run Testaro, create an options object like this:
+To run Testaro, create a report object like this:
 
 ```javascript
-{
+const report = {
+  id: '',
+  script: {…},
   log: [],
-  reports: [],
-  script: {abc},
-  batch: {def}
-}
+  acts: []
+};
 ```
 
-Replace `{abc}` with a script object, like the example script shown above.
+Replace `{…}` with a script object, like the example script shown above.
 
-Replace `{def}` with a batch object, like the example batch shown above, or omit the `batch` property if there is no batch.
+Then execute the statement `require('testaro').handleRequest(report)`. That statement will run Testaro.
 
-Then execute the statement `require('testaro').handleRequest(ghi)`, where `ghi` is replaced with the name of the options object. That statement will run Testaro.
+While it runs, Testaro will populate the `log` and `acts` arrays of the report object. When Testaro finishes, the `log` and `acts` properties will contain its results.
 
-While it runs, Testaro will populate the `log` and `reports` arrays of the options object. When Testaro finishes, the `log` and `reports` arrays will contain its results.
+Another way to run Testaro is to use Testilo, which can handle batches and saves results to files. Testilo prepopulates the report object with an `id` property consisting of a timestamp and, if a batch is used, the host ID. If Testaro finds a non-empty `id` property in the `report` object, Testaro leaves it unchanged; if not, Testaro creates an `id` property with a timestamp value.
 
 ### Environment variables
 
 If a `wave` test is included in the script, an environment variable named `TESTARO_WAVE_KEY` must exist, with your WAVE API key as its value.
 
+If a `tenon` test is included in the script, environment variables named `TESTARO_TENON_USER` and `TESTARO_TENON_PASSWORD` must exist, with your Tenon username and password, respectively, as their values.
+
+The `text` command can interpolate the value of an environment variable into text that it enters on a page, as documented in the `commands.js` file.
+
 Before executing a Testaro script, you can optionally also set the environment variables `TESTARO_DEBUG` (to `'true'` or anything else) and/or `TESTARO_WAITS` (to a non-negative integer). The effects of these variables are described in the `index.js` file.
 
+You may store these environment variables in an untracked `.env` file if you wish, and Testaro will recognize them.
+
 ## Validation
 
-Three _executors_ for Testaro validation are located in the `validation` directory. An executor is a commonJS JavaScript module that runs Testaro and reports whether the results are correct.
+_Executors_ for Testaro validation are located in the `validation` directory.
 
-The executors are:
-- `appNoBatch.js`: Reports whether Testaro runs correctly with a no-batch script.
-- `appBatch.js`: Reports whether Testaro runs correctly with a script and a batch.
+A basic executor is the `test.js` file. It runs Testaro with a simple sample script and outputs the log and the acts.
+
+The other executors are commonJS JavaScript modules that run Testaro and report whether the results are correct.
+
+The other executors are:
+- `app.js`: Reports whether Testaro runs correctly with a script.
 - `tests.js`: Runs Testaro with each custom test and reports whether the results are correct.
 
 To execute any executor `xyz.js`, call it with the statement `node validation/executors/xyz`. The results will appear in the standard output.
@@ -461,18 +462,6 @@ You can define additional Testaro commands and functionality. Contributions are
 
 The rationales motivating the Testaro-defined tests and scoring procs can be found in comments within the files of those tests and procs, in the `tests` and `procs/score` directories. Unavoidably, each test is opinionated. Testaro itself, however, can accommodate other tests representing different opinions. Testaro is intended to be neutral with respect to questions such as the criteria for accessibility, the severities of accessibility issues, whether accessibility is binary or graded, and the distinction between usability and accessibility.
 
-### Future work
-
-Further development is contemplated, is taking place, or is welcomed, on:
-- addition of Tenon to the set of packages
-- links with href="#"
-- links and buttons styled non-distinguishably
-- first focused element not first focusable element in DOM
-- never-visible skip links
-- buttons with no text content
-- modal dialogs
-- autocomplete attributes
-
 ## Testing challenges
 
 ### Activation
@@ -489,9 +478,11 @@ Test packages sometimes do redundant testing, in that two or more packages test
 
 The files in the `temp` directory are presumed ephemeral and are not tracked by `git`. When tests require temporary files to be written, Testaro writes them there.
 
-## Origin
+## Related packages
+
+[Testilo](https://www.npmjs.com/package/testilo) is an application that facilitates the use of Testaro.
 
-Testaro is derived from [Autotest](https://github.com/jrpool/autotest), which in turn is derived from accessibility test investigations beginning in 2018.
+Testaro is derived from [Autotest](https://github.com/jrpool/autotest).
 
 Testaro omits some functionalities of Autotest, such as:
 - tests producing results intended to be human-inspected
@@ -499,6 +490,44 @@ Testaro omits some functionalities of Autotest, such as:
 - file operations for score aggregation, report revision, and HTML reports
 - a web user interface
 
+## Origin
+
+Work on the custom tests in this package began in 2017, and work on the multi-package federation that Testaro implements began in early 2018. These two aspects were combined into the [Autotest](https://github.com/jrpool/autotest) package in early 2021 and into this more single-purpose package, Testaro, in January 2022.
+
 ## Etymology
 
 “Testaro” means “collection of tests” in Esperanto.
+
+## Future work
+
+### Improvements
+
+Further development is contemplated, is taking place, or is welcomed, on:
+- addition of Tenon to the set of packages
+- links with href="#"
+- links and buttons styled non-distinguishably
+- first focused element not first focusable element in DOM
+- never-visible skip links
+- buttons with no text content
+- modal dialogs
+- autocomplete attributes
+- inclusion of other test packages, such as:
+   - FAE (https://github.com/opena11y/evaluation-library)
+   - Tenon
+
+## Corrections
+
+Issues found or reported with the current version that need diagnosis and correction include:
+
+### hover
+
+There seem to be a couple of problems with the hover test:
+- The score for unhoverability is documented as 2 times the count of unhoverables, but is reported as 1 time that count.
+- The list of unhoverables in the report is empty.
+Observed after inquiry by Tobias Christian Jensen of Siteimprove on 2022-05-09.
+
+### axe
+
+Configuration to include best practices and experimental tests.
+
+Investigation of tags, including wcag2a, wcag2aa, wcag21a, wcag21aa, best-practice, wcag***, ACT, cat.*.

package/commands.js

@@ -118,6 +118,14 @@ exports.commands = {
         what: [false, 'string', 'hasLength', 'comment']
       }
     ],
+    tenonRequest: [
+      'Request a Tenon test',
+      {
+        id: [true, 'string', 'hasLength', 'ID for this test instance'],
+        withNewContent: [true, 'boolean', '', 'true: use a URL; false: use page content'],
+        what: [false, 'string', 'hasLength', 'comment']
+      }
+    ],
     text: [
       'Enter text into a text input, optionally with 1 placeholder for an all-caps literal environment variable',
       {
@@ -185,7 +193,7 @@ exports.commands = {
       {
         withItems: [true, 'boolean'],
         withNewContent: [
-          false, 'boolean', '', 'true: use a URL; false: use page content; omitted: use both'
+          false, 'boolean', '', 'true: use a URL; false: use page content; omitted: both'
         ]
       }
     ],
@@ -233,6 +241,12 @@ exports.commands = {
         withItems: [true, 'boolean']
       }
     ],
+    tenon: [
+      'Perform a Tenon test',
+      {
+        id: [true, 'string', 'hasLength', 'ID of the requested test instance']
+      }
+    ],
     wave: [
       'Perform a WebAIM WAVE test',
       {