testaro 3.0.1 → 4.1.0

package/README.md

@@ -6,19 +6,7 @@ Federated accessibility test automation
 
 Testaro is a collection of collections of web accessibility tests.
 
-The purpose of Testaro is to provide programmatic access to over 600 accessibility tests defined in several test packages and in Testaro itself.
-
-Running Testaro requires telling it which operations (including tests) to perform and which URLs to perform them on, and giving Testaro an object to put its output into.
-
-## Origin
-
-Work on the custom tests in this package began in 2017, and work on the multi-package federation that Testaro implements began in early 2018. These two aspects were combined into the [Autotest](https://github.com/jrpool/autotest) package in early 2021 and into this more limited-purpose package, Testaro, in January 2022.
-
-Testaro omits some functionalities of Autotest, such as:
-- tests producing results intended to be human-inspected
-- previous versions of scoring algorithms
-- file operations for score aggregation, report revision, and HTML reports
-- a web user interface
+The purpose of Testaro is to provide programmatic access to over 800 accessibility tests defined in several test packages and in Testaro itself.
 
 ## System requirements
 
@@ -35,6 +23,7 @@ Testaro includes some of its own accessibility tests. In addition, it performs t
 - [alfa](https://alfa.siteimprove.com/) (Siteimprove alfa)
 - [Automated Accessibility Testing Tool](https://www.npmjs.com/package/aatt) (Paypal AATT, running HTML CodeSniffer)
 - [axe-playwright](https://www.npmjs.com/package/axe-playwright) (Deque Axe-core)
+- [Tenon](https://tenon.io/documentation/what-tenon-tests.php)
 - [WAVE API](https://wave.webaim.org/api/) (WebAIM WAVE)
 
 As of this version, the counts of tests in the packages referenced above were:
@@ -42,14 +31,11 @@ As of this version, the counts of tests in the packages referenced above were:
 - Alfa: 103
 - Axe-core: 138
 - Equal Access: 163
+- Tenon: 180
 - WAVE: 110
 - subtotal: 612
 - Testaro tests: 16
-- grand total: 628
-
-## Related packages
-
-[Testilo](https://www.npmjs.com/package/testilo) is an application that facilitates the use of Testaro.
+- grand total: 808
 
 ## Code organization
 
@@ -84,7 +70,7 @@ Once you have done that, you can install Testaro as you would install any `npm`
 
 ## Payment
 
-All of the tests that Testaro can perform are free of cost, except those in the WAVE package. WebAIM requires an API key for those tests. If you wish to have Testaro perform the WAVE tests, you will need to have a WAVE API key. Visit the URL above in order to obtain your key. It costs 1 to 3 credits to perform the WAVE tests on one URL. WebAIM gives you 100 credits without cost, before you need to begin paying.
+All of the tests that Testaro can perform are free of cost, except those in the Tenon and WAVE packages. The owner of each of those packages gives new registrants a free allowance of credits before it becomes necessary to pay for use of the API of the package. The required environment variables for authentication and payment are described below under “Environment variables”.
 
 ## Specification
 
@@ -94,7 +80,7 @@ To use Testaro, you must specify what it should do. You do this with a script an
 
 ### Introduction
 
-When you run Testaro, you provide a **script** to it. The script contains **commands**. Testaro performs those commands.
+To use Testaro, you provide a **script** to it. The script contains **commands**. Testaro __runs__ the script, i.e. performs the commands in it and writes a report of the results.
 
 A script is a JSON file with the properties:
 
@@ -175,7 +161,6 @@ The subsequent commands can tell Testaro to perform any of:
 - navigations (browser launches, visits to URLs, waits for page conditions, etc.)
 - alterations (changes to the page)
 - tests (whether in dependency packages or defined within Testaro)
-- scoring (aggregating test results into total scores)
 - branching (continuing from a command other than the next one)
 
 ##### Moves
@@ -266,36 +251,23 @@ An example of a **Testaro-defined** test is:
 
 In this case, Testaro runs the `motion` test with the specified parameters.
 
-##### Scoring
-
-An example of a **scoring** command is:
+###### Tenon
 
-```json
-{
-  "type": "score",
-  "which": "asp09",
-  "what": "5 packages and 16 custom tests, with duplication discounts"
-}
-```
+The `tenon` test requires two commands:
+- A command of type `tenonRequest`.
+- A command of type `test` with `tenon` as the value of `which`.
 
-In this case, Testaro executes the procedure specified in the `asp09` score proc (in the `procs/score` directory) to compute a total score for the script or (if there is a batch) the host. The proc is a JavaScript module whose `scorer` function returns an object containing a total score and the itemized scores that yield the total.
+The reason for this is that the Tenon API operates asynchronously. You ask it to perform a test, and it puts your request into a queue. To learn whether Tenon has completed your test, you make a status request. You can continue making status requests until Tenon replies that your test has been completed. Then you submit a request for the test result, and Tenon replies with the result. (As of May 2022, status requests were observed to misreport still-running tests as completed. The `tenon` test works around that.)
 
-The `scorer` function inspects the script report to find the required data, applies specific weights and formulas to yield the itemized scores, and combines the itemized scores to yield the total score.
+Tenon says that tests are typically completed in 3 to 6 seconds but that the latency can be longer, depending on demand.
 
-The data for scores can include not only test results, but also log statistics. Testaro includes in each report the properties:
-- `logCount`: how many log items the browser generated
-- `logSize`: how large the log items were in the aggregate, in characters
-- `prohibitedCount`: how many log items contain (case-insensitively) `403` and `status`, or `prohibited`
-- `visitTimeoutCount`: how many times an attempt to visit a URL timed out
-- `visitRejectionCount`: how many times a URL visit got an HTTP status other than 200 or 304
+Therefore, you can include a `tenonRequest` command early in your script, and a `tenon` test late in your script. Tenon will move your request through its queue while Testaro is processing your script. When Testaro reaches your `tenon` test command, Tenon will most likely have completed your test. If not, the `tenon` test will wait and then make a second request before giving up.
 
-Those log statistics can provide data for a log-based test defined in a score proc.
+Thus, a `tenon` test actually does not perform any test; it merely collects the result. The page that was active when the `tenonRequest` command was performed is the one that Tenon tests.
 
-A good score proc takes account of duplications between test packages: two or more packages that discover the same accessibility defects. Score procs can apply discounts to reflect duplications between test packages, so that, if two or more packages discover the same defect, the defect will not be overweighted.
+In case you want to perform more than one `tenon` test, you can do so. Just give each pair of commands a distinct `id` property, so each `tenon` test command will request the correct result.
 
-The procedures in the `scoring` directory have produced data there that score procs can use for the calibration of discounts.
-
-Some documents are implemented in such a way that some tests are prevented from being conducted on them. When that occurs, the score proc can **infer** a score for that test.
+Tenon recommends giving it a public URL rather than giving it the content of a page, if possible. So, it is best to give the `withNewContent` property of the `tenonRequest` command the value `true`, unless the page is not public.
 
 ##### Branching
 
@@ -393,15 +365,35 @@ A typical use for an `expect` property is checking the correctness of a Testaro
 
 ## Batches
 
-In some cases you may wish to repeatedly run Testaro with the same script, changing only its `url` commands. The purpose would be to perform the same set of tests on multiple web pages. Such a use would apply only to scripts whose `url` commands are all identical, not to a script that moves from one host to another.
+You may wish to have Testaro perform the same sequence of tests on multiple web pages. In that case, you can create a _batch_, with the following structure:
 
-Testaro does not support such batch processing, but Testilo does. See its `README.md` file for instructions.
+```javascript
+{
+  what: 'Web leaders',
+  hosts: {
+    id: 'w3c',
+    which: 'https://www.w3.org/',
+    what: 'W3C'
+  },
+  {
+    id: 'wikimedia',
+    which: 'https://www.wikimedia.org/',
+    what: 'Wikimedia'
+  }
+}
+```
+
+With a batch, you can execute a single statement to run a script multiple times, one per host. On each call, Testaro takes one of the hosts in the batch and substitutes it for each host specified in a `url` command of the script. Testaro thereby creates and sequentially runs multiple scripts.
 
 ## Execution
 
 ### Invocation
 
-To run Testaro, create a report object like this:
+There are two methods for using Testaro.
+
+#### Low-level
+
+Create a report object like this:
 
 ```javascript
 const report = {
@@ -412,25 +404,63 @@ const report = {
 };
 ```
 
-Replace `{…}` with a script object, like the example script shown above.
+Replace `{…}` with a script object, like the example script shown above. The low-level method does not allow the use of batches.
+
+Then execute the `run` module with the `report` object as an argument.
+- Another Node.js package that has Testaro as a dependency can execute `require('testaro').run(report)`.
+- In a command environment with the Testaro project directory as the current directory, you can execute `node run report`.
+
+Either statement will make Testaro run the script and populate the `log` and `acts` arrays of the `report` object. When Testaro finishes, the `log` and `acts` properties will contain the results.
+
+You or a dependent package can then save or further process the `report` object as desired.
+
+#### High-level
+
+Make sure that you have defined these environment variables, with absolute or relative paths to directories as their values:
+- `SCRIPTDIR`
+- `BATCHDIR`
+- `REPORTDIR`
+
+Relative paths must be relative to the Testaro project directory. For example, if the script directory is `scripts` in a `testing` directory that is a sibling of the Testaro directory, then `SCRIPTDIR` must have the value `../testing/scripts`.
 
-Then execute the statement `require('testaro').handleRequest(report)`. That statement will run Testaro.
+Also ensure that Testaro can read all those directories and write to `REPORTDIR`.
 
-While it runs, Testaro will populate the `log` and `acts` arrays of the report object. When Testaro finishes, the `log` and `acts` properties will contain its results.
+Place a script into `SCRIPTDIR` and, optionally, a batch into `BATCHDIR`. Each should be named `idValue.json`, where `idValue` is replaced with the value of its `id` property. That value must consist of only lower-case ASCII letters and digits.
 
-Another way to run Testaro is to use Testilo, which can handle batches and saves results to files. Testilo prepopulates the report object with an `id` property consisting of a timestamp and, if a batch is used, the host ID. If Testaro finds a non-empty `id` property in the `report` object, Testaro leaves it unchanged; if not, Testaro creates an `id` property with a timestamp value.
+Then execute the statement `node job scriptID` or `node job scriptID batchID`, replacing `scriptID` and `batchID` with the `id` values of the script and the batch, respectively.
+
+The `job` module will call the `run` module on the script, or, if there is a batch, will create multiple scripts, one per host, and sequentially call the `run` module on each script. The results will be saved in report files in the `REPORTDIR` directory.
+
+If there is no batch, the report file will be named with a unique timestamp, suffixed with a `.json` extension. If there is a batch, then the base of each file’s name will be the same timestamp, suffixed with `-hostID`, where `hostID` is the value of the `id` property of the `host` object in the batch file. For example, if you execute `node job script01 wikis`, you might get these report files deposited into `REPORTDIR`:
+- `enp46j-wikipedia.json`
+- `enp45j-wiktionary.json`
+- `enp45j-wikidata.json`
 
 ### Environment variables
 
+As mentioned above, using the high-level method to run Testaro jobs requires `SCRIPTDIR`, `BATCHDIR`, and `REPORTDIR` environment variables.
+
+If a `tenon` test is included in the script, environment variables named `TESTARO_TENON_USER` and `TESTARO_TENON_PASSWORD` must exist, with your Tenon username and password, respectively, as their values.
+
 If a `wave` test is included in the script, an environment variable named `TESTARO_WAVE_KEY` must exist, with your WAVE API key as its value.
 
+The `text` command can interpolate the value of an environment variable into text that it enters on a page, as documented in the `commands.js` file.
+
 Before executing a Testaro script, you can optionally also set the environment variables `TESTARO_DEBUG` (to `'true'` or anything else) and/or `TESTARO_WAITS` (to a non-negative integer). The effects of these variables are described in the `index.js` file.
 
+You may store these environment variables in an untracked `.env` file if you wish, and Testaro will recognize them.
+
 ## Validation
 
+### Samples
+
+The `samples` directory contains scripts and a batch that you can use to test Testaro with with the high-level, by giving `SCRIPTDIR` the value `'samples/scripts'` and `BATCHDIR` the value `'samples/batches'`. Do to this, you must also define `REPORTDIR`.
+
+### Validators
+
 _Executors_ for Testaro validation are located in the `validation` directory.
 
-A basic executor is the `test.js` file. It runs Testaro with a simple sample script and outputs the log and the acts.
+A basic executor is the `test.js` file. It uses the low-level method to run Testaro with the `simple.js` sample script and outputs the log and the acts to the standard output.
 
 The other executors are commonJS JavaScript modules that run Testaro and report whether the results are correct.
 
@@ -438,6 +468,8 @@ The other executors are:
 - `app.js`: Reports whether Testaro runs correctly with a script.
 - `tests.js`: Runs Testaro with each custom test and reports whether the results are correct.
 
+There are no executors for validating the test packages.
+
 To execute any executor `xyz.js`, call it with the statement `node validation/executors/xyz`. The results will appear in the standard output.
 
 The `tests.js` executor makes use of the scripts in the `validation/tests/scripts` directory, and they, in turn, run tests on HTML files in the `validation/tests/targets` directory.
@@ -448,22 +480,7 @@ You can define additional Testaro commands and functionality. Contributions are
 
 ## Accessibility principles
 
-The rationales motivating the Testaro-defined tests and scoring procs can be found in comments within the files of those tests and procs, in the `tests` and `procs/score` directories. Unavoidably, each test is opinionated. Testaro itself, however, can accommodate other tests representing different opinions. Testaro is intended to be neutral with respect to questions such as the criteria for accessibility, the severities of accessibility issues, whether accessibility is binary or graded, and the distinction between usability and accessibility.
-
-### Future work
-
-Further development is contemplated, is taking place, or is welcomed, on:
-- addition of Tenon to the set of packages
-- links with href="#"
-- links and buttons styled non-distinguishably
-- first focused element not first focusable element in DOM
-- never-visible skip links
-- buttons with no text content
-- modal dialogs
-- autocomplete attributes
-- inclusion of other test packages, such as:
-   - FAE (https://github.com/opena11y/evaluation-library)
-   - Tenon
+The rationales motivating the Testaro-defined tests can be found in comments within the files of those tests, in the `tests` directory. Unavoidably, each test is opinionated. Testaro itself, however, can accommodate other tests representing different opinions. Testaro is intended to be neutral with respect to questions such as the criteria for accessibility, the severities of accessibility issues, whether accessibility is binary or graded, and the distinction between usability and accessibility.
 
 ## Testing challenges
 
@@ -475,22 +492,74 @@ The Playwright “Receives Events” actionability check does **not** check whet
 
 ### Test-package duplication
 
-Test packages sometimes do redundant testing, in that two or more packages test for the same issues. But such duplications are not necessarily perfect. Therefore, the scoring procs currently defined by Testaro do not select a single package to test for a single issue. Instead, they allow all packages to test for all the issues they can test for, but decrease the weights placed on issues that multiple packages test for. The more packages test for an issue, the smaller the weight placed on each package’s finding of that issue.
+Test packages sometimes do redundant testing, in that two or more packages test for the same issues, although such duplications are not necessarily perfect. This fact creates three problems:
+- One cannot be confident in excluding some tests of some packages on the assumption that they perfectly duplicate tests of other packages.
+- The Testaro report from a script documents each package’s results separately, so a single difect may be documented in multiple locations within the report, making the consumption of the report inefficient.
+- An effort to aggregate the results into a single score may distort the scores by inflating the weights of defects that happen to be discovered by multiple packages.
+
+The tests provided with Testaro do not exclude any apparently duplicative tests from packages.
+
+To deal with the above problems, you can:
+- revise package `test` commands to exclude tests that you consider duplicative
+- create derivative reports that organize results by defect types rather than by package
+- take duplication into account when defining scoring rules
+
+Some measures of these kinds are included in the scoring and reporting features of the Testilo package.
 
 ## Repository exclusions
 
-The files in the `temp` directory are presumed ephemeral and are not tracked by `git`. When tests require temporary files to be written, Testaro writes them there.
+The files in the `temp` directory are presumed ephemeral and are not tracked by `git`.
 
-## Origin
+## Related packages
 
-Testaro is derived from [Autotest](https://github.com/jrpool/autotest), which in turn is derived from accessibility test investigations beginning in 2018.
+[Testilo](https://www.npmjs.com/package/testilo) is an application that facilitates the use of Testaro.
+
+Testaro is derived from [Autotest](https://github.com/jrpool/autotest).
 
 Testaro omits some functionalities of Autotest, such as:
 - tests producing results intended to be human-inspected
-- previous versions of scoring algorithms
+- scoring
 - file operations for score aggregation, report revision, and HTML reports
 - a web user interface
 
+## Origin
+
+Work on the custom tests in this package began in 2017, and work on the multi-package federation that Testaro implements began in early 2018. These two aspects were combined into the [Autotest](https://github.com/jrpool/autotest) package in early 2021 and into this more single-purpose package, Testaro, in January 2022.
+
 ## Etymology
 
 “Testaro” means “collection of tests” in Esperanto.
+
+## Future work
+
+### Improvements
+
+Further development is contemplated, is taking place, or is welcomed, on:
+- addition of Tenon to the set of packages
+- links with href="#"
+- links and buttons styled non-distinguishably
+- first focused element not first focusable element in DOM
+- never-visible skip links
+- buttons with no text content
+- modal dialogs
+- autocomplete attributes
+- inclusion of other test packages, such as:
+   - FAE (https://github.com/opena11y/evaluation-library)
+   - Tenon
+
+## Corrections
+
+Issues found or reported with the current version that need diagnosis and correction include:
+
+### hover
+
+There seem to be a couple of problems with the hover test:
+- The score for unhoverability is documented as 2 times the count of unhoverables, but is reported as 1 time that count.
+- The list of unhoverables in the report is empty.
+Observed after inquiry by Tobias Christian Jensen of Siteimprove on 2022-05-09.
+
+### axe
+
+Configuration to include best practices and experimental tests.
+
+Investigation of tags, including wcag2a, wcag2aa, wcag21a, wcag21aa, best-practice, wcag***, ACT, cat.*.

package/commands.js

@@ -89,13 +89,6 @@ exports.commands = {
         what: [false, 'string', 'hasLength', 'comment']
       }
     ],
-    score: [
-      'Compute and report a score',
-      {
-        which: [true, 'string', 'hasLength', 'score-proc name'],
-        what: [false, 'string', 'hasLength', 'comment']
-      }
-    ],
     select: [
       'Select a select option',
       {
@@ -118,6 +111,14 @@ exports.commands = {
         what: [false, 'string', 'hasLength', 'comment']
       }
     ],
+    tenonRequest: [
+      'Request a Tenon test',
+      {
+        id: [true, 'string', 'hasLength', 'ID for this test instance'],
+        withNewContent: [true, 'boolean', '', 'true: use a URL; false: use page content'],
+        what: [false, 'string', 'hasLength', 'comment']
+      }
+    ],
     text: [
       'Enter text into a text input, optionally with 1 placeholder for an all-caps literal environment variable',
       {
@@ -233,6 +234,12 @@ exports.commands = {
         withItems: [true, 'boolean']
       }
     ],
+    tenon: [
+      'Perform a Tenon test',
+      {
+        id: [true, 'string', 'hasLength', 'ID of the requested test instance']
+      }
+    ],
     wave: [
       'Perform a WebAIM WAVE test',
       {

package/job.js

@@ -0,0 +1,92 @@
+/*
+  job.js
+  Manages jobs.
+*/
+
+// ########## IMPORTS
+
+// Module to keep secrets.
+require('dotenv').config();
+// Module to read and write files.
+const fs = require('fs/promises');
+const { handleRequest } = require('./run');
+
+// ########## CONSTANTS
+const scriptDir = process.env.SCRIPTDIR;
+const batchDir = process.env.BATCHDIR;
+const reportDir = process.env.REPORTDIR;
+
+// ########## FUNCTIONS
+
+// Converts a script to a batch-based array of scripts.
+const batchify = (script, batch, timeStamp) => {
+  const {hosts} = batch;
+  const specs = hosts.map(host => {
+    const newScript = JSON.parse(JSON.stringify(script));
+    newScript.commands.forEach(command => {
+      if (command.type === 'url') {
+        command.which = host.which;
+        command.what = host.what;
+      }
+    });
+    const spec = {
+      id: `${timeStamp}-${host.id}`,
+      script: newScript
+    };
+    return spec;
+  });
+  return specs;
+};
+// Runs a no-batch script.
+const runHost = async (id, script) => {
+  const report = {
+    id,
+    log: [],
+    script,
+    acts: []
+  };
+  await require('./run').handleRequest(report);
+  const reportJSON = JSON.stringify(report, null, 2);
+  await fs.writeFile(`${reportDir}/${id}.json`, reportJSON);
+};
+// Runs a job.
+exports.handleRequest = async (scriptID, batchID) => {
+  if (scriptID) {
+    try {
+      const scriptJSON = await fs.readFile(`${scriptDir}/${scriptID}.json`, 'utf8');
+      const script = JSON.parse(scriptJSON);
+      // Identify the start time and a timestamp.
+      const timeStamp = Math.floor((Date.now() - Date.UTC(2022, 1)) / 2000).toString(36);
+      // If there is a batch:
+      let batch = null;
+      if (batchID) {
+        // Convert the script to a batch-based set of scripts.
+        const batchJSON = await fs.readFile(`${batchDir}/${batchID}.json`, 'utf8');
+        batch = JSON.parse(batchJSON);
+        const specs = batchify(script, batch, timeStamp);
+        // For each script:
+        while (specs.length) {
+          const spec = specs.shift();
+          const {id, script} = spec;
+          // Run it and save the result with a host-suffixed ID.
+          await runHost(id, script);
+        }
+      }
+      // Otherwise, i.e. if there is no batch:
+      else {
+        // Run the script and save the result with a timestamp ID.
+        await runHost(timeStamp, script);
+      }
+    }
+    catch(error) {
+      console.log(`ERROR: ${error.message}\n${error.stack}`);
+    }
+  }
+  else {
+    console.log('ERROR: no script specified');
+  }
+};
+
+// ########## OPERATION
+
+handleRequest(process.argv[2], process.argv[3]);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "3.0.1",
+  "version": "4.1.0",
   "description": "Automation of accessibility testing",
   "main": "index.js",
   "scripts": {
@@ -30,6 +30,7 @@
     "aatt": "*",
     "accessibility-checker": "*",
     "axe-playwright": "*",
+    "dotenv": "*",
     "pixelmatch": "*",
     "playwright": "*"
   },