testaro 4.0.1 → 4.1.0

package/README.md

@@ -6,9 +6,7 @@ Federated accessibility test automation
 
 Testaro is a collection of collections of web accessibility tests.
 
-The purpose of Testaro is to provide programmatic access to over 600 accessibility tests defined in several test packages and in Testaro itself.
-
-Running Testaro requires telling it which operations (including tests) to perform and which URLs to perform them on, and giving Testaro an object to put its output into.
+The purpose of Testaro is to provide programmatic access to over 800 accessibility tests defined in several test packages and in Testaro itself.
 
 ## System requirements
 
@@ -72,7 +70,7 @@ Once you have done that, you can install Testaro as you would install any `npm`
 
 ## Payment
 
-All of the tests that Testaro can perform are free of cost, except those in the WAVE package. WebAIM requires an API key for those tests. If you wish to have Testaro perform the WAVE tests, you will need to have a WAVE API key. Visit the URL above in order to obtain your key. It costs 1 to 3 credits to perform the WAVE tests on one URL. WebAIM gives you 100 credits without cost, before you need to begin paying.
+All of the tests that Testaro can perform are free of cost, except those in the Tenon and WAVE packages. The owner of each of those packages gives new registrants a free allowance of credits before it becomes necessary to pay for use of the API of the package. The required environment variables for authentication and payment are described below under “Environment variables”.
 
 ## Specification
 
@@ -82,7 +80,7 @@ To use Testaro, you must specify what it should do. You do this with a script an
 
 ### Introduction
 
-When you run Testaro, you provide a **script** to it. The script contains **commands**. Testaro performs those commands.
+To use Testaro, you provide a **script** to it. The script contains **commands**. Testaro __runs__ the script, i.e. performs the commands in it and writes a report of the results.
 
 A script is a JSON file with the properties:
 
@@ -163,7 +161,6 @@ The subsequent commands can tell Testaro to perform any of:
 - navigations (browser launches, visits to URLs, waits for page conditions, etc.)
 - alterations (changes to the page)
 - tests (whether in dependency packages or defined within Testaro)
-- scoring (aggregating test results into total scores)
 - branching (continuing from a command other than the next one)
 
 ##### Moves
@@ -272,37 +269,6 @@ In case you want to perform more than one `tenon` test, you can do so. Just give
 
 Tenon recommends giving it a public URL rather than giving it the content of a page, if possible. So, it is best to give the `withNewContent` property of the `tenonRequest` command the value `true`, unless the page is not public.
 
-##### Scoring
-
-An example of a **scoring** command is:
-
-```json
-{
-  "type": "score",
-  "which": "asp09",
-  "what": "5 packages and 16 custom tests, with duplication discounts"
-}
-```
-
-In this case, Testaro executes the procedure specified in the `asp09` score proc (in the `procs/score` directory) to compute a total score for the script or (if there is a batch) the host. The proc is a JavaScript module whose `scorer` function returns an object containing a total score and the itemized scores that yield the total.
-
-The `scorer` function inspects the script report to find the required data, applies specific weights and formulas to yield the itemized scores, and combines the itemized scores to yield the total score.
-
-The data for scores can include not only test results, but also log statistics. Testaro includes in each report the properties:
-- `logCount`: how many log items the browser generated
-- `logSize`: how large the log items were in the aggregate, in characters
-- `prohibitedCount`: how many log items contain (case-insensitively) `403` and `status`, or `prohibited`
-- `visitTimeoutCount`: how many times an attempt to visit a URL timed out
-- `visitRejectionCount`: how many times a URL visit got an HTTP status other than 200 or 304
-
-Those log statistics can provide data for a log-based test defined in a score proc.
-
-A good score proc takes account of duplications between test packages: two or more packages that discover the same accessibility defects. Score procs can apply discounts to reflect duplications between test packages, so that, if two or more packages discover the same defect, the defect will not be overweighted.
-
-The procedures in the `scoring` directory have produced data there that score procs can use for the calibration of discounts.
-
-Some documents are implemented in such a way that some tests are prevented from being conducted on them. When that occurs, the score proc can **infer** a score for that test.
-
 ##### Branching
 
 An example of a **branching** command is:
@@ -399,15 +365,35 @@ A typical use for an `expect` property is checking the correctness of a Testaro
 
 ## Batches
 
-In some cases you may wish to repeatedly run Testaro with the same script, changing only its `url` commands. The purpose would be to perform the same set of tests on multiple web pages. Such a use would apply only to scripts whose `url` commands are all identical, not to a script that moves from one host to another.
+You may wish to have Testaro perform the same sequence of tests on multiple web pages. In that case, you can create a _batch_, with the following structure:
+
+```javascript
+{
+  what: 'Web leaders',
+  hosts: {
+    id: 'w3c',
+    which: 'https://www.w3.org/',
+    what: 'W3C'
+  },
+  {
+    id: 'wikimedia',
+    which: 'https://www.wikimedia.org/',
+    what: 'Wikimedia'
+  }
+}
+```
 
-Testaro does not support such batch processing, but Testilo does. See its `README.md` file for instructions.
+With a batch, you can execute a single statement to run a script multiple times, one per host. On each call, Testaro takes one of the hosts in the batch and substitutes it for each host specified in a `url` command of the script. Testaro thereby creates and sequentially runs multiple scripts.
 
 ## Execution
 
 ### Invocation
 
-To run Testaro, create a report object like this:
+There are two methods for using Testaro.
+
+#### Low-level
+
+Create a report object like this:
 
 ```javascript
 const report = {
@@ -418,20 +404,46 @@ const report = {
 };
 ```
 
-Replace `{…}` with a script object, like the example script shown above.
+Replace `{…}` with a script object, like the example script shown above. The low-level method does not allow the use of batches.
+
+Then execute the `run` module with the `report` object as an argument.
+- Another Node.js package that has Testaro as a dependency can execute `require('testaro').run(report)`.
+- In a command environment with the Testaro project directory as the current directory, you can execute `node run report`.
+
+Either statement will make Testaro run the script and populate the `log` and `acts` arrays of the `report` object. When Testaro finishes, the `log` and `acts` properties will contain the results.
+
+You or a dependent package can then save or further process the `report` object as desired.
+
+#### High-level
+
+Make sure that you have defined these environment variables, with absolute or relative paths to directories as their values:
+- `SCRIPTDIR`
+- `BATCHDIR`
+- `REPORTDIR`
+
+Relative paths must be relative to the Testaro project directory. For example, if the script directory is `scripts` in a `testing` directory that is a sibling of the Testaro directory, then `SCRIPTDIR` must have the value `../testing/scripts`.
+
+Also ensure that Testaro can read all those directories and write to `REPORTDIR`.
+
+Place a script into `SCRIPTDIR` and, optionally, a batch into `BATCHDIR`. Each should be named `idValue.json`, where `idValue` is replaced with the value of its `id` property. That value must consist of only lower-case ASCII letters and digits.
 
-Then execute the statement `require('testaro').handleRequest(report)`. That statement will run Testaro.
+Then execute the statement `node job scriptID` or `node job scriptID batchID`, replacing `scriptID` and `batchID` with the `id` values of the script and the batch, respectively.
 
-While it runs, Testaro will populate the `log` and `acts` arrays of the report object. When Testaro finishes, the `log` and `acts` properties will contain its results.
+The `job` module will call the `run` module on the script, or, if there is a batch, will create multiple scripts, one per host, and sequentially call the `run` module on each script. The results will be saved in report files in the `REPORTDIR` directory.
 
-Another way to run Testaro is to use Testilo, which can handle batches and saves results to files. Testilo prepopulates the report object with an `id` property consisting of a timestamp and, if a batch is used, the host ID. If Testaro finds a non-empty `id` property in the `report` object, Testaro leaves it unchanged; if not, Testaro creates an `id` property with a timestamp value.
+If there is no batch, the report file will be named with a unique timestamp, suffixed with a `.json` extension. If there is a batch, then the base of each file’s name will be the same timestamp, suffixed with `-hostID`, where `hostID` is the value of the `id` property of the `host` object in the batch file. For example, if you execute `node job script01 wikis`, you might get these report files deposited into `REPORTDIR`:
+- `enp46j-wikipedia.json`
+- `enp45j-wiktionary.json`
+- `enp45j-wikidata.json`
 
 ### Environment variables
 
-If a `wave` test is included in the script, an environment variable named `TESTARO_WAVE_KEY` must exist, with your WAVE API key as its value.
+As mentioned above, using the high-level method to run Testaro jobs requires `SCRIPTDIR`, `BATCHDIR`, and `REPORTDIR` environment variables.
 
 If a `tenon` test is included in the script, environment variables named `TESTARO_TENON_USER` and `TESTARO_TENON_PASSWORD` must exist, with your Tenon username and password, respectively, as their values.
 
+If a `wave` test is included in the script, an environment variable named `TESTARO_WAVE_KEY` must exist, with your WAVE API key as its value.
+
 The `text` command can interpolate the value of an environment variable into text that it enters on a page, as documented in the `commands.js` file.
 
 Before executing a Testaro script, you can optionally also set the environment variables `TESTARO_DEBUG` (to `'true'` or anything else) and/or `TESTARO_WAITS` (to a non-negative integer). The effects of these variables are described in the `index.js` file.
@@ -440,9 +452,15 @@ You may store these environment variables in an untracked `.env` file if you wis
 
 ## Validation
 
+### Samples
+
+The `samples` directory contains scripts and a batch that you can use to test Testaro with with the high-level, by giving `SCRIPTDIR` the value `'samples/scripts'` and `BATCHDIR` the value `'samples/batches'`. Do to this, you must also define `REPORTDIR`.
+
+### Validators
+
 _Executors_ for Testaro validation are located in the `validation` directory.
 
-A basic executor is the `test.js` file. It runs Testaro with a simple sample script and outputs the log and the acts.
+A basic executor is the `test.js` file. It uses the low-level method to run Testaro with the `simple.js` sample script and outputs the log and the acts to the standard output.
 
 The other executors are commonJS JavaScript modules that run Testaro and report whether the results are correct.
 
@@ -450,6 +468,8 @@ The other executors are:
 - `app.js`: Reports whether Testaro runs correctly with a script.
 - `tests.js`: Runs Testaro with each custom test and reports whether the results are correct.
 
+There are no executors for validating the test packages.
+
 To execute any executor `xyz.js`, call it with the statement `node validation/executors/xyz`. The results will appear in the standard output.
 
 The `tests.js` executor makes use of the scripts in the `validation/tests/scripts` directory, and they, in turn, run tests on HTML files in the `validation/tests/targets` directory.
@@ -460,7 +480,7 @@ You can define additional Testaro commands and functionality. Contributions are
 
 ## Accessibility principles
 
-The rationales motivating the Testaro-defined tests and scoring procs can be found in comments within the files of those tests and procs, in the `tests` and `procs/score` directories. Unavoidably, each test is opinionated. Testaro itself, however, can accommodate other tests representing different opinions. Testaro is intended to be neutral with respect to questions such as the criteria for accessibility, the severities of accessibility issues, whether accessibility is binary or graded, and the distinction between usability and accessibility.
+The rationales motivating the Testaro-defined tests can be found in comments within the files of those tests, in the `tests` directory. Unavoidably, each test is opinionated. Testaro itself, however, can accommodate other tests representing different opinions. Testaro is intended to be neutral with respect to questions such as the criteria for accessibility, the severities of accessibility issues, whether accessibility is binary or graded, and the distinction between usability and accessibility.
 
 ## Testing challenges
 
@@ -472,11 +492,23 @@ The Playwright “Receives Events” actionability check does **not** check whet
 
 ### Test-package duplication
 
-Test packages sometimes do redundant testing, in that two or more packages test for the same issues. But such duplications are not necessarily perfect. Therefore, the scoring procs currently defined by Testaro do not select a single package to test for a single issue. Instead, they allow all packages to test for all the issues they can test for, but decrease the weights placed on issues that multiple packages test for. The more packages test for an issue, the smaller the weight placed on each package’s finding of that issue.
+Test packages sometimes do redundant testing, in that two or more packages test for the same issues, although such duplications are not necessarily perfect. This fact creates three problems:
+- One cannot be confident in excluding some tests of some packages on the assumption that they perfectly duplicate tests of other packages.
+- The Testaro report from a script documents each package’s results separately, so a single difect may be documented in multiple locations within the report, making the consumption of the report inefficient.
+- An effort to aggregate the results into a single score may distort the scores by inflating the weights of defects that happen to be discovered by multiple packages.
+
+The tests provided with Testaro do not exclude any apparently duplicative tests from packages.
+
+To deal with the above problems, you can:
+- revise package `test` commands to exclude tests that you consider duplicative
+- create derivative reports that organize results by defect types rather than by package
+- take duplication into account when defining scoring rules
+
+Some measures of these kinds are included in the scoring and reporting features of the Testilo package.
 
 ## Repository exclusions
 
-The files in the `temp` directory are presumed ephemeral and are not tracked by `git`. When tests require temporary files to be written, Testaro writes them there.
+The files in the `temp` directory are presumed ephemeral and are not tracked by `git`.
 
 ## Related packages
 
@@ -486,7 +518,7 @@ Testaro is derived from [Autotest](https://github.com/jrpool/autotest).
 
 Testaro omits some functionalities of Autotest, such as:
 - tests producing results intended to be human-inspected
-- previous versions of scoring algorithms
+- scoring
 - file operations for score aggregation, report revision, and HTML reports
 - a web user interface
 

package/commands.js

@@ -89,13 +89,6 @@ exports.commands = {
         what: [false, 'string', 'hasLength', 'comment']
       }
     ],
-    score: [
-      'Compute and report a score',
-      {
-        which: [true, 'string', 'hasLength', 'score-proc name'],
-        what: [false, 'string', 'hasLength', 'comment']
-      }
-    ],
     select: [
       'Select a select option',
       {

package/job.js

@@ -0,0 +1,92 @@
+/*
+  job.js
+  Manages jobs.
+*/
+
+// ########## IMPORTS
+
+// Module to keep secrets.
+require('dotenv').config();
+// Module to read and write files.
+const fs = require('fs/promises');
+const { handleRequest } = require('./run');
+
+// ########## CONSTANTS
+const scriptDir = process.env.SCRIPTDIR;
+const batchDir = process.env.BATCHDIR;
+const reportDir = process.env.REPORTDIR;
+
+// ########## FUNCTIONS
+
+// Converts a script to a batch-based array of scripts.
+const batchify = (script, batch, timeStamp) => {
+  const {hosts} = batch;
+  const specs = hosts.map(host => {
+    const newScript = JSON.parse(JSON.stringify(script));
+    newScript.commands.forEach(command => {
+      if (command.type === 'url') {
+        command.which = host.which;
+        command.what = host.what;
+      }
+    });
+    const spec = {
+      id: `${timeStamp}-${host.id}`,
+      script: newScript
+    };
+    return spec;
+  });
+  return specs;
+};
+// Runs a no-batch script.
+const runHost = async (id, script) => {
+  const report = {
+    id,
+    log: [],
+    script,
+    acts: []
+  };
+  await require('./run').handleRequest(report);
+  const reportJSON = JSON.stringify(report, null, 2);
+  await fs.writeFile(`${reportDir}/${id}.json`, reportJSON);
+};
+// Runs a job.
+exports.handleRequest = async (scriptID, batchID) => {
+  if (scriptID) {
+    try {
+      const scriptJSON = await fs.readFile(`${scriptDir}/${scriptID}.json`, 'utf8');
+      const script = JSON.parse(scriptJSON);
+      // Identify the start time and a timestamp.
+      const timeStamp = Math.floor((Date.now() - Date.UTC(2022, 1)) / 2000).toString(36);
+      // If there is a batch:
+      let batch = null;
+      if (batchID) {
+        // Convert the script to a batch-based set of scripts.
+        const batchJSON = await fs.readFile(`${batchDir}/${batchID}.json`, 'utf8');
+        batch = JSON.parse(batchJSON);
+        const specs = batchify(script, batch, timeStamp);
+        // For each script:
+        while (specs.length) {
+          const spec = specs.shift();
+          const {id, script} = spec;
+          // Run it and save the result with a host-suffixed ID.
+          await runHost(id, script);
+        }
+      }
+      // Otherwise, i.e. if there is no batch:
+      else {
+        // Run the script and save the result with a timestamp ID.
+        await runHost(timeStamp, script);
+      }
+    }
+    catch(error) {
+      console.log(`ERROR: ${error.message}\n${error.stack}`);
+    }
+  }
+  else {
+    console.log('ERROR: no script specified');
+  }
+};
+
+// ########## OPERATION
+
+handleRequest(process.argv[2], process.argv[3]);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "4.0.1",
+  "version": "4.1.0",
   "description": "Automation of accessibility testing",
   "main": "index.js",
   "scripts": {

package/{index.js → run.js}

@@ -7,6 +7,7 @@
 require('dotenv').config();
 // Requirements for commands.
 const {commands} = require('./commands');
+const { handleRequest } = require('./job');
 // ########## CONSTANTS
 // Set DEBUG environment variable to 'true' to add debugging features.
 const debug = process.env.TESTARO_DEBUG === 'true';
@@ -613,17 +614,6 @@ const doActs = async (report, actIndex, page) => {
         // Identify its only page as current.
         page = browserContext.pages()[0];
       }
-      // Otherwise, if it is a score:
-      else if (act.type === 'score') {
-        // Compute and report the score.
-        try {
-          const {scorer} = require(`./procs/score/${act.which}`);
-          act.result = scorer(report.acts);
-        }
-        catch (error) {
-          act.error = `ERROR: ${error.message}\n${error.stack}`;
-        }
-      }
       // Otherwise, if a current page exists:
       else if (page) {
         // If the command is a url:
@@ -1200,25 +1190,6 @@ const doScript = async (report) => {
   report.prohibitedCount = prohibitedCount;
   report.visitTimeoutCount = visitTimeoutCount;
   report.visitRejectionCount = visitRejectionCount;
-  // If logs are to be scored, do so.
-  const scoreTables = report.acts.filter(act => act.type === 'score');
-  if (scoreTables.length) {
-    const scoreTable = scoreTables[0];
-    const {result} = scoreTable;
-    if (result) {
-      const {logWeights, scores} = result;
-      if (logWeights && scores) {
-        scores.log = Math.floor(
-          logWeights.count * logCount
-          + logWeights.size * logSize
-          + logWeights.prohibited * prohibitedCount
-          + logWeights.visitTimeout * visitTimeoutCount
-          + logWeights.visitRejection * visitRejectionCount
-        );
-        scores.total += scores.log;
-      }
-    }
-  }
   // Add the end time and duration to the report.
   const endTime = new Date();
   report.endTime = endTime.toISOString().slice(0, 19);
@@ -1285,3 +1256,7 @@ exports.handleRequest = async report => {
     console.log('ERROR: options missing or invalid');
   }
 };
+
+// ########## OPERATION
+
+handleRequest(process.argv[2]);

package/samples/batches/weborgs.json

@@ -0,0 +1,16 @@
+{
+  "id": "weborgs",
+  "what": "Web organizations",
+  "hosts": [
+    {
+      "id": "mozilla",
+      "which": "https://www.mozilla.org/en-US/",
+      "what": "Mozilla"
+    },
+    {
+      "id": "w3c",
+      "which": "https://www.w3.org/",
+      "what": "W3C"
+    }
+  ]
+}

package/scoring/correlation.js

@@ -1,76 +0,0 @@
-/*
-  correlation
-  Compiles a list of the correlations between distinct-package issue types and creates a file,
-  correlations.json, containing the list.
-*/
-const fs = require('fs');
-const compile = () => {
-  const issuesJSON = fs.readFileSync(`${__dirname}/package/issues.json`, 'utf8');
-  const issues = JSON.parse(issuesJSON);
-  const dataJSON = fs.readFileSync(`${__dirname}/package/data.json`, 'utf8');
-  const reportData = JSON.parse(dataJSON);
-  const reports = Object.values(reportData);
-  // Initialize the list.
-  const data = {
-    aatt_alfa: {},
-    aatt_axe: {},
-    aatt_ibm: {},
-    aatt_wave: {},
-    alfa_axe: {},
-    alfa_ibm: {},
-    alfa_wave: {},
-    axe_ibm: {},
-    axe_wave: {},
-    ibm_wave: {}
-  };
-  // For each pair of packages:
-  const packagePairs = Object.keys(data);
-  packagePairs.forEach(packagePair => {
-    console.log(`=== Starting package pair ${packagePair}`);
-    const packages = packagePair.split('_');
-    // Identify the reports containing results from both packages.
-    const pairReports = reports.filter(report => report[packages[0]] && report[packages[1]]);
-    // For each pair of issues:
-    issues[packages[0]].forEach(issueA => {
-      issues[packages[1]].forEach(issueB => {
-        // Initialize an array of score pairs.
-        const scorePairs = [];
-        // For each applicable report:
-        pairReports.forEach(report => {
-          // Add the scores for the issues to the array of score pairs.
-          const scorePair = [report[packages[0]][issueA] || 0, report[packages[1]][issueB] || 0];
-          scorePairs.push(scorePair);
-        });
-        // Get the correlation between the issues.
-        const aSum = scorePairs.reduce((sum, current) => sum + current[0], 0);
-        const bSum = scorePairs.reduce((sum, current) => sum + current[1], 0);
-        const abSum = scorePairs.reduce((sum, current) => sum + current[0] * current[1], 0);
-        const aSqSum = scorePairs.reduce((sum, current) => sum + current[0] ** 2, 0);
-        const bSqSum = scorePairs.reduce((sum, current) => sum + current[1] ** 2, 0);
-        const n = scorePairs.length;
-        const correlation
-          = (abSum - aSum * bSum / n) / n
-          / (Math.sqrt(aSqSum / n - (aSum / n) ** 2) * Math.sqrt(bSqSum / n - (bSum / n) ** 2));
-        // If the correlation is large enough:
-        if (correlation > 0.7) {
-          const roundedCorr = correlation.toFixed(2);
-          // Record it and the count of non-zero scores.
-          const nonZero = scorePairs.reduce(
-            (count, current) => count + current.filter(score => score).length, 0
-          );
-          const corrPlusNZ = `${roundedCorr} (${nonZero})`;
-          if (data[packagePair][issueA]) {
-            data[packagePair][issueA][issueB] = corrPlusNZ;
-          }
-          else {
-            data[packagePair][issueA] = {[issueB]: corrPlusNZ};
-          }
-        }
-      });
-    });
-  });
-  return data;
-};
-fs.writeFileSync(
-  `${__dirname}/package/correlations.json`, JSON.stringify(compile(), null, 2)
-);