testaro 2.3.9 → 3.0.0

package/README.md

@@ -1,23 +1,29 @@
 # testaro
 
-Accessibility test automation
+Federated accessibility test automation
 
 ## Summary
 
-Testaro is a collection of web accessibility tests.
+Testaro is a collection of collections of web accessibility tests.
 
 The purpose of Testaro is to provide programmatic access to over 600 accessibility tests defined in several test packages and in Testaro itself.
 
 Running Testaro requires telling it which operations (including tests) to perform and which URLs to perform them on, and giving Testaro an object to put its output into.
 
-Testaro outputs progress messages to the standard output. It populates the object with log information and test reports.
+## Origin
+
+Work on the custom tests in this package began in 2017, and work on the multi-package federation that Testaro implements began in early 2018. These two aspects were combined into the [Autotest](https://github.com/jrpool/autotest) package in early 2021 and into this more limited-purpose package, Testaro, in January 2022.
+
+Testaro omits some functionalities of Autotest, such as:
+- tests producing results intended to be human-inspected
+- previous versions of scoring algorithms
+- file operations for score aggregation, report revision, and HTML reports
+- a web user interface
 
 ## System requirements
 
 Version 14 or later of [Node.js](https://nodejs.org/en/).
 
-A file system with a directory that the Testaro user has permission to read and write in.
-
 ## Dependencies
 
 Testaro uses:
@@ -41,6 +47,10 @@ As of this version, the counts of tests in the packages referenced above were:
 - Testaro tests: 16
 - grand total: 628
 
+## Related packages
+
+[Testilo](https://github.com/jrpool/testilo) is an application that facilitates the use of Testaro.
+
 ## Code organization
 
 The main directories containing code files are:
@@ -100,25 +110,25 @@ A script is a JSON file with the properties:
 
 Here is an example of a script:
 
-```json
+```javascript
 {
-  "what": "Test example.com with alfa",
-  "strict": true,
-  "commands": [
+  what: 'Test example.com with alfa',
+  strict: true,
+  commands: [
     {
-      "type": "launch",
-      "which": "chromium",
-      "what": "Chromium browser"
+      type: 'launch',
+      which: 'chromium',
+      what: 'Chromium browser'
     },
     {
-      "type": "url",
-      "which": "https://example.com/",
-      "what": "page with a few accessibility defects"
+      type: 'url',
+      which: 'https://example.com/',
+      what: 'page with a few accessibility defects'
     },
     {
-      "type": "test",
-      "which": "alfa",
-      "what": "Siteimprove alfa package"
+      type: 'test',
+      which: 'alfa',
+      what: 'Siteimprove alfa package'
     }
   ]
 }
@@ -383,56 +393,32 @@ A typical use for an `expect` property is checking the correctness of a Testaro
 
 ## Batches
 
-There are two ways to use a script to give instructions to Testaro:
-- The script can be the complete specification of the operations to perform and the URLs to perform them on.
-- The script can specify the operations to perform, and a _batch_ can specify which URLs to perform them on.
+In some cases you may wish to repeatedly run Testaro with the same script, changing only its `url` commands. The purpose would be to perform the same set of tests on multiple web pages. Such a use would apply only to scripts whose `url` commands are all identical, not to a script that moves from one host to another.
 
-A batch is a JSON file with this format:
-
-```json
-{
-  "what": "Accessibility standards",
-  "hosts": [
-    {
-      "which": "https://www.w3.org/TR/WCAG21/",
-      "what": "WCAG 2.1"
-    },
-    {
-      "which": "https://www.w3.org/WAI/standards-guidelines/aria/",
-      "what": "W3C WAI-ARIA"
-    }
-  ]
-}
-```
-
-When you combine a script with a batch, Testaro performs the script, replacing the `which` and `what` properties of all `url` commands with the values in the first object in the `hosts` array of the batch, then again with the values in the second `host` object, and so on. Those replacements also occur in the inserted extra `url` commands mentioned above.
-
-A batch offers an efficient way to perform a uniform set of operations on every host in a set of hosts. In this way you can run the same set of tests on multiple web pages.
-
-A no-batch script offers a way to carry out a complex operation, which can include navigating from one host to another, which is not possible with a batch. Any `url` commands are performed as-is, without changes to their URLs.
+Testaro does not support such batch processing, but Testilo does. See its `README.md` file for instructions.
 
 ## Execution
 
 ### Invocation
 
-To run Testaro, create an options object like this:
+To run Testaro, create a report object like this:
 
 ```javascript
-{
+const report = {
+  id: '',
+  script: {…},
   log: [],
-  reports: [],
-  script: {abc},
-  batch: {def}
-}
+  acts: []
+};
 ```
 
-Replace `{abc}` with a script object, like the example script shown above.
+Replace `{…}` with a script object, like the example script shown above.
 
-Replace `{def}` with a batch object, like the example batch shown above, or omit the `batch` property if there is no batch.
+Then execute the statement `require('testaro').handleRequest(report)`. That statement will run Testaro.
 
-Then execute the statement `require('testaro').handleRequest(ghi)`, where `ghi` is replaced with the name of the options object. That statement will run Testaro.
+While it runs, Testaro will populate the `log` and `acts` arrays of the report object. When Testaro finishes, the `log` and `acts` properties will contain its results.
 
-While it runs, Testaro will populate the `log` and `reports` arrays of the options object. When Testaro finishes, the `log` and `reports` arrays will contain its results.
+Another way to run Testaro is to use Testilo, which can handle batches and saves results to files. Testilo prepopulates the report object with an `id` property consisting of a timestamp and, if a batch is used, the host ID. If Testaro finds a non-empty `id` property in the `report` object, Testaro leaves it unchanged; if not, Testaro creates an `id` property with a timestamp value.
 
 ### Environment variables
 
@@ -442,11 +428,14 @@ Before executing a Testaro script, you can optionally also set the environment v
 
 ## Validation
 
-Three _executors_ for Testaro validation are located in the `validation` directory. An executor is a commonJS JavaScript module that runs Testaro and reports whether the results are correct.
+_Executors_ for Testaro validation are located in the `validation` directory.
 
-The executors are:
-- `appNoBatch.js`: Reports whether Testaro runs correctly with a no-batch script.
-- `appBatch.js`: Reports whether Testaro runs correctly with a script and a batch.
+A basic executor is the `test.js` file. It runs Testaro with a simple sample script and outputs the log and the acts.
+
+The other executors are commonJS JavaScript modules that run Testaro and report whether the results are correct.
+
+The other executors are:
+- `app.js`: Reports whether Testaro runs correctly with a script.
 - `tests.js`: Runs Testaro with each custom test and reports whether the results are correct.
 
 To execute any executor `xyz.js`, call it with the statement `node validation/executors/xyz`. The results will appear in the standard output.
@@ -464,6 +453,7 @@ The rationales motivating the Testaro-defined tests and scoring procs can be fou
 ### Future work
 
 Further development is contemplated, is taking place, or is welcomed, on:
+- addition of Tenon to the set of packages
 - links with href="#"
 - links and buttons styled non-distinguishably
 - first focused element not first focusable element in DOM
@@ -472,8 +462,6 @@ Further development is contemplated, is taking place, or is welcomed, on:
 - modal dialogs
 - autocomplete attributes
 
-Additional test packages may be integratable into Testaro. The [`bbc-a11y`](https://github.com/bbc/bbc-a11y) package has been considered, but it has not been updated since 2018 and has vulnerable dependencies.
-
 ## Testing challenges
 
 ### Activation

package/index.js

@@ -194,38 +194,18 @@ const isValidScript = script => {
     && isURL(commands[1].which)
     && commands.every(command => isValidCommand(command));
 };
-// Validates a batch.
-const isValidBatch = batch => {
-  // If the batch exists:
-  if (batch) {
-    // Get its data.
-    const {what, hosts} = batch;
-    // Return whether the batch is valid:
-    return what
-      && hosts
-      && typeof what === 'string'
-      && Array.isArray(hosts)
-      && hosts.every(host => host.which && host.what && isURL(host.which));
-  }
-  // Otherwise, i.e. if the batch does not exist:
-  else {
-    // Return that it is valid, because it is optional.
-    return true;
-  }
-};
 // Validates an initialized reports array.
-const isValidReports = reports => Array.isArray(reports) && ! reports.length;
+const isValidActs = acts => Array.isArray(acts) && ! acts.length;
 // Validates an initialized log array.
 const isValidLog = log => Array.isArray(log) && ! log.length;
-// Validates an options object.
-const isValidOptions = async options => {
-  if (options) {
-    const {id, script, batch, log, reports} = options;
+// Validates a report object.
+const isValidReport = async report => {
+  if (report) {
+    const {id, script, log, acts} = report;
     return id
     && isValidScript(script)
-    && isValidBatch(batch)
     && isValidLog(log)
-    && isValidReports(reports);
+    && isValidActs(acts);
   }
   else {
     return false;
@@ -569,7 +549,7 @@ const waitError = (page, act, error, what) => {
   act.result.error = `ERROR waiting for ${what}`;
   return false;
 };
-// Recursively performs the commands in a report.
+// Recursively performs the acts in a report.
 const doActs = async (report, actIndex, page) => {
   const {acts} = report;
   // If any more commands are to be performed:
@@ -1009,7 +989,7 @@ const doActs = async (report, actIndex, page) => {
                 // Otherwise, if it is entering text on the element:
                 else if (act.type === 'text') {
                   // If the text contains a placeholder for an environment variable:
-                  let {what} = act; 
+                  let {what} = act;
                   if (/__[A-Z]+__/.test(what)) {
                     // Replace it.
                     const envKey = /__([A-Z]+)__/.exec(what)[1];
@@ -1079,7 +1059,7 @@ const doActs = async (report, actIndex, page) => {
   }
 };
 // Performs the commands in a script.
-const doScript = async (options, report) => {
+const doScript = async (report) => {
   // Reinitialize the log statistics.
   logCount = logSize = prohibitedCount = visitTimeoutCount = visitRejectionCount= 0;
   // Add the start time to the report.
@@ -1122,88 +1102,35 @@ const doScript = async (options, report) => {
   const endTime = new Date();
   report.endTime = endTime.toISOString().slice(0, 19);
   report.elapsedSeconds =  Math.floor((endTime - startTime) / 1000);
-  // Add the report to the reports array.
-  options.reports.push(report);
-};
-// Recursively performs commands on the hosts of a batch.
-const doBatch = async (options, reportTemplate, hostIndex = 0) => {
-  const {hosts} = options.batch;
-  const host = hosts[hostIndex];
-  // If the specified host exists:
-  if (host) {
-    // Create a report for it.
-    const hostReport = JSON.parse(JSON.stringify(reportTemplate));
-    // Copy the properties of the specified host to all url acts.
-    hostReport.acts.forEach(act => {
-      if (act.type === 'url') {
-        act.which = host.which;
-        act.what = host.what;
-      }
-    });
-    // Add the host’s ID to the host report.
-    hostReport.hostName = host.id;
-    // Add data from the template to the host report.
-    hostReport.orderName = options.id;
-    hostReport.id = `${options.id}-${host.id}`;
-    hostReport.orderUserName = options.userName;
-    hostReport.orderTime = options.orderTime;
-    hostReport.assignedBy = options.assignedBy;
-    hostReport.assignedTime = options.assignedTime;
-    hostReport.tester = options.tester;
-    hostReport.scriptName = options.scriptName;
-    hostReport.batchName = options.batchName;
-    hostReport.scriptIsValid = options.scriptIsValid;
-    hostReport.batchIsValid = options.batchIsValid;
-    hostReport.host = host;
-    // Perform the commands on the host and add a report to the options object.
-    await doScript(options, hostReport);
-    // Process the remaining hosts.
-    await doBatch(options, reportTemplate, hostIndex + 1);
-  }
-};
-// Performs a script, with or without a batch.
-const doScriptOrBatch = async (options, reportTemplate) => {
-  // If there is a batch:
-  if (options.batch) {
-    // Perform the script on all the hosts in the batch.
-    console.log('Starting batch');
-    await doBatch(options, reportTemplate);
-  }
-  // Otherwise, i.e. if there is no batch:
-  else {
-    // Perform the script.
-    console.log('Starting no-batch script');
-    await doScript(options, reportTemplate);
-  }
   // Add an end time to the log.
-  options.log.push({
+  report.log.push({
     event: 'endTime',
     value: ((new Date()).toISOString().slice(0, 19))
   });
 };
-// Injects url commands into a report where necessary to undo DOM changes.
-const injectURLCommands = commands => {
+// Injects url acts into a report where necessary to undo DOM changes.
+const injectURLActs = acts => {
   let injectMore = true;
   while (injectMore) {
-    const injectIndex = commands.findIndex((command, index) =>
-      index < commands.length - 1
-      && command.type === 'test'
-      && commands[index + 1].type === 'test'
-      && domChangers.has(command.which)
+    const injectIndex = acts.findIndex((act, index) =>
+      index < acts.length - 1
+      && act.type === 'test'
+      && acts[index + 1].type === 'test'
+      && domChangers.has(act.which)
     );
     if (injectIndex === -1) {
       injectMore = false;
     }
     else {
-      const lastURL = commands.reduce((url, command, index) => {
-        if (command.type === 'url' && index < injectIndex) {
-          return command.which;
+      const lastURL = acts.reduce((url, act, index) => {
+        if (act.type === 'url' && index < injectIndex) {
+          return act.which;
         }
         else {
           return url;
         }
       }, '');
-      commands.splice(injectIndex + 1, 0, {
+      acts.splice(injectIndex + 1, 0, {
         type: 'url',
         which: lastURL,
         what: 'URL'
@@ -1212,36 +1139,26 @@ const injectURLCommands = commands => {
   }
 };
 // Handles a request.
-exports.handleRequest = async options => {
-  // If the options object is valid:
-  if(isValidOptions(options)) {
-    // Add a start time and a timeStamp to the log.
-    options.log.push(
+exports.handleRequest = async report => {
+  // If the report object is valid:
+  if(isValidReport(report)) {
+    // Add a start time to the log.
+    report.log.push(
       {
         event: 'startTime',
         value: ((new Date()).toISOString().slice(0, 19))
-      },
-      {
-        event: 'timeStamp',
-        value: Math.floor((Date.now() - Date.UTC(2022, 1)) / 500).toString(36)
       }
     );
-    // Add the batch size to the log if there is a batch.
-    if (options.batch) {
-      options.log.push({
-        event: 'batchSize',
-        value: options.batch.hosts.length
-      });
+    // Add an ID to the report if none exists yet.
+    if (! report.id) {
+      report.id = Math.floor((Date.now() - Date.UTC(2022, 1)) / 2000).toString(36);
     }
-    // Create a report template, containing a copy of the commands as its acts.
-    const reportTemplate = {
-      host: '',
-      acts: JSON.parse(JSON.stringify(options.script.commands))
-    };
+    // Add the script commands to the report as its initial acts.
+    report.acts = JSON.parse(JSON.stringify(report.script.commands));
     // Inject url acts where necessary to undo DOM changes.
-    injectURLCommands(reportTemplate.acts);
-    // Perform the script, with or without a batch, asynchronously adding to the log and reports.
-    await doScriptOrBatch(options, reportTemplate);
+    injectURLActs(report.acts);
+    // Perform the script, asynchronously adding to the log and report.
+    await doScript(report);
   }
   else {
     console.log('ERROR: options missing or invalid');

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "2.3.9",
+  "version": "3.0.0",
   "description": "Automation of accessibility testing",
   "main": "index.js",
   "scripts": {

package/procs/score/asp09.js

@@ -255,7 +255,7 @@ exports.scorer = acts => {
           facts = test.result;
           if (facts && facts.content && facts.url && (facts.content.totals || facts.url.totals)) {
             rules.ibm = 'multiply violations by 4*, recommendations by 2* (*discounted); sum';
-            const scores = {
+            const ibmScores = {
               content: null,
               url: null
             };
@@ -263,7 +263,7 @@ exports.scorer = acts => {
               const totals = facts[type].totals;
               if (totals) {
                 const items = facts[type].items || [];
-                scores[type] = Math.round(items.reduce((total, item) => {
+                ibmScores[type] = Math.round(items.reduce((total, item) => {
                   const {ruleId, level} = item;
                   const rawScore = [4, 2][['violation', 'recommendation'].indexOf(level)] || 0;
                   const divisor = duplications.ibm[`${level.slice(0, 1)}:${ruleId}`] + 1 || 1;
@@ -271,8 +271,8 @@ exports.scorer = acts => {
                 }, 0));
               }
             });
-            if (scores.content !== null || scores.url !== null) {
-              scores.ibm = Math.max(scores.content || 0, scores.url || 0);
+            if (ibmScores.content !== null || ibmScores.url !== null) {
+              scores.ibm = Math.max(ibmScores.content || 0, ibmScores.url || 0);
               scores.total += scores.ibm;
             }
           }

package/samples/scripts/simple.json

@@ -0,0 +1,21 @@
+{
+  "what": "Test example.com with alfa",
+  "strict": true,
+  "commands": [
+    {
+      "type": "launch",
+      "which": "chromium",
+      "what": "Chromium browser"
+    },
+    {
+      "type": "url",
+      "which": "https://example.com/",
+      "what": "page with a few accessibility defects"
+    },
+    {
+      "type": "test",
+      "which": "alfa",
+      "what": "Siteimprove alfa package"
+    }
+  ]
+}

package/validation/executors/{appNoBatch.js → app.js}

@@ -1,7 +1,7 @@
 // app.js
 // Validator for Testaro application with a no-batch script.
 
-const options = {
+const report = {
   script: {
     what: 'Sample Testaro executor with 1 test',
     strict: true,
@@ -29,14 +29,14 @@ const options = {
     ]
   },
   log: [],
-  reports: []
+  acts: []
 };
 const {handleRequest} = require(`${__dirname}/../../index`);
-handleRequest(options)
+handleRequest(report)
 .then(
   () => {
-    const {log, reports} = options;
-    if (log.length === 3 && log[1].event === 'timeStamp' && /^[a-z0-9]+$/.test(log[1].value)) {
+    const {log, acts} = report;
+    if (log.length === 2 && log[1].event === 'endTime' && /^\d{4}-.+$/.test(log[1].value)) {
       console.log('Success: Log has been correctly populated');
     }
     else {
@@ -44,21 +44,19 @@ handleRequest(options)
       console.log(JSON.stringify(log, null, 2));
     }
     if (
-      reports.length === 1
-      && reports[0].acts
-      && reports[0].acts.length === 4
-      && reports[0].acts[2].result
-      && reports[0].acts[2].result.visibleElements
-      && typeof reports[0].acts[2].result.visibleElements === 'number'
-      && reports[0].acts[3].result
-      && typeof reports[0].acts[3].result === 'string'
-      && reports[0].acts[3].result.startsWith('ERROR')
+      acts.length === 4
+      && acts[2].result
+      && acts[2].result.visibleElements
+      && typeof acts[2].result.visibleElements === 'number'
+      && acts[3].result
+      && typeof acts[3].result === 'string'
+      && acts[3].result.startsWith('ERROR')
     ) {
-      console.log('Success: Reports have been correctly populated');
+      console.log('Success: Acts have been correctly populated');
     }
     else {
-      console.log('Failure: Reports empty or invalid');
-      console.log(JSON.stringify(reports, null, 2));
+      console.log('Failure: Acts empty or invalid');
+      console.log(JSON.stringify(acts, null, 2));
     }
   },
   rejection => {

package/validation/executors/test.js

@@ -0,0 +1,18 @@
+// test.js
+// Test executor.
+
+const fs = require('fs');
+const {handleRequest} = require('../../index');
+const scriptJSON = fs.readFileSync('samples/scripts/simple.json', 'utf8');
+const script = JSON.parse(scriptJSON);
+const report = {
+  id: '',
+  script,
+  log: [],
+  acts: []
+};
+(async () => {
+  await handleRequest(report);
+  console.log(`Report log:\n${JSON.stringify(report.log, null, 2)}\n`);
+  console.log(`Report acts:\n${JSON.stringify(report.acts, null, 2)}`);
+})();

package/validation/executors/tests.js

@@ -15,12 +15,12 @@ const validateTests = async () => {
     const scriptJSON = rawScriptJSON
     .replace(/__targets__/g, `file://${__dirname}/../tests/targets`);
     const script = JSON.parse(scriptJSON);
-    const options = {script};
-    options.log = [];
-    options.reports = [];
-    await handleRequest(options);
-    const {log, reports} = options;
-    if (log.length === 3 && log[1].event === 'timeStamp' && /^[a-z0-9]+$/.test(log[1].value)) {
+    const report = {script};
+    report.log = [];
+    report.acts = [];
+    await handleRequest(report);
+    const {log, acts} = report;
+    if (log.length === 2 && log[1].event === 'endTime' && /^\d{4}-.+$/.test(log[1].value)) {
       console.log('Success: Log has been correctly populated');
     }
     else {
@@ -28,19 +28,17 @@ const validateTests = async () => {
       console.log(JSON.stringify(log, null, 2));
     }
     if (
-      reports.length === 1
-      && reports[0].acts
-      && reports[0].acts.length === script.commands.length
-      && reports[0].acts.every(
+      acts.length === script.commands.length
+      && acts.every(
         act => act.type && act.type === 'test'
-          ? act.result && act.result.failureCount !== undefined 
+          ? act.result && act.result.failureCount !== undefined
           : true
       )
     ) {
       totals.attempts++;
       totals.successes++;
       console.log('Success: Reports have been correctly populated');
-      if (reports[0].acts.every(
+      if (acts.every(
         act => act.type === 'test' ? act.result.failureCount === 0 : true
       )) {
         totals.attempts++;
@@ -50,13 +48,13 @@ const validateTests = async () => {
       else {
         totals.attempts++;
         console.log('Failure: At least one test has at least one failure');
-        console.log(JSON.stringify(reports, null, 2));
+        console.log(JSON.stringify(acts, null, 2));
       }
     }
     else {
       totals.attempts++;
       console.log('Failure: Reports empty or invalid');
-      console.log(JSON.stringify(reports, null, 2));
+      console.log(JSON.stringify(acts, null, 2));
     }
   }
   console.log(`Grand totals: attempts ${totals.attempts}, successes ${totals.successes}`);

package/validation/executors/appBatch.js

@@ -1,87 +0,0 @@
-// app.js
-// Validator for Testaro application with a script and a batch.
-
-const options = {
-  script: {
-    what: 'Sample Testaro executor with 1 test',
-    strict: true,
-    commands: [
-      {
-        type: 'launch',
-        which: 'chromium',
-        what: 'Chromium browser'
-      },
-      {
-        type: 'url',
-        which: 'https://*',
-        what: 'URL to be replaced with URLs in the batch'
-      },
-      {
-        type: 'test',
-        which: 'bulk',
-        what: 'bulk'
-      },
-      {
-        type: 'test',
-        which: 'noSuchTest',
-        what: 'test that does not exist'
-      }
-    ]
-  },
-  batch: {
-    what: 'Two websites',
-    hosts: [
-      {
-        which: 'https://www.w3.org/',
-        what: 'W3C'
-      },
-      {
-        which: 'https://www.wikimedia.org/',
-        what: 'Wikimedia'
-      }
-    ]
-  },
-  log: [],
-  reports: []
-};
-const {handleRequest} = require(`${__dirname}/../../index`);
-const isValidReport = (reports, index) => {
-  const isValid = reports[index].acts
-  && reports[index].acts.length === 4
-  && reports[index].acts[2].result
-  && reports[index].acts[2].result.visibleElements
-  && typeof reports[index].acts[2].result.visibleElements === 'number'
-  && reports[index].acts[3].result
-  && typeof reports[index].acts[3].result === 'string'
-  && reports[index].acts[3].result.startsWith('ERROR');
-  return isValid;
-};
-handleRequest(options)
-.then(
-  () => {
-    const {log, reports} = options;
-    if (
-      log.length === 4
-      && log[1].event === 'timeStamp'
-      && /^[a-z0-9]+$/.test(log[1].value)
-      && log[2].event === 'batchSize'
-      && log[2].value === 2
-    ) {
-      console.log('Success: Log has been correctly populated');
-    }
-    else {
-      console.log('Failure: Log empty or invalid');
-      console.log(JSON.stringify(log, null, 2));
-    }
-    if (reports.length === 2 && isValidReport(reports, 0) && isValidReport(reports, 1)) {
-      console.log('Success: Reports have been correctly populated');
-    }
-    else {
-      console.log('Failure: Reports empty or invalid');
-      console.log(JSON.stringify(reports, null, 2));
-    }
-  },
-  rejection => {
-    console.log(`Failure: ${rejection}`);
-  }
-);