testaro 27.0.0 → 28.0.0

package/CONTRIBUTING.md

@@ -10,7 +10,9 @@ Testaro can benefit from contributions of various types, such as:
 
 ## Adding tools
 
-To come.
+Tools that may merit consideration include:
+- Arc (by TPGi). Costs $0.05 per page tested, and requires each user to have an account with the tool maker and to allow the tool maker to charge a charge account for payment.
+- Aslint (by Essential Accessibility). Apparently requires using a bundle file that is in the repository but is also listed in the `.gitignore` file and is missing from the published package, so would need to be physically included in the Testaro code rather than referenced as a dependency.
 
 ## Improving execution speed
 

package/README.md

@@ -10,6 +10,8 @@ The purposes of Testaro are to:
 - provide programmatic access to accessibility tests defined by several tools
 - facilitate the integration of the reports of the tools into a unified report
 
+Testaro is described in [Testaro: Efficient Ensemble Testing for Web Accessibility](https://arxiv.org/abs/2309.10167).
+
 The need for multi-tool integration, and its costs, are discussed in [Accessibility Metatesting: Comparing Nine Testing Tools](https://arxiv.org/abs/2304.07591).
 
 Testaro launches and controls web browsers, performing operations, conducting tests, and recording results.
@@ -43,6 +45,7 @@ Testaro uses:
 Testaro performs tests of these tools:
 - [accessibility-checker](https://www.npmjs.com/package/accessibility-checker) (IBM)
 - [alfa](https://alfa.siteimprove.com/) (Siteimprove)
+- [aslint](https://www.npmjs.com/package/@essentialaccessibility/aslint) (eSSENTIAL Accessibility)
 - [axe-playwright](https://www.npmjs.com/package/axe-playwright) (Deque)
 - [HTML CodeSniffer](https://www.npmjs.com/package/html_codesniffer) (Squiz Labs)
 - [Nu Html Checker](https://github.com/validator/validator) (World Wide Web Consortium)
@@ -50,11 +53,11 @@ Testaro performs tests of these tools:
 - [Testaro](https://www.npmjs.com/package/testaro) (CVS Health)
 - [WAVE API](https://wave.webaim.org/api/) (WebAIM)
 
-Some of the tests of Testaro are designed to act as approximate alternatives to tests of vulnerable, restricted, or no longer available tools. One such tool is the [BBC Accessibility Standards Checker](https://github.com/bbc/bbc-a11y). In all such cases the Testaro rules are independently designed and implemented, without reference to the code of the tests that inspired them.
+Some of the tests of Testaro are designed to act as approximate alternatives to tests of vulnerable, restricted, or no longer available tools. In all such cases the Testaro rules are independently designed and implemented, without reference to the code of the tests that inspired them.
 
 ## Rules
 
-Each tool accessed with Testaro defines _rules_ and tests _targets_ for compliance with its rules. In total, the eight tools define about 650 rules. Some of the tools are under active development, and their rule counts change over time.
+Each tool accessed with Testaro defines _rules_ and tests _targets_ for compliance with its rules. In total, the nine tools define about 760 rules. Some of the tools are under active development, and their rule counts change over time.
 
 When you ask Testaro to run tests of a tool, you may specify a subset of the rules of that tool, and the report will give you the results of only the tests for those rules. These tools will perform only those tests:
 - `alfa`
@@ -68,6 +71,8 @@ These tools always perform a fixed set of tests, and Testaro disregards irreleva
 - `nuVal`
 - `wave`
 
+The `aslint` tool does not yet allow rule specification.
+
 ## Job data
 
 A report produced by Testaro discloses:
@@ -392,6 +397,8 @@ When you include a `rules` property, you limit the tests of the tool that are pe
 
 The `nuVal`, `qualWeb`, and `testaro` tools require specific formats for the `rules` property. Those formats are described below in the sections about those tools.
 
+The `aslint` tool does not yet allow rule specification.
+
 ###### Examples
 
 An example of a `test` act is:
@@ -468,6 +475,10 @@ A typical use for an `expect` property is checking the correctness of a Testaro
 
 When a `test` act has an `expect` property, the result for that act has an `expectations` property reporting whether the expectations were satisfied. The value of `expectations` is an array of objects, one object per expectation. Each object includes a `property` property identifying the expectation, and a `passed` property with `true` or `false` value reporting whether the expectation was satisfied. If applicable, it also has other properties identifying what was expected and what was actually reported.
 
+###### ASLint
+
+The `aslint` tool makes use of the [`aslint-testaro` fork](https://www.npmjs.com/package/aslint-testaro) of the [`aslint` repository](https://github.com/essentialaccessibility/aslint), which, unlike the published `aslint` package, contains the `aslint.bundle.js` file.
+
 ###### HTML CodeSniffer
 
 The `htmlcs` tool makes use of the `htmlcs/HTMLCS.js` file. That file was created, and can be recreated if necessary, as follows:
@@ -664,7 +675,7 @@ The validity criterion named in item 2 may be any of these:
 - `'isBrowserType'`: is `'chromium'`, `'firefox'`, or `'webkit'`
 - `'isFocusable'`: is `'a'`, `'button'`, `'input'`, `'select'`, or `'option'`
 - `'isState'`: is `'loaded'` or `'idle'`
-- `'isTest'`: is the name of a test
+- `'isTest'`: is the name of a tool
 - `'isWaitable'`: is `'url'`, `'title'`, or `'body'`
 - `'areStrings'`: is an array of strings
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "27.0.0",
+  "version": "28.0.0",
   "description": "Run 650 web accessibility tests from 8 tools and get a standardized report",
   "main": "index.js",
   "scripts": {
@@ -33,6 +33,7 @@
     "@siteimprove/alfa-rules": "*",
     "@siteimprove/alfa-scraper": "*",
     "accessibility-checker": "*",
+    "aslint-testaro": "*",
     "axe-playwright": "*",
     "dotenv": "*",
     "node-fetch": "<3.0.0",

package/procs/aslint.js

@@ -0,0 +1,83 @@
+/*
+  aslint
+  Injects and captures results from the ASLint bundle.
+*/
+const options = {
+  asyncRunner: true,
+  context: document.documentElement,
+  includeElementReference: true,
+  reportFormat: {JSON: true},
+  watchDomChanges: false
+};
+const bundleEl = document.getElementById('aslintBundle');
+window
+.aslint
+.config(options)
+.addListener('onValidatorStarted', function () {
+  console.log('@ Validator started');
+})
+.addListener('onValidatorComplete', function (error, report) {
+  console.log('@ Validator Complete');
+})
+.addFilter('onBeforeRuleReport', function (report) {
+  return report;
+})
+.setRule('turn-me-off', {isSelectedForScanning: false})
+.run()
+.then(function (result) {
+  const resultEl = document.createElement('pre');
+  resultEl.id = 'aslintResult';
+  if (result.rules) {
+    const ruleIDs = Object.keys(result.rules);
+    ruleIDs.forEach(ruleID => {
+      const {results} = result.rules[ruleID];
+      if (results && results.length) {
+        results.forEach(ruleResult => {
+          delete ruleResult.data;
+          delete ruleResult.element.reference;
+          delete ruleResult.message.expected;
+          delete ruleResult.skipReason;
+          ruleResult.element.html = ruleResult
+          .element
+          .html
+          .replace(/&lt;/g, '<')
+          .replace(/&gt;/g, '>')
+          .replace(/&quot;/g, '\"')
+          .replace(/&#x3D;/g, '=')
+          .replace(/&#x2F;/g, '/')
+          .replace(/&#39;/g, '\'')
+          .replace(/&amp;(?:amp;)*/g, '&')
+          .replace(/%3A/g, ':')
+          .replace(/%2F/g, '/');
+          ruleResult.message.actual.description = ruleResult
+          .message
+          .actual
+          .description
+          .replace(/&lt;/g, '<')
+          .replace(/&gt;/g, '>')
+          .replace(/&quot;/g, '\"')
+          .replace(/&#x3D;/g, '=')
+          .replace(/&#x2F;/g, '/')
+          .replace(/&amp;(?:amp;)*/g, '&')
+          .replace(/%2F/g, '/');
+        });
+      }
+    });
+    ruleIDs.forEach(ruleID => {
+      try {
+        JSON.stringify(result.rules[ruleID]);
+      }
+      catch(error) {
+        console.log(`ERROR: Rule ${ruleID} result not stringifiable so its results deleted`);
+        delete result.rules[ruleID].results;
+        result.rules[ruleID].success = false;
+        result.rules[ruleID].error = 'Result property not stringifiable so deleted';
+      }
+    });
+  }
+  resultEl.textContent = JSON.stringify(result, null, 2);
+  document.body.insertAdjacentElement('beforeend', resultEl);
+})
+.catch(error => {
+  console.error('[ASLint error]', error);
+});

package/procs/nav.js

@@ -37,6 +37,28 @@ let browser;
 
 // Returns a string with any final slash removed.
 const deSlash = string => string.endsWith('/') ? string.slice(0, -1) : string;
+// Gets the script nonce from a response.
+const getNonce = async response => {
+  let nonce = '';
+  // If the response includes a content security policy:
+  const headers = await response.allHeaders();
+  const cspWithQuotes = headers && headers['content-security-policy'];
+  if (cspWithQuotes) {
+    // If it requires scripts to have a nonce:
+    const csp = cspWithQuotes.replace(/'/g, '');
+    const directives = csp.split(/ *; */).map(directive => directive.split(/ +/));
+    const scriptDirective = directives.find(dir => dir[0] === 'script-src');
+    if (scriptDirective) {
+      const nonceSpec = scriptDirective.find(valPart => valPart.startsWith('nonce-'));
+      if (nonceSpec) {
+        // Return the nonce.
+        nonce = nonceSpec.replace(/^nonce-/, '');
+      }
+    }
+  }
+  // Return the nonce, if any.
+  return nonce;
+};
 // Visits a URL and returns the response of the server.
 const goTo = async (report, page, url, timeout, waitUntil) => {
   // If the URL is a file path:
@@ -68,8 +90,11 @@ const goTo = async (report, page, url, timeout, waitUntil) => {
       else {
         // Press the Escape key to dismiss any modal dialog.
         await page.keyboard.press('Escape');
-        // Return the response.
-        return response;
+        // Return the result of the navigation.
+        return {
+          success: true,
+          response
+        };
       }
     }
     // Otherwise, i.e. if the response status was abnormal:
@@ -78,6 +103,7 @@ const goTo = async (report, page, url, timeout, waitUntil) => {
       console.log(`ERROR: Visit to ${url} got status ${httpStatus}`);
       report.jobData.visitRejectionCount++;
       return {
+        success: false,
         error: 'badStatus'
       };
     }
@@ -85,6 +111,7 @@ const goTo = async (report, page, url, timeout, waitUntil) => {
   catch(error) {
     console.log(`ERROR visiting ${url} (${error.message.slice(0, 200)})`);
     return {
+      success: false,
       error: 'noVisit'
     };
   }
@@ -127,9 +154,12 @@ const launch = async (report, typeName, url, debug, waits, isLowMotion = false)
     // If the launch failed:
     .catch(async error => {
       healthy = false;
-      console.log(`ERROR launching browser (${errorStart(error)})`);
+      console.log(`ERROR launching browser (${error.message.slice(0, 200)})`);
       // Return this.
-      return false;
+      return {
+        success: false,
+        error: 'Browser launch failed'
+      };
     });
     // Open a context (i.e. browser tab), with reduced motion if specified.
     const options = {reduceMotion: isLowMotion ? 'reduce' : 'no-preference'};
@@ -187,36 +217,48 @@ const launch = async (report, typeName, url, debug, waits, isLowMotion = false)
       await currentPage.waitForLoadState('domcontentloaded', {timeout: 5000});
       // Navigate to the specified URL.
       const navResult = await goTo(report, currentPage, url, 15000, 'domcontentloaded');
-      // If the navigation failed:
-      if (navResult.error) {
-        // Return this.
-        return false;
-      }
-      // Otherwise, i.e. if it succeeded:
-      else if (! navResult.error) {
+      // If the navigation succeeded:
+      if (navResult.success) {
         // Update the name of the current browser type and store it in the page.
         currentPage.browserTypeName = typeName;
-        // Return the browser context and the page.
+        // Return the response, the browser context, and the page.
         return {
+          success: true,
+          response: navResult.response,
           browserContext,
           currentPage
         };
       }
+      // If the navigation failed:
+      if (navResult.error) {
+        // Return this.
+        return {
+          success: false,
+          error: 'Navigation failed'
+        };
+      }
     }
     // If it fails to become stable by the deadline:
     catch(error) {
       // Return this.
       console.log(`ERROR: Blank page load in new tab timed out (${error.message})`);
-      return false;
+      return {
+        success: false,
+        error: 'Blank page load in new tab timed out'
+      };
     }
   }
   // Otherwise, i.e. if it does not exist:
   else {
     // Return this.
     console.log(`ERROR: Browser of type ${typeName} could not be launched`);
-    return false;
+    return {
+      success: false,
+      error: `${typeName} browser launch failed`
+    };
   }
 };
 exports.browserClose = browserClose;
+exports.getNonce = getNonce;
 exports.goTo = goTo;
 exports.launch = launch;

package/procs/standardize.js

@@ -259,6 +259,42 @@ const convert = (toolName, result, standardResult) => {
       standardResult.instances.push(instance);
     });
   }
+  // aslint
+  else if (toolName === 'aslint' && result.summary && result.summary.byIssueType) {
+    standardResult.totals = [
+      result.summary.byIssueType.warning, 0, 0, result.summary.byIssueType.error
+    ];
+    Object.keys(result.rules).forEach(ruleID => {
+      const ruleResults = result.rules[ruleID].results;
+      if (ruleResults && ruleResults.length) {
+        ruleResults.forEach(ruleResult => {
+          const {issueType} = result.rules[ruleID];
+          const xpath = ruleResult.element && ruleResult.element.xpath || '';
+          const tagName = xpath && xpath.replace(/^.*\//, '').replace(/[^-\w].*$/, '').toUpperCase()
+          || '';
+          const excerpt = ruleResult.element && ruleResult.element.html || '';
+          const idDraft = excerpt && excerpt.replace(/^[^[>]+id="/, 'id=').replace(/".*$/, '');
+          const id = idDraft && idDraft.length > 3 && idDraft.startsWith('id=')
+            ? idDraft.slice(3)
+            : '';
+          const instance = {
+            ruleID,
+            what: ruleResult.message.actual.description,
+            ordinalSeverity: ['warning', 0, 0, 'error'].indexOf(issueType),
+            tagName,
+            id,
+            location: {
+              doc: 'dom',
+              type: 'xpath',
+              spec: xpath
+            },
+            excerpt
+          };
+          standardResult.instances.push(instance);
+        });
+      }
+    });
+  }
   // axe
   else if (
     toolName === 'axe'

package/run.js

@@ -10,7 +10,7 @@ require('dotenv').config();
 // Requirements for acts.
 const {actSpecs} = require('./actSpecs');
 // Navigation.
-const {browserClose, goTo, launch} = require('./procs/nav');
+const {browserClose, getNonce, goTo, launch} = require('./procs/nav');
 // Module to standardize report formats.
 const {standardize} = require('./procs/standardize');
 
@@ -36,6 +36,7 @@ const moves = {
 // Names and descriptions of tools.
 const tests = {
   alfa: 'alfa',
+  aslint: 'ASLint',
   axe: 'Axe',
   htmlcs: 'HTML CodeSniffer WCAG 2.1 AA ruleset',
   ibm: 'IBM Accessibility Checker',
@@ -515,17 +516,21 @@ const doActs = async (report, actIndex, page) => {
       // Otherwise, if the act is a launch:
       else if (act.type === 'launch') {
         // Launch the specified browser and navigate to the specified URL.
-        const browserData = await launch(
+        const launchResult = await launch(
           report, act.which, act.url, debug, waits, act.lowMotion ? 'reduce' : 'no-preference'
         );
         // If the launch and navigation succeeded:
-        if (browserData) {
+        if (launchResult && launchResult.success) {
           // Save the browser data.
-          browserContext = browserData.browserContext;
-          currentPage = browserData.currentPage;
+          const {response, currentPage} = launchResult;
           page = currentPage;
           // Add the actual URL to the act.
           act.actualURL = page.url();
+          // Add any nonce to the Act.
+          const scriptNonce = await getNonce(response);
+          if (scriptNonce) {
+            report.jobData.lastScriptNonce = scriptNonce;
+          }
         }
         // Otherwise, i.e. if the launch or navigation failed:
         else {
@@ -541,26 +546,32 @@ const doActs = async (report, actIndex, page) => {
           const resolved = act.which.replace('__dirname', __dirname);
           requestedURL = resolved;
           // Visit it and wait until the DOM is loaded.
-          const response = await goTo(report, page, requestedURL, 15000, 'domcontentloaded');
-          // If the visit failed:
-          if (response.error) {
-            // Report this and quit.
-            addError(act, 'failure', 'ERROR: Visits failed');
-            await abortActs();
-          }
-          // Otherwise, i.e. if the visit succeeded:
-          else {
-            // If a prohibited redirection occurred:
-            if (response.exception === 'badRedirection') {
-              // Report this and quit.
-              addError(act, 'badRedirection', 'ERROR: Navigation illicitly redirected');
-              await abortActs();
+          const navResult = await goTo(report, page, requestedURL, 15000, 'domcontentloaded');
+          // If the visit succeeded:
+          if (navResult.success) {
+            // Add the script nonce, if any, to the act.
+            const {response} = navResult;
+            const scriptNonce = getNonce(response);
+            if (scriptNonce) {
+              report.jobData.lastScriptNonce = scriptNonce;
             }
             // Add the resulting URL to the act.
             if (! act.result) {
               act.result = {};
             }
             act.result.url = page.url();
+            // If a prohibited redirection occurred:
+            if (response.exception === 'badRedirection') {
+              // Report this and quit.
+              addError(act, 'badRedirection', 'ERROR: Navigation illicitly redirected');
+              await abortActs();
+            }
+          }
+          // If the visit failed:
+          if (response.error) {
+            // Report this and quit.
+            addError(act, 'failure', 'ERROR: Visits failed');
+            await abortActs();
           }
         }
         // Otherwise, if the act is a wait for text:
@@ -684,7 +695,10 @@ const doActs = async (report, actIndex, page) => {
             // Add a description of the test to the act.
             act.what = tests[act.which];
             // Initialize the options argument.
-            const options = {};
+            const options = {
+              act,
+              scriptNonce: report.jobData.lastScriptNonce || ''
+            };
             // Add any specified arguments to it.
             Object.keys(act).forEach(key => {
               if (! ['type', 'which'].includes(key)) {
@@ -700,8 +714,7 @@ const doActs = async (report, actIndex, page) => {
             };
             // Perform the specified tests of the tool and get a report.
             try {
-              const args = [page, options];
-              toolReport = await require(`./tests/${act.which}`).reporter(... args);
+              toolReport = await require(`./tests/${act.which}`).reporter(page, options);
               toolReport.result.success = true;
             }
             // If the testing failed:

package/tests/aslint.js

@@ -0,0 +1,83 @@
+/*
+  aslint
+  This test implements the ASLint ruleset for accessibility.
+*/
+
+// IMPORTS
+
+// Module to handle files.
+const fs = require('fs/promises');
+
+// FUNCTIONS
+
+// Conducts and reports an ASLint test.
+exports.reporter = async (page, options) => {
+  // Initialize the report.
+  let data = {};
+  // Get the ASLint runner and bundle scripts.
+  const aslintRunner = await fs.readFile(`${__dirname}/../procs/aslint.js`, 'utf8');
+  const aslintBundle = await fs.readFile(
+    `${__dirname}/../node_modules/aslint-testaro/aslint.bundle.js`, 'utf8'
+  );
+  // Get the nonce, if any.
+  const {scriptNonce} = options;
+  // Inject the ASLint bundle and runner into the page.
+  await page.evaluate(args => {
+    const {scriptNonce, aslintBundle, aslintRunner} = args;
+    // Bundle.
+    const bundleEl = document.createElement('script');
+    bundleEl.id = 'aslintBundle';
+    if (scriptNonce) {
+      bundleEl.nonce = scriptNonce;
+      console.log(`Added nonce ${scriptNonce} to bundle`);
+    }
+    bundleEl.textContent = aslintBundle;
+    document.head.insertAdjacentElement('beforeend', bundleEl);
+    // Runner.
+    const runnerEl = document.createElement('script');
+    if (scriptNonce) {
+      runnerEl.nonce = scriptNonce;
+      console.log(`Added nonce ${scriptNonce} to runner`);
+    }
+    runnerEl.textContent = aslintRunner;
+    document.body.insertAdjacentElement('beforeend', runnerEl);
+  }, {scriptNonce, aslintBundle, aslintRunner})
+  .catch(error => {
+    console.log(`ERROR: ASLint injection failed (${error.message.slice(0, 400)})`);
+    data.prevented = true;
+    data.error = 'ERROR: ASLint injection failed';
+  });
+  // If the injection succeeded:
+  if (! data.prevented) {
+    // Wait for the test results.
+    const reportLoc = page.locator('#aslintResult');
+    await reportLoc.waitFor({
+      state: 'attached',
+      timeout: 10000
+    });
+    // Get them.
+    const report = await reportLoc.textContent();
+    // Populate the tool report.
+    data = JSON.parse(report);
+    // Delete irrelevant properties from the tool report details.
+    if (data.rules) {
+      Object.keys(data.rules).forEach(ruleID => {
+        if (['passed', 'skipped'].includes(data.rules[ruleID].status.type)) {
+          delete data.rules[ruleID];
+        }
+      });
+    }
+  }
+  // Return the result.
+  try {
+    JSON.stringify(data);
+  }
+  catch(error) {
+    console.log(`ERROR: ASLint result cannot be made JSON (${error.message.slice(0, 200)})`);
+    data = {
+      prevented: true,
+      error: `ERROR: ASLint result cannot be made JSON (${error.message.slice(0, 200)})`
+    };
+  }
+  return {result: data};
+};

package/tests/axe.js

@@ -18,9 +18,13 @@
   category (“tag”), such as 'wcag21aaa'. Scoring can consider test categories by getting the value
   of the 'tags' property of each rule.
 */
+
 // IMPORTS
+
 const {injectAxe, getAxeResults} = require('axe-playwright');
+
 // FUNCTIONS
+
 // Conducts and reports an Axe test.
 exports.reporter = async (page, options) => {
   const {detailLevel, rules} = options;

package/tests/qualWeb.js

@@ -2,14 +2,20 @@
   qualWeb
   This test implements the QualWeb ruleset for accessibility.
 */
+
 // IMPORTS
+
 const {QualWeb} = require('@qualweb/core');
+
 // CONSTANTS
+
 const qualWeb = new QualWeb({});
 const clusterOptions = {
   timeout: 25 * 1000
 };
+
 // FUNCTIONS
+
 // Conducts and reports a QualWeb test.
 exports.reporter = async (page, options) => {
   const {withNewContent, rules} = options;