testaro 4.5.0 → 4.5.3

package/README.md

@@ -23,7 +23,7 @@ Testaro includes some of its own accessibility tests. In addition, it performs t
 - [alfa](https://alfa.siteimprove.com/) (Siteimprove alfa)
 - [Automated Accessibility Testing Tool](https://www.npmjs.com/package/aatt) (Paypal AATT, running HTML CodeSniffer)
 - [axe-playwright](https://www.npmjs.com/package/axe-playwright) (Deque Axe-core)
-- [Tenon](https://tenon.io/documentation/what-tenon-tests.php)
+- [Tenon](https://tenon.io/documentation/what-tenon-tests.php) (Level Access)
 - [WAVE API](https://wave.webaim.org/api/) (WebAIM WAVE)
 
 As of this version, the counts of tests in the packages referenced above were:
@@ -560,6 +560,8 @@ The `tests` executor makes use of the scripts in the `validation/tests/scripts`
 
 You can define additional Testaro commands and functionality. Contributions are welcome.
 
+Please report any issues, including feature requests, at the [repository](https://github.com/jrpool/testaro/issues).
+
 ## Accessibility principles
 
 The rationales motivating the Testaro-defined tests can be found in comments within the files of those tests, in the `tests` directory. Unavoidably, each test is opinionated. Testaro itself, however, can accommodate other tests representing different opinions. Testaro is intended to be neutral with respect to questions such as the criteria for accessibility, the severities of accessibility issues, whether accessibility is binary or graded, and the distinction between usability and accessibility.
@@ -594,53 +596,23 @@ The files in the `temp` directory are presumed ephemeral and are not tracked by
 
 ## Related packages
 
-[Testilo](https://www.npmjs.com/package/testilo) is an application that facilitates the use of Testaro.
+[Testilo](https://www.npmjs.com/package/testilo) is an application that:
+- produces scores and adds them to the JSON report files of Testaro
+- produces human-oriented HTML digests from scored reports
+- produces human-oriented HTML reports comparing the scores of hosts
 
 Testaro is derived from [Autotest](https://github.com/jrpool/autotest).
 
 Testaro omits some functionalities of Autotest, such as:
 - tests producing results intended to be human-inspected
-- scoring
+- scoring (performed now by Testilo)
 - file operations for score aggregation, report revision, and HTML reports
 - a web user interface
 
 ## Origin
 
-Work on the custom tests in this package began in 2017, and work on the multi-package federation that Testaro implements began in early 2018. These two aspects were combined into the [Autotest](https://github.com/jrpool/autotest) package in early 2021 and into this more single-purpose package, Testaro, in January 2022.
+Work on the custom tests in this package began in 2017, and work on the multi-package federation that Testaro implements began in early 2018. These two aspects were combined into the [Autotest](https://github.com/jrpool/autotest) package in early 2021 and into the more single-purpose packages, Testaro and Testilo, in January 2022.
 
 ## Etymology
 
 “Testaro” means “collection of tests” in Esperanto.
-
-## Future work
-
-### Improvements
-
-Further development is contemplated, is taking place, or is welcomed, on:
-- addition of Tenon to the set of packages
-- links with href="#"
-- links and buttons styled non-distinguishably
-- first focused element not first focusable element in DOM
-- never-visible skip links
-- buttons with no text content
-- modal dialogs
-- autocomplete attributes
-- inclusion of other test packages, such as:
-   - FAE (https://github.com/opena11y/evaluation-library)
-
-## Corrections
-
-Issues found or reported with the current version that need diagnosis and correction include:
-
-### hover
-
-There seem to be a couple of problems with the hover test:
-- The score for unhoverability is documented as 2 times the count of unhoverables, but is reported as 1 time that count.
-- The list of unhoverables in the report is empty.
-Observed after inquiry by Tobias Christian Jensen of Siteimprove on 2022-05-09.
-
-### axe
-
-Configuration to include best practices and experimental tests.
-
-Investigation of tags, including wcag2a, wcag2aa, wcag21a, wcag21aa, best-practice, wcag***, ACT, cat.*.

package/commands.js

@@ -146,7 +146,8 @@ exports.commands = {
     aatt: [
       'Perform an AATT test with HTML CodeSniffer',
       {
-        waitLong: [false, 'boolean', '', 'whether to wait 20 instead of 10 seconds for the result']
+        waitLong: [false, 'boolean', '', 'whether to wait 12 instead of 6 seconds for a result'],
+        tryLimit: [false, 'number', '', 'times to try the test before giving up; default 4']
       }
     ],
     axe: [

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "4.5.0",
+  "version": "4.5.3",
   "description": "Automation of accessibility testing",
   "main": "index.js",
   "scripts": {

package/run.js

@@ -291,7 +291,7 @@ const textOf = async (page, element) => {
     if (['A', 'BUTTON', 'INPUT', 'SELECT'].includes(tagName)) {
       // Return its visible labels, descriptions, and legend if the first input in a fieldset.
       totalText = await page.evaluate(element => {
-        const {tagName} = element;
+        const {tagName, ariaLabel} = element;
         let ownText = '';
         if (['A', 'BUTTON'].includes(tagName)) {
           ownText = element.textContent;
@@ -302,6 +302,9 @@ const textOf = async (page, element) => {
         // HTML link elements have no labels property.
         const labels = tagName !== 'A' ? Array.from(element.labels) : [];
         const labelTexts = labels.map(label => label.textContent);
+        if (ariaLabel) {
+          labelTexts.push(ariaLabel);
+        }
         const refIDs = new Set([
           element.getAttribute('aria-labelledby') || '',
           element.getAttribute('aria-describedby') || ''
@@ -616,7 +619,7 @@ const doActs = async (report, actIndex, page) => {
         if (truth[1]) {
           // If the performance of commands is to stop:
           if (act.jump === 0) {
-            // Set the act index to cause a stop.
+            // Stop.
             actIndex = -2;
           }
           // Otherwise, if there is a numerical jump:
@@ -897,7 +900,7 @@ const doActs = async (report, actIndex, page) => {
             // Otherwise, if the act is a move:
             else if (moves[act.type]) {
               const selector = typeof moves[act.type] === 'string' ? moves[act.type] : act.what;
-              // Wait 10 seconds until the element to perform the move on is identified.
+              // Try up to 5 times, every 2 seconds, to identify the element to perform the move on.
               let matchResult = {success: false};
               let tries = 0;
               while (tries++ < 5 && ! matchResult.success) {
@@ -1019,9 +1022,10 @@ const doActs = async (report, actIndex, page) => {
               }
               // Otherwise, i.e. if no match was found:
               else {
-                const report = 'ERROR: Specified element not found';
-                act.result = report;
-                console.log(report);
+                // Stop.
+                act.result = matchResult;
+                console.log('ERROR: Specified element not found');
+                actIndex = -2;
               }
             }
             // Otherwise, if the act is a keypress:

package/tests/aatt.js

@@ -7,31 +7,59 @@
 const {evaluate} = require('aatt');
 
 // FUNCTIONS
-exports.reporter = async (page, waitLong) => {
-  // Set the limit in seconds on the wait for the result.
-  const timeLimit = waitLong ? 20 : 10;
-  // Get the HTML of the document body.
-  const source = await page.content();
-  // Return the result of a test with the HTML CodeSniffer WCAG 2.1 AA ruleset as a string.
-  const report = evaluate({
-    source,
-    output: 'json',
-    engine: 'htmlcs',
-    level: 'WCAG2AA'
-  });
-  // Wait for it until the time limit expires.
-  let timeoutID;
-  const wait = new Promise(resolve => {
-    timeoutID = setTimeout(() => {
-      resolve('');
-    }, 1000 * timeLimit);
-  });
-  const resultIfFast = await Promise.race([report, wait]);
-  // If it arrived within the time limit:
-  if (resultIfFast) {
-    clearTimeout(timeoutID);
-    // Remove the non-JSON prefix and (if any) suffix from the string.
-    const reportJSON = resultIfFast.replace(/^.+?Object]\s+|\s+done\s*$/sg, '');
+// Recursively test a page with HTML CodeSniffer for WCAC 2.1 AAA.
+const retest = async (page, waitLong, triesLeft) => {
+  // If the limit on tries has not been exhausted:
+  if (triesLeft) {
+    // Set the limit in seconds on the wait for the result.
+    const timeLimit = waitLong ? 12 : 6;
+    // Get the HTML of the document body.
+    const source = await page.content();
+    // Return the result of a test with the HTML CodeSniffer WCAG 2.1 AA ruleset as a string.
+    const report = evaluate({
+      source,
+      output: 'json',
+      engine: 'htmlcs',
+      level: 'WCAG2AAA'
+    });
+    // Wait for it until the time limit expires.
+    let timeoutID;
+    const wait = new Promise(resolve => {
+      timeoutID = setTimeout(() => {
+        resolve('');
+      }, 1000 * timeLimit);
+    });
+    const result = await Promise.race([report, wait]);
+    // If it arrived within the time limit:
+    if (result) {
+      clearTimeout(timeoutID);
+      // Return the result as JSON.
+      const reportJSON = result.replace(/^.+?Object.?\s+|\s+done\s*$/sg, '');
+      return {
+        triesLeft,
+        reportJSON
+      };
+    }
+    else {
+      console.log(`ERROR: Test aatt timed out at ${timeLimit} seconds; tries left: ${triesLeft - 1}`);
+      return retest(page, waitLong, --triesLeft);
+    }
+  }
+  // Otherwise, i.e. if the limit on tries has been exhausted:
+  else {
+    // Return this.
+    return {
+      triesLeft,
+      reportJSON: ''
+    };
+  }
+};
+exports.reporter = async (page, waitLong, tryLimit = 4) => {
+  // Try the test up to the limit on tries.
+  const result = await retest(page, waitLong, tryLimit);
+  const {triesLeft, reportJSON} = result;
+  // If any try succeeded:
+  if (reportJSON) {
     try {
       // Convert the JSON string to an array.
       const issueArray = JSON.parse(reportJSON);
@@ -49,10 +77,18 @@ exports.reporter = async (page, waitLong) => {
           .join('+');
         }
       });
-      return {result: nonNotices};
+      return {
+        result: {
+          report: nonNotices,
+          triesLeft
+        }
+      };
     }
     catch (error) {
       console.log(`ERROR processing AATT report (${error.message})`);
+      console.log(
+        `JSON report starts with ${reportJSON.slice(0, 50)} and ends with ${reportJSON.slice(-50)}`
+      );
       return {
         result: {
           prevented: true,
@@ -61,14 +97,15 @@ exports.reporter = async (page, waitLong) => {
       };
     }
   }
-  // Otherwise, i.e. if the result did not arrive within the time limit:
+  // Otherwise, i.e. if the limit on tries was exhausted:
   else {
     // Report the failure.
-    console.log('ERROR: getting report took too long');
+    const error = 'ERROR: Getting AATT report took too long';
+    console.log(error);
     return {
       result: {
         prevented: true,
-        error: 'ERROR: getting AATT report took too long'
+        error
       }
     };
   }