testaro 4.5.1 → 4.6.0

package/README.md

@@ -23,7 +23,7 @@ Testaro includes some of its own accessibility tests. In addition, it performs t
 - [alfa](https://alfa.siteimprove.com/) (Siteimprove alfa)
 - [Automated Accessibility Testing Tool](https://www.npmjs.com/package/aatt) (Paypal AATT, running HTML CodeSniffer)
 - [axe-playwright](https://www.npmjs.com/package/axe-playwright) (Deque Axe-core)
-- [Tenon](https://tenon.io/documentation/what-tenon-tests.php)
+- [Tenon](https://tenon.io/documentation/what-tenon-tests.php) (Level Access)
 - [WAVE API](https://wave.webaim.org/api/) (WebAIM WAVE)
 
 As of this version, the counts of tests in the packages referenced above were:
@@ -560,6 +560,8 @@ The `tests` executor makes use of the scripts in the `validation/tests/scripts`
 
 You can define additional Testaro commands and functionality. Contributions are welcome.
 
+Please report any issues, including feature requests, at the [repository](https://github.com/jrpool/testaro/issues).
+
 ## Accessibility principles
 
 The rationales motivating the Testaro-defined tests can be found in comments within the files of those tests, in the `tests` directory. Unavoidably, each test is opinionated. Testaro itself, however, can accommodate other tests representing different opinions. Testaro is intended to be neutral with respect to questions such as the criteria for accessibility, the severities of accessibility issues, whether accessibility is binary or graded, and the distinction between usability and accessibility.
@@ -594,53 +596,23 @@ The files in the `temp` directory are presumed ephemeral and are not tracked by
 
 ## Related packages
 
-[Testilo](https://www.npmjs.com/package/testilo) is an application that facilitates the use of Testaro.
+[Testilo](https://www.npmjs.com/package/testilo) is an application that:
+- produces scores and adds them to the JSON report files of Testaro
+- produces human-oriented HTML digests from scored reports
+- produces human-oriented HTML reports comparing the scores of hosts
 
 Testaro is derived from [Autotest](https://github.com/jrpool/autotest).
 
 Testaro omits some functionalities of Autotest, such as:
 - tests producing results intended to be human-inspected
-- scoring
+- scoring (performed now by Testilo)
 - file operations for score aggregation, report revision, and HTML reports
 - a web user interface
 
 ## Origin
 
-Work on the custom tests in this package began in 2017, and work on the multi-package federation that Testaro implements began in early 2018. These two aspects were combined into the [Autotest](https://github.com/jrpool/autotest) package in early 2021 and into this more single-purpose package, Testaro, in January 2022.
+Work on the custom tests in this package began in 2017, and work on the multi-package federation that Testaro implements began in early 2018. These two aspects were combined into the [Autotest](https://github.com/jrpool/autotest) package in early 2021 and into the more single-purpose packages, Testaro and Testilo, in January 2022.
 
 ## Etymology
 
 “Testaro” means “collection of tests” in Esperanto.
-
-## Future work
-
-### Improvements
-
-Further development is contemplated, is taking place, or is welcomed, on:
-- addition of Tenon to the set of packages
-- links with href="#"
-- links and buttons styled non-distinguishably
-- first focused element not first focusable element in DOM
-- never-visible skip links
-- buttons with no text content
-- modal dialogs
-- autocomplete attributes
-- inclusion of other test packages, such as:
-   - FAE (https://github.com/opena11y/evaluation-library)
-
-## Corrections
-
-Issues found or reported with the current version that need diagnosis and correction include:
-
-### hover
-
-There seem to be a couple of problems with the hover test:
-- The score for unhoverability is documented as 2 times the count of unhoverables, but is reported as 1 time that count.
-- The list of unhoverables in the report is empty.
-Observed after inquiry by Tobias Christian Jensen of Siteimprove on 2022-05-09.
-
-### axe
-
-Configuration to include best practices and experimental tests.
-
-Investigation of tags, including wcag2a, wcag2aa, wcag21a, wcag21aa, best-practice, wcag***, ACT, cat.*.

package/commands.js

@@ -146,7 +146,8 @@ exports.commands = {
     aatt: [
       'Perform an AATT test with HTML CodeSniffer',
       {
-        waitLong: [false, 'boolean', '', 'whether to wait 20 instead of 10 seconds for the result']
+        waitLong: [false, 'boolean', '', 'whether to wait 12 instead of 6 seconds for a result'],
+        tryLimit: [false, 'number', '', 'times to try the test before giving up; default 4']
       }
     ],
     axe: [
@@ -178,6 +179,7 @@ exports.commands = {
     hover: [
       'Perform a hover test',
       {
+        sampleSize: [false, 'number', '', 'number of triggers to sample, if fewer than all'],
         withItems: [true, 'boolean']
       }
     ],

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "4.5.1",
+  "version": "4.6.0",
   "description": "Automation of accessibility testing",
   "main": "index.js",
   "scripts": {

package/run.js

@@ -900,7 +900,7 @@ const doActs = async (report, actIndex, page) => {
             // Otherwise, if the act is a move:
             else if (moves[act.type]) {
               const selector = typeof moves[act.type] === 'string' ? moves[act.type] : act.what;
-              // Wait 10 seconds until the element to perform the move on is identified.
+              // Try up to 5 times, every 2 seconds, to identify the element to perform the move on.
               let matchResult = {success: false};
               let tries = 0;
               while (tries++ < 5 && ! matchResult.success) {

package/tests/aatt.js

@@ -7,31 +7,59 @@
 const {evaluate} = require('aatt');
 
 // FUNCTIONS
-exports.reporter = async (page, waitLong) => {
-  // Set the limit in seconds on the wait for the result.
-  const timeLimit = waitLong ? 20 : 10;
-  // Get the HTML of the document body.
-  const source = await page.content();
-  // Return the result of a test with the HTML CodeSniffer WCAG 2.1 AA ruleset as a string.
-  const report = evaluate({
-    source,
-    output: 'json',
-    engine: 'htmlcs',
-    level: 'WCAG2AA'
-  });
-  // Wait for it until the time limit expires.
-  let timeoutID;
-  const wait = new Promise(resolve => {
-    timeoutID = setTimeout(() => {
-      resolve('');
-    }, 1000 * timeLimit);
-  });
-  const resultIfFast = await Promise.race([report, wait]);
-  // If it arrived within the time limit:
-  if (resultIfFast) {
-    clearTimeout(timeoutID);
-    // Remove the non-JSON prefix and (if any) suffix from the string.
-    const reportJSON = resultIfFast.replace(/^.+?Object]\s+|\s+done\s*$/sg, '');
+// Recursively test a page with HTML CodeSniffer for WCAC 2.1 AAA.
+const retest = async (page, waitLong, triesLeft) => {
+  // If the limit on tries has not been exhausted:
+  if (triesLeft) {
+    // Set the limit in seconds on the wait for the result.
+    const timeLimit = waitLong ? 12 : 6;
+    // Get the HTML of the document body.
+    const source = await page.content();
+    // Return the result of a test with the HTML CodeSniffer WCAG 2.1 AA ruleset as a string.
+    const report = evaluate({
+      source,
+      output: 'json',
+      engine: 'htmlcs',
+      level: 'WCAG2AAA'
+    });
+    // Wait for it until the time limit expires.
+    let timeoutID;
+    const wait = new Promise(resolve => {
+      timeoutID = setTimeout(() => {
+        resolve('');
+      }, 1000 * timeLimit);
+    });
+    const result = await Promise.race([report, wait]);
+    // If it arrived within the time limit:
+    if (result) {
+      clearTimeout(timeoutID);
+      // Return the result as JSON.
+      const reportJSON = result.replace(/^.+?Object.?\s+|\s+done\s*$/sg, '');
+      return {
+        triesLeft,
+        reportJSON
+      };
+    }
+    else {
+      console.log(`ERROR: Test aatt timed out at ${timeLimit} seconds; tries left: ${triesLeft - 1}`);
+      return retest(page, waitLong, --triesLeft);
+    }
+  }
+  // Otherwise, i.e. if the limit on tries has been exhausted:
+  else {
+    // Return this.
+    return {
+      triesLeft,
+      reportJSON: ''
+    };
+  }
+};
+exports.reporter = async (page, waitLong, tryLimit = 4) => {
+  // Try the test up to the limit on tries.
+  const result = await retest(page, waitLong, tryLimit);
+  const {triesLeft, reportJSON} = result;
+  // If any try succeeded:
+  if (reportJSON) {
     try {
       // Convert the JSON string to an array.
       const issueArray = JSON.parse(reportJSON);
@@ -49,10 +77,18 @@ exports.reporter = async (page, waitLong) => {
           .join('+');
         }
       });
-      return {result: nonNotices};
+      return {
+        result: {
+          report: nonNotices,
+          triesLeft
+        }
+      };
     }
     catch (error) {
       console.log(`ERROR processing AATT report (${error.message})`);
+      console.log(
+        `JSON report starts with ${reportJSON.slice(0, 50)} and ends with ${reportJSON.slice(-50)}`
+      );
       return {
         result: {
           prevented: true,
@@ -61,14 +97,15 @@ exports.reporter = async (page, waitLong) => {
       };
     }
   }
-  // Otherwise, i.e. if the result did not arrive within the time limit:
+  // Otherwise, i.e. if the limit on tries was exhausted:
   else {
     // Report the failure.
-    console.log('ERROR: getting report took too long');
+    const error = 'ERROR: Getting AATT report took too long';
+    console.log(error);
     return {
       result: {
         prevented: true,
-        error: 'ERROR: getting AATT report took too long'
+        error
       }
     };
   }

package/tests/hover.js

@@ -9,9 +9,9 @@
   has the tag name 'A' or 'BUTTON' or otherwise the descendants of the element. The only
   elements counted as being made visible by hovering are those with tag names 'A', 'BUTTON',
   'INPUT', and 'SPAN', and those with 'role="menuitem"' attributes. The test waits 700 ms after
-  each hover in case of delayed effects. Despite this delay, the test makes the execution time
-  practical by randomly sampling targets instead of hovering over all of them. Therefore, the
-  results may vary from one execution to another.
+  each hover in case of delayed effects. Despite this delay, the test can make the execution time
+  practical by randomly sampling targets instead of hovering over all of them. When sampling is
+  performed, the results may vary from one execution to another.
 */
 
 // CONSTANTS
@@ -53,14 +53,14 @@ const getSample = (population, sampleSize) => {
     return [];
   }
 };
+// Returns the text of an element.
+const textOf = async (element, limit) => {
+  let text = await element.textContent();
+  text = text.trim() || await element.innerHTML();
+  return text.trim().replace(/\s*/sg, '').slice(0, limit);
+};
 // Recursively finds and reports triggers and targets.
 const find = async (withItems, page, triggers) => {
-  if (withItems) {
-    data.items = {
-      triggers: [],
-      unhoverables: []
-    };
-  }
   // If any potential disclosure triggers remain:
   if (triggers.length) {
     // Identify the first of them.
@@ -181,12 +181,6 @@ const find = async (withItems, page, triggers) => {
       }
       catch (error) {
         console.log('ERROR hovering');
-        // Returns the text of an element.
-        const textOf = async (element, limit) => {
-          let text = await element.textContent();
-          text = text.trim() || await element.innerHTML();
-          return text.trim().replace(/\s*/sg, '').slice(0, limit);
-        };
         data.totals.unhoverables++;
         if (withItems) {
           data.items.unhoverables.push({
@@ -201,7 +195,16 @@ const find = async (withItems, page, triggers) => {
     await find(withItems, page, triggers.slice(1));
   }
 };
-exports.reporter = async (page, withItems) => {
+// Performs hover test and reports results.
+exports.reporter = async (page, sampleSize = Infinity, withItems) => {
+  // If details are to be reported:
+  if (withItems) {
+    // Add properties for details to the initialized result.
+    data.items = {
+      triggers: [],
+      unhoverables: []
+    };
+  }
   // Identify the triggers.
   const selectors = [
     'body a:visible',
@@ -217,8 +220,7 @@ exports.reporter = async (page, withItems) => {
   });
   // If they number more than the sample size limit, sample them.
   const triggerCount = triggers.length;
-  const sampleSize = 15;
-  const triggerSample = triggerCount > sampleSize ? getSample(triggers, 15) : triggers;
+  const triggerSample = triggerCount > sampleSize ? getSample(triggers, sampleSize) : triggers;
   // Find and document the hover-triggered disclosures.
   await find(withItems, page, triggerSample);
   // If the triggers were sampled: