testaro 64.11.0 → 65.0.0

package/README.md

@@ -725,9 +725,10 @@ Here is an example of an instance in a standard result:
     type: 'xpath',
     spec: '/html[1]/body[1]/section[3]/div[2]/div[1]/ul[1]/li[1]/button[1]'
   },
-  excerpt: '<button type="link"></button>',
+  excerpt: '<button type="link">Create an account</button>',
   boxID: '12:340:46:50',
-  pathID: '/html/body/section[3]/div[2]/div/ul/li[1]/button[1]'
+  pathID: '/html/body/section[3]/div[2]/div/ul/li[1]/button[1]',
+  text: 'Create an account'
 }
 ```
 
@@ -741,14 +742,15 @@ The element has no `id` attribute to distinguish it from other `button` elements
 - `box` (coordinates, width, and height of the element box): Editoria11y, Testaro
 - none: HTML CodeSniffer
 
-The tool also reproduces an excerpt of the element code.
+The tool or Testaro also reproduces an excerpt of the element code.
 
 ##### Element identification
 
-While the above properties can help you find the offending element, Testaro makes this easier by adding, where practical, two standard element identifiers to each standard instance:
+While the above properties can help you find the offending element, Testaro makes this easier by adding, where practical, three standard element identifiers to each standard instance:
 
 - `boxID`: a compact representation of the x, y, width, and height of the element bounding box, if the element can be identified and is visible.
 - `pathID`: the XPath of the element, if the element can be identified.
+- `text`: the text content of the element, if the element can be identified.
 
 These standard identifiers can help you determine whether violations reported by different tools belong to the same element or different elements. The `boxID` property can also support the making of images of the violating elements.
 
@@ -757,7 +759,7 @@ Some tools limit the efficacy of the current algorithm for standard identifiers:
 - HTML CodeSniffer does not report element locations, and the reported code excerpts exclude all text content.
 - Nu Html Checker reports line and column boundaries of element start tags and truncates element text content in reported code excerpts.
 
-Testaro aims to overcome these limitations by inserting uniquely identifying attributes into all elements of the pages being tested by these tools. Those attribute values appear in the excerpts produced by the tools and permit Testaro to identify the elements in the tested page. Except for elements excluded from the DOM, such as descendants of `noscript` elements, this mechanism allows Testaro to provide a `pathID` property in almost all standard instances. The `boxID` property is less universal, since some elements, such as `script` elements and hidden elements, have no bounding boxes.
+Testaro aims to overcome these limitations by inserting uniquely identifying attributes into all elements of the pages being tested by these tools. Those attribute values permit Testaro to identify the elements in the tested page. Except for elements excluded from the DOM, such as descendants of `noscript` elements, this mechanism allows Testaro to provide a `pathID` property in almost all standard instances. The `boxID` property is less universal, since some elements, such as `script` elements and hidden elements, have no bounding boxes.
 
 Testing can change the pages being tested, and such changes can cause a particular element to change its physical or logical location. In such cases, an element may appear multiple times in a tool report with different `boxID` or `pathID` values, even though it is, for practical purposes, the same element.
 
@@ -966,7 +968,9 @@ The rationales motivating the Testaro-defined tests can be found in comments wit
 
 On some occasions a test throws an error that cannot be handled with a `try`-`catch` structure. It has been observed, for example, that the `ibm` test does this when the page content, rather than the page URL, is given to `getCompliance()` and the target is `https://globalsolutions.org`, `https://monsido.com`, or `https://www.ambetterhealth.com/`.
 
-Some tools take apparently infinite time to perform their tests on some pages. To handle such stalling, Testaro subjects all tools to time limits. The limitation is implemented with forked child processes. Specifically, the `procs/doTestAct.js` module is run as a forked process with a `timeout` option for each of the 11 tools.
+Some tools take apparently infinite time to perform their tests on some pages. One website whose pages prevent 5 of the tools from ever completing their tests is the site of BrowserStack.
+
+To handle such fatal errors andstalling, Testaro runs the tests of each tool in a separate forked child process that executes the `doTestAct.js` module. The parent process subjects each tool to a time limit and kills the child if the time limit expires.
 
 ### Activation
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "64.11.0",
+  "version": "65.0.0",
   "description": "Run 1000 web accessibility tests from 11 tools and get a standardized report",
   "main": "index.js",
   "scripts": {

package/procs/doTestAct.js

@@ -22,12 +22,21 @@ const {browserClose, launch} = require(`${__dirname}/../run`);
 // Module to set operating-system constants.
 const os = require('os');
 
-// CONSTANTS
-
-const tmpDir = os.tmpdir();
-
 // FUNCTIONS
 
+// Sends a message to the parent process.
+const sendMessage = message => {
+  try {
+    if (typeof process.send === 'function') {
+      process.send(message);
+    }
+  }
+  catch(error) {
+    console.log(
+      `ERROR: process.send threw ${error.message} trying to send message ${message} to parent`
+    );
+  }
+};
 // Performs the tests of an act.
 const doTestAct = async (reportPath, actIndex) => {
   // Get the report from the temporary directory.
@@ -58,7 +67,11 @@ const doTestAct = async (reportPath, actIndex) => {
       const reportJSON = JSON.stringify(report);
       await fs.writeFile(reportPath, reportJSON);
       // Report this.
-      process.send('ERROR: Job aborted');
+      sendMessage({
+        status: 'error',
+        error: 'Page launch aborted'
+      });
+      process.exit(1);
     }
     // Otherwise, i.e. if the launch did not abort the job:
     else {
@@ -85,7 +98,10 @@ const doTestAct = async (reportPath, actIndex) => {
       // Save the revised report.
       await fs.writeFile(reportPath, reportJSON);
       // Send a completion message.
-      process.send('Act completed');
+      sendMessage({
+        status: 'ok',
+      });
+      process.exit(0);
     }
     // If the tool invocation failed:
     catch(error) {
@@ -97,7 +113,11 @@ const doTestAct = async (reportPath, actIndex) => {
       // Report the failure.
       const message = error.message.slice(0, 400);
       console.log(`ERROR: Test act ${act.which} failed (${message})`);
-      process.send('ERROR performing the act');
+      sendMessage({
+        status: 'error',
+        error: 'ERROR performing the act'
+      });
+      process.exit(1);
     };
   }
   // Otherwise, i.e. if the page does not exist:
@@ -116,9 +136,32 @@ const doTestAct = async (reportPath, actIndex) => {
     // Report this.
     const message = 'ERROR: No page';
     console.log(message);
-    process.send(message);
+    sendMessage({
+      status: 'error',
+      error: message
+    });
+    process.exit(1);
   }
 };
 
+process.on('uncaughtException', error => {
+  console.log(`ERROR: uncaughtException (${error.message})`);
+  sendMessage({
+    status: 'error',
+    error: 'uncaughtException'
+  });
+  process.exit(1);
+});
+
+process.on('unhandledRejection', error => {
+  const message = error && error.message ? error.message : String(error);
+  console.log(`ERROR: unhandledRejection (${message})`);
+  sendMessage({
+    status: 'error',
+    error: 'unhandledRejection'
+  });
+  process.exit(1);
+});
+
 const args = process.argv;
 doTestAct(args[2], Number.parseInt(args[3]));

package/run.js

@@ -791,7 +791,7 @@ const launchSpecs = (act, report) => [
 ];
 // Performs the acts in a report and adds the results to the report.
 const doActs = async (report, opts = {}) => {
-  const {acts} = report;
+  let {acts} = report;
   // Get the granular observation options, if any.
   const {onProgress = null, signal = null} = opts;
   // Get the standardization specification.
@@ -885,9 +885,7 @@ const doActs = async (report, opts = {}) => {
         // If this failed:
         if (! page) {
           // Add this to the act.
-          act.data ??= {};
-          act.data.prevented = true;
-          act.data.error = page.error || '';
+          addError(false, false, report, actIndex, page.error || '');
         }
       }
       // Otherwise, if the act is a test act:
@@ -898,72 +896,154 @@ const doActs = async (report, opts = {}) => {
         const startTime = Date.now();
         // Add it to the act.
         act.startTime = startTime;
-        // Save the report.
         let reportJSON = JSON.stringify(report);
+        // Save a copy of the report.
         await fs.writeFile(reportPath, reportJSON);
-        // Create a process to perform the act and add the result to the saved report.
+        let timedOut = false;
+        const limitMs = timeoutMultiplier * 1000 * (timeLimits[act.which] || 15);
+        // Create a child process to perform the act and add the result to the saved report.
         const actResult = await new Promise(resolve => {
           let closed = false;
-          const child = fork(
-            `${__dirname}/procs/doTestAct`, [reportPath, actIndex], {timeout: timeoutMultiplier * 1000 * (timeLimits[act.which] || 15)}
-          );
+          const child = fork(`${__dirname}/procs/doTestAct`, [reportPath, actIndex]);
+          let killTimer = null;
+          // Start a timeout timer for the child process.
+          const timeoutTimer = setTimeout(() => {
+            if (! timedOut) {
+              timedOut = true;
+              console.log(`ERROR: Timed out at ${Math.round(limitMs / 1000)} seconds`);
+              child.kill('SIGTERM');
+              killTimer = setTimeout(() => {
+                if (! closed) {
+                  console.log('ERROR: Failed to exit on SIGTERM from parent')
+                }
+                child.kill('SIGKILL');
+              }, 2000);
+            }
+          }, limitMs);
+          // Clears any current timers.
+          const clearTimers = () => {
+            [timeoutTimer, killTimer].forEach(timer => {
+              if (timer) {
+                clearTimeout(timer);
+              }
+            });
+          };
+          // If the child process sends a message (normally Act completed):
           child.on('message', message => {
             if (! closed) {
               closed = true;
-              resolve(message);
+              clearTimers();
+              // Return the message.
+              resolve({
+                kind: 'message',
+                message
+              });
+            }
+          });
+          // If the child process sends an error:
+          child.on('error', error => {
+            if (! closed) {
+              closed = true;
+              clearTimers();
+              // Return the error message.
+              resolve({
+                kind: 'error',
+                error: error.message
+              });
             }
           });
-          child.on('close', code => {
+          // If the child process closes:
+          child.on('close', (code, signal) => {
             if (! closed) {
               closed = true;
-              resolve(`Page closed with code ${code}`);
+              clearTimers();
+              // Return the exit code, signal, and timeout status.
+              resolve({
+                kind: 'close',
+                code,
+                signal,
+                timedOut
+              });
             }
           });
         });
-        // Get the revised report.
-        reportJSON = await fs.readFile(reportPath, 'utf8');
-        report = JSON.parse(reportJSON);
-        // Get the revised act.
-        act = report.acts[actIndex];
-        // If the result is an error code:
-        if (typeof actResult === 'number') {
-          // Add the error data to the act.
-          act.data ??= {};
-          act.data.prevented = true;
-          act.data.error = actResult;
+        // If the child process sent a message:
+        if (actResult.kind === 'message') {
+          // Get the revised report file.
+          reportJSON = await fs.readFile(reportPath, 'utf8');
+          try {
+            // Convert it from JSON to an object and replace the report with the object.
+            report = JSON.parse(reportJSON);
+            // Redefine the acts as those in the revised report.
+            acts = report.acts;
+          }
+          // If the conversion fails, leaving the report and its acts unchanged:
+          catch (error) {
+            // Report this.
+            console.log(
+              `ERROR: Tool sent message ${actResult.message}. Report is no longer JSON (${error.message}) but is instead a(n) ${typeof reportJSON} of length ${reportJSON.length}:\n${reportJSON}`
+            );
+            // Add the error data to the act.
+            addError(
+              false,
+              false,
+              report,
+              actIndex,
+              `Non-JSON report file after message ${actResult.message}`
+            );
+          }
         }
-        // Otherwise, i.e. if it is not an error code:
+        // Otherwise, i.e. if the child process closed abnormally:
         else {
-          // Add the elapsed time of the tool to the report.
-          const time = Math.round((Date.now() - startTime) / 1000);
-          const {toolTimes} = report.jobData;
-          toolTimes[act.which] ??= 0;
-          toolTimes[act.which] += time;
-          // If the act was not prevented:
-          if (act.data && ! act.data.prevented) {
-            // If the act has expectations:
-            const expectations = act.expect;
-            if (expectations) {
-              // Initialize whether they were fulfilled.
-              act.expectations = [];
-              let failureCount = 0;
-              // For each expectation:
-              expectations.forEach(spec => {
-                // Add its result to the act.
-                const truth = isTrue(act, spec);
-                act.expectations.push({
-                  property: spec[0],
-                  relation: spec[1],
-                  criterion: spec[2],
-                  actual: truth[0],
-                  passed: truth[1]
-                });
-                if (! truth[1]) {
-                  failureCount++;
-                }
+          // Add the error data to the act.
+          const {code, error, kind, signal} = actResult;
+          if (kind === 'close' && timedOut) {
+            addError(
+              false, false, report, actIndex, `Timed out at ${Math.round(limitMs / 1000)} seconds`
+            );
+          }
+          else if (kind === 'close') {
+            addError(
+              true, false, report, actIndex, `Closed with code ${code} and signal ${signal})`
+            );
+          }
+          else {
+            addError(
+              true, false, report, actIndex, `Terminated with error ${error}`
+            );
+          }
+        }
+        // Get the (usually revised) act.
+        act = acts[actIndex];
+        // Add the elapsed time of the tool to the report.
+        const time = Math.round((Date.now() - startTime) / 1000);
+        const {toolTimes} = report.jobData;
+        toolTimes[act.which] ??= 0;
+        toolTimes[act.which] += time;
+        // If the act was not prevented:
+        if (act.data && ! act.data.prevented) {
+          const expectations = act.expect;
+          // If the act has expectations:
+          if (expectations) {
+            // Initialize whether they were fulfilled.
+            act.expectations = [];
+            let failureCount = 0;
+            // For each expectation:
+            expectations.forEach(spec => {
+              // Add its result to the act.
+              const truth = isTrue(act, spec);
+              act.expectations.push({
+                property: spec[0],
+                relation: spec[1],
+                criterion: spec[2],
+                actual: truth[0],
+                passed: truth[1]
               });
-              act.expectationFailures = failureCount;
-            }
+              if (! truth[1]) {
+                failureCount++;
+              }
+            });
+            act.expectationFailures = failureCount;
           }
         }
       }
@@ -1602,10 +1682,9 @@ const doActs = async (report, opts = {}) => {
         // Notify the observer of the act and log it.
         tellServer(report, messageParams, message);
       }
-
+    }
     // Notify the observer and log the start of standardization.
     tellServer(report, '', 'Starting result standardization');
-    }
     const launchSpecActs = {};
     // For each act:
     report.acts.forEach((act, index) => {
@@ -1645,20 +1724,36 @@ const doActs = async (report, opts = {}) => {
           };
           // Populate it.
           standardize(act);
+          // If the original-format result is not to be included in the report:
+          if (standard === 'only') {
+            // Remove it.
+            delete act.result;
+          }
+          // Notify the observer and log the start of identification.
+          tellServer(report, '', 'Starting element identification');
           // For each of its standard instances:
           for (const instance of act.standardResult.instances) {
-            // If the instance does not have both a box ID and a path ID:
-            if (! (instance.boxID && instance.pathID)) {
+            let {boxID, pathID} = instance;
+            // If the instance does not have both a box ID and a valid path ID:
+            if (! boxID && (! pathID || pathID.includes(' '))) {
               const elementID = await identify(instance, page);
               // If it has no box ID but the element has a bounding box:
-              if (elementID.boxID && ! instance.boxID) {
-                // Add a box ID.
+              if (elementID.boxID && ! boxID) {
+                // Add a box ID to the instance.
                 instance.boxID = elementID.boxID;
               }
-              // If it has no path ID but the element has one:
-              if (elementID.pathID && ! instance.pathID) {
-                // Add a path ID.
-                instance.pathID = elementID.pathID;
+              // If it has no valid path ID:
+              if (! pathID || pathID.includes(' ')) {
+                // If the element has a valid path ID:
+                if (elementID.pathID && ! elementID.pathID.includes(' ')) {
+                  // Add or replace the path ID of the instance.
+                  instance.pathID = elementID.pathID;
+                }
+                // Otherwise, if the instance has an invalid but uncorrectable path ID:
+                else if (pathID) {
+                  // Delete it.
+                  delete instance.pathID;
+                }
               }
             }
             // If the instance excerpt contains a unique Testaro identifier attribute:
@@ -1669,12 +1764,22 @@ const doActs = async (report, opts = {}) => {
               .replace(/ data-testaro-id="[^" ]*("|$)/g, '')
               .replace(/ data-testaro-id="[^" ]* /g, ' ');
             }
+            pathID = instance.pathID;
+            // If the instance has a pathID and no text property:
+            if (pathID && ! instance.text) {
+              // Get the element.
+              const elementLoc = page.locator(`xpath=${pathID}`, {hasText: /.+/});
+              // If it exists and is unique:
+              if (await elementLoc.count() === 1) {
+                let text = await elementLoc.textContent();
+                if (text.length > 200) {
+                  text = `${text.slice(0, 150)} … ${text.slice(-50)}`;
+                }
+                // Add the text content or its ends to the instance.
+                instance.text = text.trim();
+              }
+            }
           };
-          // If the original-format result is not to be included in the report:
-          if (standard === 'only') {
-            // Remove it.
-            delete act.result;
-          }
         };
       }
       // Otherwise, i.e. if the launch or navigation failed:
@@ -1682,9 +1787,9 @@ const doActs = async (report, opts = {}) => {
         console.log(`ERROR: Launch or navigation to standardize ${specString} acts failed`);
       }
     };
-    // Close the last browser launched for standardization.
+    // Close the last browser launched for standardization and element identification.
     await browserClose();
-    console.log('Standardization completed');
+    console.log('Standardization and element identification completed');
     const {acts} = report;
     const idData = {};
     // For each act:
@@ -1697,15 +1802,17 @@ const doActs = async (report, opts = {}) => {
           instanceCount: 0,
           boxIDCount: 0,
           pathIDCount: 0,
+          textCount: 0,
           boxIDPercent: null,
-          pathIDPercent: null
+          pathIDPercent: null,
+          textPercent: null
         };
         const actIDData = idData[which];
         const {standardResult} = act;
         const {instances} = standardResult;
         // For each standard instance in the act:
         for (const instance of instances) {
-          const {boxID, pathID} = instance;
+          const {boxID, pathID, text} = instance;
           // Increment the instance count.
           actIDData.instanceCount++;
           // If the instance has a box ID:
@@ -1718,13 +1825,19 @@ const doActs = async (report, opts = {}) => {
             // Increment the path ID count.
             actIDData.pathIDCount++;
           }
+          // If the instance has a text:
+          if (text) {
+            // Increment the text count.
+            actIDData.textCount++;
+          }
         }
-        const {instanceCount, boxIDCount, pathIDCount} = actIDData;
+        const {instanceCount, boxIDCount, pathIDCount, textCount} = actIDData;
         // If there are any instances:
         if (instanceCount) {
-          // Add the box ID and path ID percentages to the iData property.
+          // Add the box ID path ID, and text percentages to the iData property.
           actIDData.boxIDPercent = Math.round(100 * boxIDCount / instanceCount);
           actIDData.pathIDPercent = Math.round(100 * pathIDCount / instanceCount);
+          actIDData.textPercent = Math.round(100 * textCount / instanceCount);
         }
       }
     }