testaro 5.7.2 → 5.7.6

package/README.md

@@ -93,6 +93,7 @@ A script is a JSON file with the properties:
 
 ```json
 {
+  "id": "string consisting of lower-case ASCII letters and digits",
   "what": "string: description of the script",
   "strict": "boolean: whether redirections should be treated as failures",
   "timeLimit": "number: limit in seconds on the execution of this script",
@@ -100,32 +101,33 @@ A script is a JSON file with the properties:
 }
 ```
 
-The `timeLimit` property is optional. If it is omitted, a default of 300 seconds (5 minutes) is set.
+The `timeLimit` property is optional. If it is omitted, a default of 300 seconds (5 minutes) is set. 
 
 ### Example
 
 Here is an example of a script:
 
-```javascript
+```json
 {
-  what: 'Test example.com with alfa',
+  "id": "samplescript",
+  what: "Test example.com with alfa",
   strict: true,
-  timeLimit: 15,
+  timeLimit: 65,
   commands: [
     {
-      type: 'launch',
-      which: 'chromium',
-      what: 'Chromium browser'
+      type: "launch",
+      which: "chromium",
+      what: "Chromium browser"
     },
     {
-      type: 'url',
-      which: 'https://example.com/',
-      what: 'page with a few accessibility defects'
+      type: "url",
+      which: "https://example.com/",
+      what: "page with a few accessibility defects"
     },
     {
-      type: 'test',
-      which: 'alfa',
-      what: 'Siteimprove alfa package'
+      type: "test",
+      which: "alfa",
+      what: "Siteimprove alfa package"
     }
   ]
 }
@@ -302,6 +304,10 @@ In case you want to perform more than one `tenon` test with the same script, you
 
 Tenon recommends giving it a public URL rather than giving it the content of a page, if possible. So, it is best to give the `withNewContent` property of the `tenonRequest` command the value `true`, unless the page is not public.
 
+###### Continuum
+
+The `continuum` test makes use of the files in the `continuum` directory. The test inserts the contents of all three files into the page as scripts and then uses them to perform the tests of the Continuum package.
+
 ###### HTML CodeSniffer
 
 The `htmlcs` test makes use of the`htmlcs/HTMLCS.js` file. That file was created, and can be recreated if necessary, as follows:
@@ -336,7 +342,7 @@ The changes in `htmlcs/HTMLCS.js` are:
 
 ###### BBC Accessibility Standards Checker
 
-The BBC Accessibility Standards Checker has obsolete dependencies with security vulnerabilities. Therefore, it is not used as a dependency of Testaro. Instead, 6 of its tests were reimplemented, in some case with revisions, as Testaro tests. They were drawn from the 18 automated tests of the Checker. The other 12 tests were found too duplicative of other tests to justify reimplementation.
+The BBC Accessibility Standards Checker has obsolete dependencies with security vulnerabilities. Therefore, it is not used as a dependency of Testaro. Instead, 6 of its tests are reimplemented, in some case with revisions, as Testaro tests. They are drawn from the 18 automated tests of the Checker. The other 12 tests were found too duplicative of other tests to justify reimplementation.
 
 ##### Branching
 
@@ -436,18 +442,18 @@ A typical use for an `expect` property is checking the correctness of a Testaro
 
 You may wish to have Testaro perform the same sequence of tests on multiple web pages. In that case, you can create a _batch_, with the following structure:
 
-```javascript
+```json
 {
-  what: 'Web leaders',
+  what: "Web leaders",
   hosts: {
-    id: 'w3c',
-    which: 'https://www.w3.org/',
-    what: 'W3C'
+    id: "w3c",
+    which: "https://www.w3.org/",
+    what: "W3C"
   },
   {
-    id: 'wikimedia',
-    which: 'https://www.wikimedia.org/',
-    what: 'Wikimedia'
+    id: "wikimedia",
+    which: "https://www.wikimedia.org/",
+    what: "Wikimedia"
   }
 }
 ```
@@ -487,7 +493,7 @@ The argument of `require` is a path relative to the directory of the module in w
 
 Another Node.js package that has Testaro as a dependency can execute the same statements, except changing `'./run'` to `'testaro/run'`.
 
-Testaro will run the script and populate the `log` and `acts` arrays of the `report` object. When Testaro finishes, the `log` and `acts` properties will contain the results. The final statement can further process the `report` object as desired in the `then` callback.
+Testaro will run the script and modify the properties of the `report` object. When Testaro finishes, the `log`, `acts`, and other properties of `report` will contain the results. The final statement can further process the `report` object as desired in the `then` callback.
 
 #### High-level
 
@@ -500,7 +506,7 @@ Relative paths must be relative to the Testaro project directory. For example, i
 
 Also ensure that Testaro can read all those directories and write to `REPORTDIR`.
 
-Place a script into `SCRIPTDIR` and, optionally, a batch into `BATCHDIR`. Each should be named `idvalue.json`, where `idvalue` is replaced with the value of its `id` property. That value must consist of only lower-case ASCII letters and digits.
+Place a script into `SCRIPTDIR` and, optionally, a batch into `BATCHDIR`. Each should be named with a `.json` extension., where `idvalue` is replaced with the value of its `id` property. That value must consist of only lower-case ASCII letters and digits.
 
 Then execute the statement `node high scriptID` or `node high scriptID batchID`, replacing `scriptID` and `batchID` with the `id` values of the script and the batch, respectively.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "5.7.2",
+  "version": "5.7.6",
   "description": "Automation of accessibility testing",
   "main": "index.js",
   "scripts": {

package/run.js

@@ -1,5 +1,5 @@
 /*
-  index.js
+  run.js
   testaro main script.
 */
 
@@ -103,8 +103,10 @@ let errorLogSize = 0;
 let prohibitedCount = 0;
 let visitTimeoutCount = 0;
 let visitRejectionCount = 0;
+let visitLatency = 0;
 let actCount = 0;
 // Facts about the current browser.
+let browser;
 let browserContext;
 let browserTypeName;
 let requestedURL = '';
@@ -256,12 +258,24 @@ const isValidReport = async report => {
 
 // ########## OTHER FUNCTIONS
 
+// Closes the current browser.
+const browserClose = async () => {
+  if (browser) {
+    const contexts = browser.contexts();
+    for (const context of contexts) {
+      await context.close();
+    }
+    await browser.close();
+  }
+};
 // Launches a browser.
 const launch = async typeName => {
   const browserType = require('playwright')[typeName];
   // If the specified browser type exists:
   if (browserType) {
-    // Launch a browser of the specified type.
+    // Close the current browser, if any.
+    await browserClose();
+    // Launch a browser of that type.
     const browserOptions = {};
     if (debug) {
       browserOptions.headless = false;
@@ -270,7 +284,7 @@ const launch = async typeName => {
       browserOptions.slowMo = waits;
     }
     let healthy = true;
-    const browser = await browserType.launch(browserOptions)
+    browser = await browserType.launch(browserOptions)
     .catch(error => {
       healthy = false;
       console.log(`ERROR launching browser: ${error.message.replace(/\n.+/s, '')}`);
@@ -511,6 +525,7 @@ const goto = async (page, url, timeout, waitUntil, isStrict) => {
     url = url.replace('file://', `file://${__dirname}/`);
   }
   // Visit the URL.
+  const startTime = Date.now();
   const response = await page.goto(url, {
     timeout,
     waitUntil
@@ -520,6 +535,7 @@ const goto = async (page, url, timeout, waitUntil, isStrict) => {
     visitTimeoutCount++;
     return 'error';
   });
+  visitLatency += Math.round((Date.now() - startTime) / 1000);
   // If the visit succeeded:
   if (typeof response !== 'string') {
     const httpStatus = response.status();
@@ -733,7 +749,7 @@ const doActs = async (report, actIndex, page) => {
         // If the command is a url:
         if (act.type === 'url') {
           // Visit it and wait until it is stable.
-          page = await visit(act, page, report.strict);
+          page = await visit(act, page, report.isStrict);
         }
         // Otherwise, if the act is a wait for text:
         else if (act.type === 'wait') {
@@ -1335,6 +1351,7 @@ const doScript = async (report) => {
   report.prohibitedCount = prohibitedCount;
   report.visitTimeoutCount = visitTimeoutCount;
   report.visitRejectionCount = visitRejectionCount;
+  report.visitLatency = visitLatency;
   // Add the end time and duration to the report.
   const endTime = new Date();
   report.endTime = endTime.toISOString().slice(0, 19);

package/tests/hover.js

@@ -28,6 +28,10 @@
   value in the same location being the target.
 */
 
+// VARIABLES
+
+let hasTimedOut = false;
+
 // FUNCTIONS
 
 // Samples a population and returns the sample.
@@ -59,153 +63,172 @@ const textOf = async (element, limit) => {
 };
 // Recursively reports impacts of hovering over triggers.
 const find = async (data, withItems, page, region, sample, popRatio) => {
-  // If any potential triggers remain:
-  if (sample.length) {
-    // Identify the first of them.
-    const firstTrigger = sample[0];
-    const tagNameJSHandle = await firstTrigger.getProperty('tagName')
-    .catch(error => {
-      return '';
-    });
-    if (tagNameJSHandle) {
-      const tagName = await tagNameJSHandle.jsonValue();
-      // Identify the root of a subtree likely to contain impacted elements.
-      let root = firstTrigger;
-      if (['A', 'BUTTON', 'LI'].includes(tagName)) {
-        const rootJSHandle = await page.evaluateHandle(
-          firstTrigger => {
-            const parent = firstTrigger.parentElement || firstTrigger;
-            const grandparent = parent.parentElement || parent;
-            const greatGrandparent = grandparent.parentElement || parent;
-            return firstTrigger.tagName === 'LI' ? grandparent : greatGrandparent;
-          },
-          firstTrigger
-        );
-        root = rootJSHandle.asElement();
-      }
-      // Identify all the descendants of the root.
-      const preDescendants = await root.$$(':visible');
-      // Identify their opacities.
-      const preOpacities = await page.evaluate(elements => elements.map(
-        element => window.getComputedStyle(element).opacity
-      ), preDescendants);
-      try {
-        // Hover over the trigger.
-        await firstTrigger.hover({
-          timeout: 500,
-          noWaitAfter: true
-        });
-        // Repeatedly seeks impacts.
-        const getImpacts = async (interval, triesLeft) => {
-          // If the allowed trial count has not yet been exhausted:
-          if (triesLeft--) {
-            // Get the collection of descendants of the root.
-            const postDescendants = await root.$$(':visible');
-            // Identify the prior descandants of the root still in existence.
-            const remainerIndexes = await page.evaluate(args => {
-              const preDescendants = args[0];
-              const postDescendants = args[1];
-              const remainerIndexes = preDescendants
-              .map((element, index) => postDescendants.includes(element) ? index : -1)
-              .filter(index => index > -1);
-              return remainerIndexes;
-            }, [preDescendants, postDescendants]);
-            // Get the count of elements added by the hover event.
-            const additionCount = postDescendants.length - remainerIndexes.length;
-            const removalCount = preDescendants.length - remainerIndexes.length;
-            const remainers = [];
-            for (const index of remainerIndexes) {
-              remainers.push({
-                element: preDescendants[index],
-                preOpacity: preOpacities[index],
-                postOpacity: await page.evaluate(
-                  element => window.getComputedStyle(element).opacity, preDescendants[index]
-                )
-              });
-            }
-            const opacityChangers = remainers
-            .filter(remainer => remainer.postOpacity !== remainer.preOpacity);
-            const opacityImpact = opacityChangers ? await page.evaluate(changers => changers.reduce(
-              (total, current) => total + current.element.querySelectorAll('*').length, 0
-            ), opacityChangers) : 0;
-            if (additionCount || removalCount || opacityChangers.length) {
-              return {
-                additionCount,
-                removalCount,
-                opacityChangers,
-                opacityImpact
-              };
-            }
-            else {
-              return await new Promise(resolve => {
-                setTimeout(() => {
-                  resolve(getImpacts(interval, triesLeft));
-                }, interval);
-              });
-            }
-          }
-          else {
-            return null;
-          }
-        };
-        // Repeatedly seek impacts of the hover at intervals.
-        const impacts = await getImpacts(300, 4);
-        // If there were any:
-        if (impacts) {
-          // Hover over the upper-left corner of the page, to undo any impacts.
-          await page.hover('body', {
-            position: {
-              x: 0,
-              y: 0
+  // If any potential triggers remain and the test has not timed out:
+  if (sample.length && ! hasTimedOut) {
+    // Get and report the impacts until and unless the test times out.
+    try {
+      // Identify the first of them.
+      const firstTrigger = sample[0];
+      const tagNameJSHandle = await firstTrigger.getProperty('tagName')
+      .catch(error => '');
+      if (tagNameJSHandle) {
+        const tagName = await tagNameJSHandle.jsonValue();
+        // Identify the root of a subtree likely to contain impacted elements.
+        let root = firstTrigger;
+        if (['A', 'BUTTON', 'LI'].includes(tagName)) {
+          const rootJSHandle = await page.evaluateHandle(
+            firstTrigger => {
+              const parent = firstTrigger.parentElement || firstTrigger;
+              const grandparent = parent.parentElement || parent;
+              const greatGrandparent = grandparent.parentElement || parent;
+              return firstTrigger.tagName === 'LI' ? grandparent : greatGrandparent;
             },
+            firstTrigger
+          );
+          root = rootJSHandle.asElement();
+        }
+        // Identify all the visible descendants of the root.
+        const preDescendants = await root.$$(':visible');
+        // Identify their opacities.
+        const preOpacities = await page.evaluate(elements => elements.map(
+          element => window.getComputedStyle(element).opacity
+        ), preDescendants);
+        try {
+          // Hover over the trigger.
+          await firstTrigger.hover({
             timeout: 500,
-            force: true,
             noWaitAfter: true
           });
-          // Wait for any delayed and/or slowed hover reaction.
-          await page.waitForTimeout(200);
-          await root.waitForElementState('stable');
-          // Increment the counts of triggers and impacts.
-          const {additionCount, removalCount, opacityChangers, opacityImpact} = impacts;
-          data.totals.impactTriggers += popRatio;
-          data.totals.additions += popRatio * additionCount;
-          data.totals.removals += popRatio * removalCount;
-          data.totals.opacityChanges += popRatio * opacityChangers.length;
-          data.totals.opacityImpact += popRatio * opacityImpact;
-          // If details are to be reported:
-          if (withItems) {
-            // Report them.
-            data.items[region].impactTriggers.push({
-              tagName,
-              text: await textOf(firstTrigger, 50),
-              additions: additionCount,
-              removals: removalCount,
-              opacityChanges: opacityChangers.length,
-              opacityImpact
+          // FUNCTION DEFINITION START
+          // Repeatedly seeks impacts.
+          const getImpacts = async (interval, triesLeft) => {
+            // If the allowed trial count has not yet been exhausted:
+            if (triesLeft-- && ! hasTimedOut) {
+              // Get the collection of descendants of the root.
+              const postDescendants = await root.$$(':visible');
+              // Identify the prior descandants of the root still in existence.
+              const remainerIndexes = await page.evaluate(args => {
+                const preDescendants = args[0];
+                const postDescendants = args[1];
+                const remainerIndexes = preDescendants
+                .map((element, index) => postDescendants.includes(element) ? index : -1)
+                .filter(index => index > -1);
+                return remainerIndexes;
+              }, [preDescendants, postDescendants]);
+              // Get the count of elements added by the hover event.
+              const additionCount = postDescendants.length - remainerIndexes.length;
+              const removalCount = preDescendants.length - remainerIndexes.length;
+              const remainers = [];
+              for (const index of remainerIndexes) {
+                remainers.push({
+                  element: preDescendants[index],
+                  preOpacity: preOpacities[index],
+                  postOpacity: await page.evaluate(
+                    element => window.getComputedStyle(element).opacity, preDescendants[index]
+                  )
+                });
+              }
+              const opacityChangers = remainers
+              .filter(remainer => remainer.postOpacity !== remainer.preOpacity);
+              const opacityImpact = opacityChangers ? await page.evaluate(changers => changers.reduce(
+                (total, current) => total + current.element.querySelectorAll('*').length, 0
+              ), opacityChangers) : 0;
+              if (additionCount || removalCount || opacityChangers.length) {
+                return {
+                  additionCount,
+                  removalCount,
+                  opacityChangers,
+                  opacityImpact
+                };
+              }
+              else {
+                return await new Promise(resolve => {
+                  setTimeout(() => {
+                    resolve(getImpacts(interval, triesLeft));
+                  }, interval);
+                });
+              }
+            }
+            else {
+              return null;
+            }
+          };
+          // FUNCTION DEFINITION END
+          // Repeatedly seek impacts of the hover at intervals.
+          const impacts = await getImpacts(300, 4);
+          // If there were any:
+          if (impacts) {
+            // Hover over the upper-left corner of the page, to undo any impacts.
+            await page.hover('body', {
+              position: {
+                x: 0,
+                y: 0
+              },
+              timeout: 500,
+              force: true,
+              noWaitAfter: true
             });
+            // Wait for any delayed and/or slowed hover reaction.
+            await page.waitForTimeout(200);
+            await root.waitForElementState('stable');
+            // Increment the counts of triggers and impacts.
+            const {additionCount, removalCount, opacityChangers, opacityImpact} = impacts;
+            if (hasTimedOut) {
+              return Promise.resolve('');
+            }
+            else {
+              data.totals.impactTriggers += popRatio;
+              data.totals.additions += popRatio * additionCount;
+              data.totals.removals += popRatio * removalCount;
+              data.totals.opacityChanges += popRatio * opacityChangers.length;
+              data.totals.opacityImpact += popRatio * opacityImpact;
+              // If details are to be reported:
+              if (withItems) {
+                // Report them.
+                data.items[region].impactTriggers.push({
+                  tagName,
+                  text: await textOf(firstTrigger, 50),
+                  additions: additionCount,
+                  removals: removalCount,
+                  opacityChanges: opacityChangers.length,
+                  opacityImpact
+                });
+              }
+            }
           }
         }
-      }
-      catch (error) {
-        console.log(`ERROR hovering (${error.message.replace(/\n.+/s, '')})`);
-        data.totals.unhoverables++;
-        if (withItems) {
-          try {
-            const id = await firstTrigger.getAttribute('id');
-            data.items[region].unhoverables.push({
-              tagName,
-              id: id || '',
-              text: await textOf(firstTrigger, 50)
-            });
+        catch (error) {
+          console.log(`ERROR hovering (${error.message.replace(/\n.+/s, '')})`);
+          if (hasTimedOut) {
+            return Promise.resolve('');
           }
-          catch(error) {
-            console.log('ERROR itemizing unhoverable element');
+          else {
+            data.totals.unhoverables++;
+            if (withItems) {
+              try {
+                const id = await firstTrigger.getAttribute('id');
+                data.items[region].unhoverables.push({
+                  tagName,
+                  id: id || '',
+                  text: await textOf(firstTrigger, 50)
+                });
+              }
+              catch(error) {
+                console.log('ERROR itemizing unhoverable element');
+              }
+            }
           }
         }
       }
+      // Process the remaining potential triggers.
+      await find(data, withItems, page, region, sample.slice(1), popRatio);
     }
-    // Process the remaining potential triggers.
-    await find(data, withItems, page, region, sample.slice(1), popRatio);
+    catch(error) {
+      console.log(`ERROR: Test quit when remaining sample size was ${sample.length}`);
+    }
+  }
+  else {
+    return Promise.resolve('');
   }
 };
 // Performs the hover test and reports results.
@@ -213,7 +236,7 @@ exports.reporter = async (
   page, headSize = 0, headSampleSize = -1, tailSampleSize = -1, withItems
 ) => {
   // Initialize the result.
-  const data = {
+  let data = {
     totals: {
       triggers: 0,
       headTriggers: 0,
@@ -259,17 +282,34 @@ exports.reporter = async (
   // Get the head and tail samples.
   const headSample = getSample(headTriggers, headSampleSize);
   const tailSample = tailSampleSize === -1 ? tailTriggers : getSample(tailTriggers, tailSampleSize);
+  // Set a time limit to handle pages that slow the operations of this test.
+  const timeLimit = Math.round(1.3 * (headSample.length + tailSample.length));
+  const timeout = setTimeout(async () => {
+    await page.close();
+    console.log(
+      `ERROR: hover test timed out at ${timeLimit} seconds; page closed`
+    );
+    hasTimedOut = true;
+    data = {
+      prevented: true,
+      error: 'ERROR: hover test timed out'
+    };
+    clearTimeout(timeout);
+  }, 1000 * timeLimit);
   // Find and document the impacts.
-  if (headSample.length) {
+  if (headSample.length && ! hasTimedOut) {
     await find(data, withItems, page, 'head', headSample, headTriggerCount / headSample.length);
   }
-  if (tailSample.length) {
+  if (tailSample.length && ! hasTimedOut) {
     await find(data, withItems, page, 'tail', tailSample, tailTriggerCount / tailSample.length);
   }
+  clearTimeout(timeout);
   // Round the reported totals.
-  Object.keys(data.totals).forEach(key => {
-    data.totals[key] = Math.round(data.totals[key]);
-  });
+  if (! hasTimedOut) {
+    Object.keys(data.totals).forEach(key => {
+      data.totals[key] = Math.round(data.totals[key]);
+    });
+  }
   // Return the result.
   return {result: data};
 };

package/tests/nuVal.js

@@ -1,16 +1,24 @@
 /*
   nuVal
   This test subjects a page to the Nu Html Checker.
+  That API erratically replaces left and right double quotation marks with invalid UTF-8, which
+  appears as 2 or 3 successive instances of the replacement character (U+fffd). Therefore, this
+  test removes all such quotation marks and the replacement character. That causes
+  'Bad value “” for' to become 'Bad value  for'. Since the corruption of quotation marks is
+  erratic, no better solution is known.
 */
 const https = require('https');
 exports.reporter = async page => {
   const pageContent = await page.content();
   // Get the data from a Nu validation.
-  const data = await new Promise((resolve, reject) => {
+  const dataPromise = new Promise(resolve => {
     try {
       const request = https.request(
         {
-          // Alternatives (more timeout-prone): host=validator.nu; path=/?parser=html@out=json
+          /*
+            Alternatives (more timeout-prone): host=validator.nu; path=/?parser=html@out=json
+            That host crashes instead of ending with a fatal error.
+          */
           host: 'validator.w3.org',
           path: '/nu/?parser=html&out=json',
           method: 'POST',
@@ -27,12 +35,12 @@ exports.reporter = async page => {
           // When the data arrive:
           response.on('end', async () => {
             try {
-              // Delete unnecessary properties.
-              const result = JSON.parse(report);
+              // Delete left and right quotation marks and their erratic invalid replacements.
+              const result = JSON.parse(report.replace(/[\u{fffd}“”]/ug, ''));
               return resolve(result);
             }
             catch (error) {
-              console.log(`Validation failed (${error.message})`);
+              console.log(`ERROR: Validation failed (${error.message})`);
               return resolve({
                 prevented: true,
                 error: error.message,
@@ -46,7 +54,7 @@ exports.reporter = async page => {
       request.end();
       request.on('error', error => {
         console.log(error.message);
-        return reject({
+        return resolve({
           prevented: true,
           error: error.message
         });
@@ -54,11 +62,22 @@ exports.reporter = async page => {
     }
     catch(error) {
       console.log(error.message);
-      return reject({
+      return resolve({
         prevented: true,
         error: error.message
       });
     }
   });
+  const timeoutPromise = new Promise(resolve => {
+    const timeLimit = 12;
+    const timeoutID = setTimeout(() => {
+      resolve({
+        prevented: true,
+        error: `ERROR: Validation timed out at ${timeLimit} seconds`
+      });
+      clearTimeout(timeoutID);
+    }, 1000 * timeLimit);
+  });
+  const data = await Promise.race([dataPromise, timeoutPromise]);
   return {result: data};
 };