testaro 5.2.2 → 5.5.0

package/README.md

@@ -89,10 +89,13 @@ A script is a JSON file with the properties:
 {
   "what": "string: description of the script",
   "strict": "boolean: whether redirections should be treated as failures",
+  "timeLimit": "number: limit in seconds on the execution of this script",
   "commands": "array of objects: the commands to be performed"
 }
 ```
 
+The `timeLimit` property is optional. If it is omitted, a default of 300 seconds (5 minutes) is set.
+
 ### Example
 
 Here is an example of a script:
@@ -101,6 +104,7 @@ Here is an example of a script:
 {
   what: 'Test example.com with alfa',
   strict: true,
+  timeLimit: 15,
   commands: [
     {
       type: 'launch',
@@ -486,13 +490,13 @@ Relative paths must be relative to the Testaro project directory. For example, i
 
 Also ensure that Testaro can read all those directories and write to `REPORTDIR`.
 
-Place a script into `SCRIPTDIR` and, optionally, a batch into `BATCHDIR`. Each should be named `idValue.json`, where `idValue` is replaced with the value of its `id` property. That value must consist of only lower-case ASCII letters and digits.
+Place a script into `SCRIPTDIR` and, optionally, a batch into `BATCHDIR`. Each should be named `idvalue.json`, where `idvalue` is replaced with the value of its `id` property. That value must consist of only lower-case ASCII letters and digits.
 
 Then execute the statement `node high scriptID` or `node high scriptID batchID`, replacing `scriptID` and `batchID` with the `id` values of the script and the batch, respectively.
 
 The `high` module will call the `runJob` function of the `create` module, which in turn will call the `handleRequest` function of the `run` module. The results will be saved in report files in the `REPORTDIR` directory.
 
-If there is no batch, the report file will be named with a unique timestamp, suffixed with a `.json` extension. If there is a batch, then the base of each report file’s name will be the same timestamp, suffixed with `-hostID`, where `hostID` is the value of the `id` property of the `host` object in the batch file. For example, if you execute `node create script01 wikis`, you might get these report files deposited into `REPORTDIR`:
+If there is no batch, the report file will be named with a unique timestamp, suffixed with a `.json` extension. If there is a batch, then the base of each report file’s name will be the same timestamp, suffixed with `-hostid`, where `hostid` is the value of the `id` property of the `host` object in the batch file. For example, if you execute `node create script01 wikis`, you might get these report files deposited into `REPORTDIR`:
 - `enp46j-wikipedia.json`
 - `enp45j-wiktionary.json`
 - `enp45j-wikidata.json`
@@ -602,6 +606,16 @@ The rationales motivating the Testaro-defined tests can be found in comments wit
 
 ## Testing challenges
 
+### Abnormal termination
+
+On rare occasions a test throws an error that terminates the Node process and cannot be handled with a `try`-`catch` structure. It has been observed, for example, that the `ibm` test does this when run on the host at `https://zenythgroup.com/index` or `https://monsido.com`.
+
+If a single process performed all of the commands in a batch-based script, the process could perform tens of thousands of commands, and such an error could stop the process at any point.
+
+To handle this risk, Testaro processes batch-based jobs by forking a new process for each host. If such an error occurs, it crashes the child process, preventing a report for that host from being written. The parent process waits for the report to appear in the `REPORTDIR` directory until the time limit. When it fails to appear, the parent process continues to the next host.
+
+If you are using high-level invocation, your terminal will show the standard output of the parent process and, if there is a batch, the current child process, too. If you interrupt the process with `CTRL-c`, you will send a `SIGINT` signal to the parent process, which will handle it by sending a message to the child process telling it to terminate itself, and then the parent process will terminate by skipping the remaining hosts.
+
 ### Activation
 
 Testing to determine what happens when a control or link is activated is straightforward, except in the context of a comprehensive set of tests of a single page. There, activating a control or link can change the page or navigate away from it, interfering with the remaining planned tests of the page.

package/create.js

@@ -9,6 +9,7 @@
 require('dotenv').config();
 // Module to read and write files.
 const fs = require('fs/promises');
+const {fork} = require('child_process');
 const {handleRequest} = require('./run');
 // Module to convert a script and a batch to a batch-based array of scripts.
 const {batchify} = require('./batchify');
@@ -21,10 +22,10 @@ const reportDir = process.env.REPORTDIR;
 // ########## FUNCTIONS
 
 // Runs one script and writes a report file.
-const runHost = async (id, script, host = {}) => {
+const runHost = async (id, script) => {
   const report = {
     id,
-    host,
+    host: {},
     log: [],
     script,
     acts: []
@@ -35,10 +36,19 @@ const runHost = async (id, script, host = {}) => {
 };
 // Runs a file-based job and writes a report file for the script or each host.
 exports.runJob = async (scriptID, batchID) => {
+  let healthy = true;
+  let childAlive = true;
+  process.on('SIGINT', () => {
+    console.log('ERROR: Terminal interrupted runJob');
+    healthy = false;
+  });
   if (scriptID) {
     try {
       const scriptJSON = await fs.readFile(`${scriptDir}/${scriptID}.json`, 'utf8');
       const script = JSON.parse(scriptJSON);
+      // Get the time limit of the script or, if none, set it to 5 minutes.
+      let {timeLimit} = script;
+      timeLimit = timeLimit || 300;
       // Identify the start time and a timestamp.
       const timeStamp = Math.floor((Date.now() - Date.UTC(2022, 1)) / 2000).toString(36);
       // If there is a batch:
@@ -48,28 +58,89 @@ exports.runJob = async (scriptID, batchID) => {
         const batchJSON = await fs.readFile(`${batchDir}/${batchID}.json`, 'utf8');
         batch = JSON.parse(batchJSON);
         const specs = batchify(script, batch, timeStamp);
-        // For each host script:
-        while (specs.length) {
-          const spec = specs.shift();
-          const {id, host, script} = spec;
-          // Run it and save the result with a host-suffixed ID.
-          await runHost(id, script, host);
-        }
+        const batchSize = specs.length;
+        const sizedRep = `${batchSize} report${batchSize > 1 ? 's' : ''}`;
+        const timeoutHosts = [];
+        const crashHosts = [];
+        // FUNCTION DEFINITION START
+        // Recursively runs host scripts.
+        const runHosts = specs => {
+          // If any scripts remain to be run and the process has not been interrupted:
+          if (specs.length && healthy) {
+            childAlive = true;
+            // Run the first one and save the report with a host-suffixed ID.
+            const spec = specs.shift();
+            const {id, host, script} = spec;
+            const subprocess = fork(
+              'runHost', [id, JSON.stringify(script), JSON.stringify(host)],
+              {
+                detached: true,
+                stdio: [0, 1, 'ipc']
+              }
+            );
+            subprocess.on('exit', () => {
+              childAlive = false;
+            });
+            const startTime = Date.now();
+            // At 5-second intervals:
+            const reCheck = setInterval(async () => {
+              // If the user has not interrupted the process:
+              if (healthy) {
+                // If there is no need to keep checking:
+                const reportNames = await fs.readdir(reportDir);
+                const timedOut = Date.now() - startTime > 1000 * timeLimit;
+                if (timedOut || reportNames.includes(`${id}.json`) || ! childAlive) {
+                  // Stop checking.
+                  clearInterval(reCheck);
+                  // If the cause is a timeout:
+                  if (timedOut) {
+                    // Add the host to the array of timed-out hosts.
+                    timeoutHosts.push(id);
+                  }
+                  // Otherwise, if the cause is a child crash:
+                  else if (! childAlive) {
+                    // Add the host to the array of crashed hosts.
+                    crashHosts.push(id);
+                  }
+                  // Run the script of the next host.
+                  runHosts(specs);
+                }
+              }
+              // Otherwise, i.e. if the user has interrupted the process:
+              else {
+                // Tell the script run to quit.
+                subprocess.send('interrupt');
+                // Stop checking.
+                clearInterval(reCheck);
+              }
+            }, 5000);
+          }
+          else {
+            console.log(`${sizedRep} ${timeStamp}-....json in ${process.env.REPORTDIR}`);
+            if (timeoutHosts.length) {
+              console.log(`Reports not created:\n${JSON.stringify(timeoutHosts), null, 2}`);
+            }
+            if (crashHosts.length) {
+              console.log(`Hosts crashed:\n${JSON.stringify(crashHosts), null, 2}`);
+            }
+          }
+        };
+        // FUNCTION DEFINITION END
+        // Recursively run each host script and save the reports.
+        runHosts(specs);
       }
       // Otherwise, i.e. if there is no batch:
       else {
         // Run the script and save the result with a timestamp ID.
         await runHost(timeStamp, script);
+        console.log(`Report ${timeStamp}.json in ${process.env.REPORTDIR}`);
       }
-      return timeStamp;
     }
     catch(error) {
       console.log(`ERROR: ${error.message}\n${error.stack}`);
-      return null;
     }
   }
   else {
     console.log('ERROR: no script specified');
-    return null;
   }
 };

package/data/roles.txt

@@ -0,0 +1,62 @@
+input type=checkbox	role=checkbox
+td	role=cell if the ancestor table element is exposed as a role=table
+td	role=gridcell if the ancestor table element is exposed as a role=grid or treegrid
+th	role=columnheader, rowheader or cell if the ancestor table element is exposed as a role=table
+td	role=columnheader, rowheader or gridcell if the ancestor table element is exposed as a role=grid or treegrid
+dd	role=definition	
+dt	role=term	
+hr	role=separator	
+li	role=listitem	
+ol	role=list	
+tr	role=row	
+ul	role=list	
+dfn	role=term
+nav	role=navigation	
+SVG	role=graphics-document	
+html	role=document
+main	role=main	
+math	role=math	
+menu	role=list	
+aside	role=complementary	
+table	role=table	
+tbody	role=rowgroup	
+tfoot	role=rowgroup	
+thead	role=rowgroup	
+button	role=button	
+dialog	role=dialog	
+figure	role=figure	
+output	role=status	
+article	role=article	
+details	role=group	
+section	role=region if the section element has an accessible name
+summary	role=button	
+datalist	role=listbox	
+fieldset	role=group	
+h1 to h6	role=heading, aria-level = the number in the element's tag name
+\
+progress	role=progressbar	
+textarea	role=textbox	
+a with href	role=link	
+area with href	role=link	
+img with alt=""	role=presentation
+input type=image	role=button	
+input type=radio	role=radio	
+input type=range	role=slider	
+input type=reset	role=button	
+input type=button	role=button	
+input type=number	role=spinbutton	
+input type=submit	role=button	
+img with alt="some text"	role=img	
+img without an alt attribute	role=img	
+input type=url with no list attribute	role=textbox	
+input type=tel, with no list attribute	role=textbox	
+input type=email with no list attribute	role=textbox	
+input type=search, with no list attribute	role=searchbox	
+form	If the form element has an accessible name: role=form. Otherwise, no corresponding role.	
+input type=text or with a missing or invalid type, with no list attribute	role=textbox	
+select (with a multiple attribute or a size attribute having value greater than 1)	role=listbox	
+select (with NO multiple attribute and NO size attribute having value greater than 1)	role=combobox	
+option element that is in a list of options or that represents a suggestion in a datalist	role=option	
+footer	If not a descendant of an article, aside, main, nav or section element, or an element with role=article, complementary, main, navigation or region then role=contentinfo. Otherwise no corresponding role.	
+header	If not a descendant of an article, aside, main, nav or section element, or an element with role=article, complementary, main, navigation or region then role=banner. Otherwise no corresponding role	
+input type=text, search, tel, url, email, or with a missing or invalid type, with a list attribute	role=combobox	

package/high.js

@@ -1,14 +1,10 @@
 /*
   high.js
   Invokes Testaro with the high-level method.
-  Usage example: node high tp11 weborgs
+  Usage example: node high tp12 weborgs
 */
 
 const {runJob} = require('./create');
 const scriptID = process.argv[2];
 const batchID = process.argv[3];
-const run = async (scriptID, batchID) => {
-  const timeStamp = await runJob(scriptID, batchID);
-  console.log(`Reports in ${process.env.REPORTDIR}; ID base ${timeStamp}`);
-};
-run(scriptID, batchID);
+runJob(scriptID, batchID);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "5.2.2",
+  "version": "5.5.0",
   "description": "Automation of accessibility testing",
   "main": "index.js",
   "scripts": {

package/run.js

@@ -633,6 +633,12 @@ const wait = ms => {
 };
 // Recursively performs the acts in a report.
 const doActs = async (report, actIndex, page) => {
+  process.on('message', message => {
+    if (message === 'interrupt') {
+      console.log('ERROR: Terminal interrupted doActs');
+      process.exit();
+    }
+  });
   const {acts} = report;
   // If any more commands are to be performed:
   if (actIndex > -1 && actIndex < acts.length) {
@@ -1310,8 +1316,7 @@ const doScript = async (report) => {
 const injectLaunches = acts => {
   let injectMore = true;
   while (injectMore) {
-    const injectIndex = acts.findIndex(
-      (act, index) =>
+    const injectIndex = acts.findIndex((act, index) =>
       index < acts.length - 1
       && act.type === 'test'
       && acts[index + 1].type === 'test'

package/runHost.js

@@ -0,0 +1,36 @@
+/*
+  runHost.js
+  Runs a host job and writes a report file.
+*/
+
+// ########## IMPORTS
+
+// Module to keep secrets.
+require('dotenv').config();
+// Module to read and write files.
+const fs = require('fs/promises');
+const {handleRequest} = require('./run');
+
+// ########## CONSTANTS
+const reportDir = process.env.REPORTDIR;
+
+// ########## FUNCTIONS
+
+// Runs one script and writes a report file.
+const runHost = async (id, scriptJSON, hostJSON) => {
+  const report = {
+    id,
+    host: JSON.parse(hostJSON),
+    log: [],
+    script: JSON.parse(scriptJSON),
+    acts: []
+  };
+  await handleRequest(report);
+  const reportJSON = JSON.stringify(report, null, 2);
+  await fs.writeFile(`${reportDir}/${id}.json`, reportJSON);
+  process.disconnect();
+  process.exit();
+};
+
+// ########## OPERATION
+runHost(... process.argv.slice(2));

package/tests/focInd.js

@@ -138,7 +138,7 @@ exports.reporter = async (page, revealAll, allowedDelay, withItems) => {
           const hasIndicator
           = hasDiffOutline
           || hasDiffBorder
-          || diff('box-shadow')
+          || diff('boxShadow')
           || diff('fontSize')
           || diff('fontStyle')
           || diff('textDecorationLine')

package/tests/hover.js

@@ -28,10 +28,6 @@
   value in the same location being the target.
 */
 
-// CONSTANTS
-
-const data = {};
-
 // FUNCTIONS
 
 // Samples a population and returns the sample.
@@ -62,14 +58,13 @@ const textOf = async (element, limit) => {
   return text.trim().replace(/\s*/sg, '').slice(0, limit);
 };
 // Recursively reports impacts of hovering over triggers.
-const find = async (withItems, page, region, sample, popRatio) => {
+const find = async (data, withItems, page, region, sample, popRatio) => {
   // If any potential triggers remain:
   if (sample.length) {
     // Identify the first of them.
     const firstTrigger = sample[0];
     const tagNameJSHandle = await firstTrigger.getProperty('tagName')
     .catch(error => {
-      console.log(`ERROR getting trigger tag name (${error.message})`);
       return '';
     });
     if (tagNameJSHandle) {
@@ -89,7 +84,7 @@ const find = async (withItems, page, region, sample, popRatio) => {
         root = rootJSHandle.asElement();
       }
       // Identify all the descendants of the root.
-      const preDescendants = await root.$$('*');
+      const preDescendants = await root.$$(':visible');
       // Identify their opacities.
       const preOpacities = await page.evaluate(elements => elements.map(
         element => window.getComputedStyle(element).opacity
@@ -102,8 +97,11 @@ const find = async (withItems, page, region, sample, popRatio) => {
         });
         // Repeatedly seeks impacts.
         const getImpacts = async (interval, triesLeft) => {
+          // If the allowed trial count has not yet been exhausted:
           if (triesLeft--) {
-            const postDescendants = await root.$$('*');
+            // Get the collection of descendants of the root.
+            const postDescendants = await root.$$(':visible');
+            // Identify the prior descandants of the root still in existence.
             const remainerIndexes = await page.evaluate(args => {
               const preDescendants = args[0];
               const postDescendants = args[1];
@@ -112,6 +110,7 @@ const find = async (withItems, page, region, sample, popRatio) => {
               .filter(index => index > -1);
               return remainerIndexes;
             }, [preDescendants, postDescendants]);
+            // Get the count of elements added by the hover event.
             const additionCount = postDescendants.length - remainerIndexes.length;
             const removalCount = preDescendants.length - remainerIndexes.length;
             const remainers = [];
@@ -158,7 +157,10 @@ const find = async (withItems, page, region, sample, popRatio) => {
             position: {
               x: 0,
               y: 0
-            }
+            },
+            timeout: 500,
+            force: true,
+            noWaitAfter: true
           });
           // Wait for any delayed and/or slowed hover reaction.
           await page.waitForTimeout(200);
@@ -198,7 +200,7 @@ const find = async (withItems, page, region, sample, popRatio) => {
       }
     }
     // Process the remaining potential triggers.
-    await find(withItems, page, region, sample.slice(1), popRatio);
+    await find(data, withItems, page, region, sample.slice(1), popRatio);
   }
 };
 // Performs the hover test and reports results.
@@ -206,16 +208,18 @@ exports.reporter = async (
   page, headSize = 0, headSampleSize = -1, tailSampleSize = -1, withItems
 ) => {
   // Initialize the result.
-  data.totals = {
-    triggers: 0,
-    headTriggers: 0,
-    tailTriggers: 0,
-    impactTriggers: 0,
-    additions: 0,
-    removals: 0,
-    opacityChanges: 0,
-    opacityImpact: 0,
-    unhoverables: 0
+  const data = {
+    totals: {
+      triggers: 0,
+      headTriggers: 0,
+      tailTriggers: 0,
+      impactTriggers: 0,
+      additions: 0,
+      removals: 0,
+      opacityChanges: 0,
+      opacityImpact: 0,
+      unhoverables: 0
+    }
   };
   // If details are to be reported:
   if (withItems) {
@@ -252,10 +256,10 @@ exports.reporter = async (
   const tailSample = tailSampleSize === -1 ? tailTriggers : getSample(tailTriggers, tailSampleSize);
   // Find and document the impacts.
   if (headSample.length) {
-    await find(withItems, page, 'head', headSample, headTriggerCount / headSample.length);
+    await find(data, withItems, page, 'head', headSample, headTriggerCount / headSample.length);
   }
   if (tailSample.length) {
-    await find(withItems, page, 'tail', tailSample, tailTriggerCount / tailSample.length);
+    await find(data, withItems, page, 'tail', tailSample, tailTriggerCount / tailSample.length);
   }
   // Round the reported totals.
   Object.keys(data.totals).forEach(key => {

package/tests/htmlcs.js

@@ -28,7 +28,7 @@ exports.reporter = async page => {
         }
         return issues;
       }, standard);
-      if (nextIssues) {
+      if (nextIssues && nextIssues.every(issue => typeof issue === 'string')) {
         messageStrings.push(... nextIssues);
       }
       else {

package/tests/ibm.js

@@ -104,7 +104,7 @@ exports.reporter = async (page, withItems, withNewContent) => {
     result.content = await doTest(typeContent, withItems, timeLimit);
     if (result.content.prevented) {
       result.prevented = true;
-      console.log('ERROR: Getting ibm test report from page took too long');
+      console.log(`ERROR: Getting ibm test report from page timed out at ${timeLimit} seconds`);
     }
   }
   // If a test with new content is to be performed:
@@ -114,7 +114,7 @@ exports.reporter = async (page, withItems, withNewContent) => {
     result.url = await doTest(typeContent, withItems, timeLimit);
     if (result.url.prevented) {
       result.prevented = true;
-      console.log('ERROR: Getting ibm test report from URL took too long');
+      console.log(`ERROR: Getting ibm test report from URL timed out at ${timeLimit} seconds`);
     }
   }
   await close();

package/tests/role.js

@@ -120,6 +120,264 @@ exports.reporter = async page => await page.$eval('body', body => {
     'treegrid',
     'treeitem',
   ]);
+  // Implicit roles
+  const implicitRoles = {
+    article: 'article',
+    aside: 'complementary',
+    button: 'button',
+    datalist: 'listbox',
+    dd: 'definition',
+    details: 'group',
+    dfn: 'term',
+    dialog: 'dialog',
+    dt: 'term',
+    fieldset: 'group',
+    figure: 'figure',
+    hr: 'separator',
+    html: 'document',
+    li: 'listitem',
+    main: 'main',
+    math: 'math',
+    menu: 'list',
+    nav: 'navigation',
+    ol: 'list',
+    output: 'status',
+    progress: 'progressbar',
+    summary: 'button',
+    SVG: 'graphics-document',
+    table: 'table',
+    tbody: 'rowgroup',
+    textarea: 'textbox',
+    tfoot: 'rowgroup',
+    thead: 'rowgroup',
+    tr: 'row',
+    ul: 'list'
+  };
+  const implicitAttributes = {
+    a: [
+      {
+        role: 'link',
+        attributes: {
+          href: /./
+        }
+      }
+    ],
+    area: [
+      {
+        role: 'link',
+        attributes: {
+          href: /./
+        }
+      }
+    ],
+    h1: [
+      {
+        role: 'heading',
+        attributes: {
+          'aria-level': /^1$/
+        }
+      }
+    ],
+    h2: [
+      {
+        role: 'heading',
+        attributes: {
+          'aria-level': /^2$/
+        }
+      },
+      {
+        role: 'heading',
+        attributes: {
+          'aria-level': false
+        }
+      }
+    ],
+    h3: [
+      {
+        role: 'heading',
+        attributes: {
+          'aria-level': /^3$/
+        }
+      }
+    ],
+    h4: [
+      {
+        role: 'heading',
+        attributes: {
+          'aria-level': /^4$/
+        }
+      }
+    ],
+    h5: [
+      {
+        role: 'heading',
+        attributes: {
+          'aria-level': /^5$/
+        }
+      }
+    ],
+    h6: [
+      {
+        role: 'heading',
+        attributes: {
+          'aria-level': /^6$/
+        }
+      }
+    ],
+    input: [
+      {
+        role: 'checkbox',
+        attributes: {
+          type: /^checkbox$/
+        }
+      },
+      {
+        role: 'button',
+        attributes: {
+          type: /^(?:button|image|reset|submit)$/
+        }
+      },
+      {
+        role: 'combobox',
+        attributes: {
+          type: /^(?:email|search|tel|text|url)$/,
+          list: true
+        }
+      },
+      {
+        role: 'combobox',
+        attributes: {
+          type: false,
+          list: true
+        }
+      },
+      {
+        role: 'radio',
+        attributes: {
+          type: /^radio$/
+        }
+      },
+      {
+        role: 'searchbox',
+        attributes: {
+          type: /^search$/,
+          list: false
+        }
+      },
+      {
+        role: 'slider',
+        attributes: {
+          type: /^range$/
+        }
+      },
+      {
+        role: 'spinbutton',
+        attributes: {
+          type: /^number$/
+        }
+      },
+      {
+        role: 'textbox',
+        attributes: {
+          type: /^(?:email|tel|text|url)$/,
+          list: false
+        }
+      },
+      {
+        role: 'textbox',
+        attributes: {
+          type: false,
+          list: false
+        }
+      },
+      {
+        role: 'checkbox',
+        attributes: {
+          type: /^checkbox$/
+        }
+      },
+      {
+        role: 'checkbox',
+        attributes: {
+          type: /^checkbox$/
+        }
+      },
+    ],
+    img: [
+      {
+        role: 'presentation',
+        attributes: {
+          alt: /^$/
+        }
+      },
+      {
+        role: 'img',
+        attributes: {
+          alt: /./
+        }
+      },
+      {
+        role: 'img',
+        attributes: {
+          alt: false
+        }
+      }
+    ],
+    select: [
+      {
+        role: 'listbox',
+        attributes: {
+          multiple: true
+        }
+      },
+      {
+        role: 'listbox',
+        attributes: {
+          size: /^(?:[2-9]|[1-9]\d+)$/
+        }
+      },
+      {
+        role: 'combobox',
+        attributes: {
+          multiple: false,
+          size: false
+        }
+      },
+      {
+        role: 'combobox',
+        attributes: {
+          multiple: false,
+          size: /^1$/
+        }
+      }
+    ]
+  };
+  // Array of th and td elements with redundant roles.
+  const redundantCells = [];
+  // FUNCTIONS
+  const dataInit = (data, tagName, role) => {
+    if (! data.tagNames[tagName]) {
+      data.tagNames[tagName] = {};
+    }
+    if (! data.tagNames[tagName][role]) {
+      data.tagNames[tagName][role] = {
+        bad: 0,
+        redundant: 0
+      };
+    }
+  };
+  const tallyTableRedundancy = (elements, okRoles, tagName) => {
+    elements.forEach(element => {
+      const role = element.getAttribute('role');
+      if (okRoles.includes(role)) {
+        dataInit(data, tagName, role);
+        data.redundantRoleElements++;
+        data.tagNames[tagName][role].redundant++;
+        redundantCells.push(element);
+      }
+    });
+  };
+  // OPERATION
   // Remove the deprecated roles from the non-abstract roles.
   goodRoles.forEach(role => {
     if (badRoles.has(role)) {
@@ -128,34 +386,84 @@ exports.reporter = async page => await page.$eval('body', body => {
   });
   // Identify all elements with role attributes.
   const roleElements = Array.from(body.querySelectorAll('[role]'));
-  // Identify those with roles that are either deprecated or invalid.
-  const bads = roleElements.filter(element => {
-    const role = element.getAttribute('role');
-    return badRoles.has(role) || ! goodRoles.has(role);
-  });
   // Initialize the result.
   const data = {
     roleElements: roleElements.length,
-    badRoleElements: bads.length,
+    badRoleElements: 0,
+    redundantRoleElements: 0,
     tagNames: {}
   };
-  // For each element with a deprecated role:
-  bads.forEach(element => {
-    // Identify its facts.
-    const tagName = element.tagName;
+  // Identify the th and td elements with redundant roles.
+  const gridHeaders = Array.from(
+    document.body.querySelectorAll('table[role=grid] th, table[role=treegrid] th')
+  );
+  const gridCells = Array.from(
+    document.body.querySelectorAll('table[role=grid] td, table[role=treegrid] td')
+  );
+  const tableHeaders = Array.from(
+    document.body.querySelectorAll('table[role=table] th, table:not([role]) th')
+  );
+  const tableCells = Array.from(
+    document.body.querySelectorAll('table[role=table] td, table:not([role]) td')
+  );
+  tallyTableRedundancy(gridHeaders, ['columnheader', 'rowheader', 'gridcell'], 'TH');
+  tallyTableRedundancy(gridCells, ['gridcell'], 'TD');
+  tallyTableRedundancy(tableHeaders, ['columnheader', 'rowheader', 'cell'], 'TH');
+  tallyTableRedundancy(tableCells, ['cell'], 'TD');
+  // Identify the additional elements with redundant roles and bad roles.
+  roleElements.filter(element => ! redundantCells.includes(element)).forEach(element => {
     const role = element.getAttribute('role');
-    // Add them to the result.
-    if (data.tagNames[tagName]) {
-      if (data.tagNames[tagName][role]) {
-        data.tagNames[tagName][role]++;
+    const tagName = element.tagName;
+    // If the role is not absolutely valid:
+    if (! goodRoles.has(role)) {
+      // If it is bad or redundant:
+      if (badRoles.has(role)) {
+        dataInit(data, tagName, role);
+        const lcTagName = tagName.toLowerCase();
+        // If it is simply redundant:
+        if (role === implicitRoles[lcTagName]) {
+          data.redundantRoleElements++;
+          data.tagNames[tagName][role].redundant++;
+        }
+        // Otherwise, if it is attributionally redundant:
+        else if (
+          implicitAttributes[lcTagName] && implicitAttributes[lcTagName].some(
+            criterion => role === criterion.role && Object.keys(criterion.attributes).every(
+              attributeName => {
+                const rule = criterion.attributes[attributeName];
+                const exists = element.hasAttribute(attributeName);
+                const value = exists ? element.getAttribute(attributeName) : null;
+                if (rule === true) {
+                  return exists;
+                }
+                else if (rule === false) {
+                  return ! exists;
+                }
+                else {
+                  return rule.test(value);
+                }
+              }
+            )
+          )
+        ) {
+          data.redundantRoleElements++;
+          data.tagNames[tagName][role].redundant++;
+        }
+        // Otherwise, i.e. if it is absolutely invalid:
+        else {
+          data.badRoleElements++;
+          data.tagNames[tagName][role].bad++;
+        }
       }
+      // Otherwise, i.e. if it is absolutely invalid:
       else {
-        data.tagNames[tagName][role] = 1;
+        // Add the facts to the result.
+        data.badRoleElements++;
+        dataInit(data, tagName, role);
+        data.tagNames[tagName][role].bad++;
       }
     }
-    else {
-      data.tagNames[tagName] = {[role]: 1};
-    }
   });
+  // Return the result.
   return {result: data};
 });

package/tests/tabNav.js

@@ -103,7 +103,9 @@ exports.reporter = async (page, withItems) => {
         });
       })
       .catch(error => {
-        console.log(`ERROR: could not click tab element ${itemData.text} (${error.message})`);
+        console.log(
+          `ERROR clicking tab element ${itemData.text} (${error.message.replace(/\n.+/s, '')})`
+        );
       pressed = false;
       });
       // Increment the counts of navigations and key navigations.

package/validation/executors/debug.js

@@ -0,0 +1,62 @@
+// app.js
+// Validator for Testaro tests.
+
+const fs = require('fs').promises;
+const {handleRequest} = require(`${__dirname}/../../run`);
+const validateTests = async () => {
+  const totals = {
+    attempts: 0,
+    successes: 0
+  };
+  const scriptFileNames = await fs.readdir(`${__dirname}/../tests/scripts`);
+  for (const scriptFileName of ['hover.json']) {
+    const rawScriptJSON = await fs
+    .readFile(`${__dirname}/../tests/scripts/${scriptFileName}`, 'utf8');
+    const scriptJSON = rawScriptJSON
+    .replace(/__targets__/g, `file://${__dirname}/../tests/targets`);
+    const script = JSON.parse(scriptJSON);
+    const report = {script};
+    report.log = [];
+    report.acts = [];
+    await handleRequest(report);
+    const {log, acts} = report;
+    if (log.length === 2 && log[1].event === 'endTime' && /^\d{4}-.+$/.test(log[1].value)) {
+      console.log('Success: Log has been correctly populated');
+    }
+    else {
+      console.log('Failure: Log empty or invalid');
+      console.log(JSON.stringify(log, null, 2));
+    }
+    if (
+      acts.length === script.commands.length
+      && acts.every(
+        act => act.type && act.type === 'test'
+          ? act.result && act.result.failureCount !== undefined
+          : true
+      )
+    ) {
+      totals.attempts++;
+      totals.successes++;
+      console.log('Success: Reports have been correctly populated');
+      if (acts.every(
+        act => act.type === 'test' ? act.result.failureCount === 0 : true
+      )) {
+        totals.attempts++;
+        totals.successes++;
+        console.log('Success: No failures');
+      }
+      else {
+        totals.attempts++;
+        console.log('Failure: At least one test has at least one failure');
+        console.log(JSON.stringify(acts, null, 2));
+      }
+    }
+    else {
+      totals.attempts++;
+      console.log('Failure: Reports empty or invalid');
+      console.log(JSON.stringify(acts, null, 2));
+    }
+  }
+  console.log(`Grand totals: attempts ${totals.attempts}, successes ${totals.successes}`);
+};
+validateTests();

package/validation/tests/scripts/hover.json

@@ -16,12 +16,19 @@
       "type": "test",
       "which": "hover",
       "what": "hover",
-      "withItems": false,
+      "headSize": 3,
+      "headSampleSize": 3,
+      "tailSampleSize": 30,
+      "withItems": true,
       "expect": [
-        ["totals.triggers", "=", 0],
-        ["totals.madeVisible", "=", 0],
-        ["totals.opacityChanged", "=", 0],
-        ["totals.opacityAffected", "=", 0],
+        ["totals.triggers", "=", 2],
+        ["totals.headTriggers", "=", 2],
+        ["totals.tailTriggers", "=", 0],
+        ["totals.impactTriggers", "=", 0],
+        ["totals.additions", "=", 0],
+        ["totals.removals", "=", 0],
+        ["totals.opacityChanges", "=", 0],
+        ["totals.opacityImpact", "=", 0],
         ["totals.unhoverables", "=", 0]
       ]
     },
@@ -34,12 +41,19 @@
       "type": "test",
       "which": "hover",
       "what": "hover",
-      "withItems": false,
+      "headSize": 3,
+      "headSampleSize": 3,
+      "tailSampleSize": 30,
+      "withItems": true,
       "expect": [
-        ["totals.triggers", "=", 2],
-        ["totals.madeVisible", "=", 2],
-        ["totals.opacityChanged", "=", 1],
-        ["totals.opacityAffected", "=", 2],
+        ["totals.triggers", "=", 4],
+        ["totals.headTriggers", "=", 3],
+        ["totals.tailTriggers", "=", 1],
+        ["totals.impactTriggers", "=", 2],
+        ["totals.additions", "=", 3],
+        ["totals.removals", "=", 0],
+        ["totals.opacityChanges", "=", 1],
+        ["totals.opacityImpact", "=", 1],
         ["totals.unhoverables", "=", 1]
       ]
     }

package/validation/tests/scripts/role.json

@@ -18,7 +18,8 @@
       "what": "role",
       "expect": [
         ["roleElements", "=", 1],
-        ["badRoleElements", "=", 0]
+        ["badRoleElements", "=", 0],
+        ["redundantRoleElements", "=", 0]
       ]
     },
     {
@@ -31,11 +32,19 @@
       "which": "role",
       "what": "role",
       "expect": [
-        ["roleElements", "=", 3],
-        ["badRoleElements", "=", 3],
-        ["tagNames.SECTION.main", "=", 1],
-        ["tagNames.SECTION.section", "=", 1],
-        ["tagNames.H2.heading", "=", 1]
+        ["roleElements", "=", 9],
+        ["badRoleElements", "=", 6],
+        ["redundantRoleElements", "=", 3],
+        ["tagNames.SECTION.main.bad", "=", 1],
+        ["tagNames.SECTION.section.bad", "=", 1],
+        ["tagNames.H2.heading.bad", "=", 0],
+        ["tagNames.H2.heading.redundant", "=", 1],
+        ["tagNames.H3.heading.bad", "=", 2],
+        ["tagNames.H3.heading.redundant", "=", 1],
+        ["tagNames.INPUT.spinbutton.redundant", "=", 1],
+        ["tagNames.INPUT.textbox.bad", "=", 1],
+        ["tagNames.INPUT.textbox.redundant", "=", 0],
+        ["tagNames.INPUT.combobox.bad", "=", 1]
       ]
     }
   ]

package/validation/tests/targets/hover/bad.html

@@ -9,10 +9,10 @@
   <body>
     <main>
       <h1>Page with deviant hover behavior</h1>
-      <p>This page contains a link to <a href="https://en.wikipedia.org" onmouseover="document.getElementById('translucent').style.opacity = 1">information</a>, a <button type="button" onmouseover="document.getElementById('hiddenP').style.display = 'block'">button</button>, and a <label>text input <input type="text"></label>. All three elements can be hovered over.</p>
+      <p>This page contains a link to <a href="https://en.wikipedia.org" onmouseover="document.getElementById('translucent').style.opacity = 1">information</a> and a <button type="button" onmouseover="document.getElementById('hiddenP').style.display = 'block'">bothersome button</button>. Both elements can be hovered over.</p>
       <p id="hiddenP" style="display: none">The button, when hovered over, makes this paragraph, and therefore this <a href="https://en.wikipedia.org/wiki/Web_accessibility">link on web accessibility</a> and this <button type="button">new button</button>, visible.</p>
       <p id="translucent" style="opacity: 0.4">The first link, when hovered over, changes the opacity of this paragraph from 0.4 to 1. That indirectly changes the opacity of this <span>word</span>, too.</p>
-      <p>The small button is mostly covered by a large one below.</p>
+      <p>The small button is mostly covered by a large one here, preventing the small button from receiving a hover event.</p>
       <p style="position: relative"><button style="position: absolute; left: 10rem">button</button><button style="position: absolute; left: 11rem; top: -0.5rem; font-size: x-large">bigger button</button></p>
     </main>
   </body>

package/validation/tests/targets/hover/good.html

@@ -9,7 +9,7 @@
   <body>
     <main>
       <h1>Page with standard hover behavior</h1>
-      <p>This paragraph contains a link to <a href="https://en.wikipedia.org">information</a>, a <button type="button">button</button>, and a <label>text input <input type="text"></label>. All three can be hovered over, and hovering over any of them does not trigger any change in content.</p>
+      <p>This paragraph contains a link to <a href="https://en.wikipedia.org">information</a> and a <button type="button">button</button>. Both of them can be hovered over, and hovering over either of them does not trigger any change in content.</p>
     </main>
   </body>
 </html>

package/validation/tests/targets/role/bad.html

@@ -11,15 +11,37 @@
       <h1>Page with deviant role elements</h1>
       <section role="section">
         <h2>Abstraction</h2>
-        <p>This section has an abstract role.</p>
+        <p>This section has an abstract role, so it is bad.</p>
       </section>
       <section>
         <h2 role="heading">Redundancy</h2>
-        <p>This section has a redundant role.</p>
+        <p>The heading of this section has a redundant role, with an inferred level.</p>
+        <section>
+          <h3 role="heading" aria-level="3">Redundancy with explicit level</h3>
+          <p>The heading of this section has a redundant role because the stated level is 3 and the implicit level is 3.</p>
+        </section>
+        <section>
+          <h3 role="heading">Failed redundancy for missing level</h3>
+          <p>The heading of this section has a heading that fails redundancy because the inferred level is 2 but the explicit level is 3. So the element role is bad.</p>
+        </section>
+        <section>
+          <h3 role="heading" aria-level="4">Failed redundancy for wrong level</h3>
+          <p>The heading of this section has a heading that fails redundancy because the attributional level is 4 but the explicit level is 3. So the element role is bad.</p>
+        </section>
       </section>
       <section>
         <h2>Conflict</h2>
-        <p>The parent section of these sections has an unnecessary explicit role, instead of an element with the same implicit role.</p>
+        <p>The parent section of these sections has an unnecessary explicit role <code>main</code>, instead of a <code>main</code> element with the same implicit role.</p>
+      </section>
+      <section>
+        <h2>Attributes</h2>
+        <h3>Valid redundancy</h3>
+        <p>This paragraph contains an input for a number. It has an implicit <code>spinbutton</code> role and the same redundant explicit role. <input type="number" role="spinbutton"></p>
+        <h3>Failed redundancy</h3>
+        <p>This paragraph contains an input for a number. It has an implicit <code>spinbutton</code> role but an explicit <code>textbox</code> role, so its role is bad. <input type="number" role="textbox"></p>
+        <h3>Attribute existence</h3>
+        <datalist id="options"><option value="a"></option><option value="b"></option></datalist>
+        <p>This input is identical, except that it omits the <code>list</code> attribute. That makes its implicit role <code>textbox</code>, so the explicit role of <code>combobox</code> is bad. <input role="combobox"></p>
       </section>
     </section>
   </body>