testaro 14.6.2 → 14.7.0

package/README.md

@@ -65,7 +65,7 @@ As of this version, the counts of tests of the tools referenced above were:
 - Testaro: 29
 - total: 1356
 
-Of the 29 Testaro tests, 26 are evaluative (they discover accessibility issues), and the other 3 (`elements`, `textNodes`, and `title`) are informative (they report conditions specified by the user).
+Of the 29 Testaro tests, 26 are evaluative (they discover accessibility problems), and the other 3 (`elements`, `textNodes`, and `title`) are informative (they report conditions specified by the user).
 
 ## Quasi-tests
 
@@ -195,21 +195,21 @@ While each tool produces a _tool report_ of the results of its tests, Testaro al
 
 ##### Tool formats
 
-The tools listed above as dependencies write their tool reports in various formats. They differ in how they organize multiple instances of the same issue, how they classify issue severity and certainty, how they point to the locations of issue instances, how they name issues, etc.
-
-Integrating reports from 10 different tools is a complex task, as analyzed in [“Accessibility Metatesting”](https://arxiv.org/abs/2304.07591). The diversity of their reporting formats makes the task more complex than it would otherwise be.
+The tools listed above as dependencies write their tool reports in various formats. They differ in how they organize multiple instances of the same problem, how they classify severity and certainty, how they point to the locations of problems, how they name problems, etc.
 
 ##### Standard format
 
 Testaro helps overcome this format diversity by offering to represent the main facts in the report of each tool in a single standardized format.
 
+In the conceptual scheme underlying the format standardization of Testaro, each tool has its own set of _rules_, where a rule is an algorithm for evaluating a target and determining whether instances of some kind of problem exist in it. With standardization, Testaro reports, in a uniform way, the outcomes from the application of rules by tools to a target.
+
 If the `STANDARD` environment variable has the value `also` (which it has by default) or `only`, Testaro converts some data in each tool report to a standard format. That permits you to ignore the format idiosyncrasies of the tools. If `STANDARD` has the value `also`, the job report includes both formats. If the value is `only`, the job report includes only the standard format. If the value is `no`, the job report includes only the original format of each tool.
 
 The standard format of each tool report has two properties:
-- `totals`: an array of 4 integers, representing the counts of issue instances classified by the tool into 4 ordinal degrees of severity. For example, `[2, 13, 0, 5]` would mean that the tool reported 2 instances at the lowest severity, 13 at the next-lowest, none at the third-lowest, and 5 at the highest.
+- `totals`: an array of 4 integers, representing the counts of problem instances classified by the tool into 4 ordinal degrees of severity. For example, `[2, 13, 0, 5]` would mean that the tool reported 2 instances at the lowest severity, 13 at the next-lowest, none at the third-lowest, and 5 at the highest.
 - `instances`: an array of objects describing facts about issue instances reported by the tool. This object has these properties, some of which have empty strings as values when the tool does not provide values:
-    - `issueID`: a code identifying the issue
-    - `what`: a description of the issue
+    - `ruleID`: a code identifying a rule
+    - `what`: a description of the rule
     - `count` (optional): the count of instances if this instance represents multiple instances
     - `ordinalSeverity`: how the tool ranks the severity of the instance, on a 4-point ordinal scale
     - `tagName`: upper-case tagName of the affected element
@@ -227,7 +227,7 @@ standardResult: {
   totals: [2, 0, 17, 0],
   instances: [
     {
-      issueID: 'rule01',
+      ruleID: 'rule01',
       what: 'Button type invalid',
       ordinalSeverity: 0,
       tagName: 'BUTTON'
@@ -240,7 +240,7 @@ standardResult: {
       excerpt: '<button type="link"></button>'
     },
     {
-      issueID: 'rule01',
+      ruleID: 'rule01',
       what: 'Button type invalid',
       ordinalSeverity: 1,
       tagName: 'BUTTON',
@@ -253,7 +253,7 @@ standardResult: {
       excerpt: '<button type="important">Submit</button>'
     },
     {
-      issueID: 'rule02',
+      ruleID: 'rule02',
       what: 'Links have empty href attributes',
       count: 17,
       ordinalSeverity: 3,
@@ -272,7 +272,7 @@ standardResult: {
 
 If a tool has the option to be used without itemization and is being so used, the `instances` array may be empty.
 
-The standard format is not opinionated about issue classifications. It treats an issue ID from any tool as an identifier of what that tool considers to be the issue. Useful reporting from multi-tool testing still requires issue classification. If tool `A` identifies an issue as `alt-incomplete` and tool `B` identifies an issue as `image_alt_meaningless`, Testaro does not decide whether those are really the same issue or different issues. That decision belongs to whoever consumes Testaro reports. The standardization of tool reports by Testaro eliminates some of the drudgery in issue classification, but not any of the judgment required for issue classification.
+This standard format is not opinionated about issue classifications. A rule ID identifies something deemed to be an issue by a tool. Useful reporting from multi-tool testing still requires the classification of tool **rules** into **issues**. If tool `A` has `alt-incomplete` as a rule ID and tool `B` has `image_alt_stub` as a rule ID, Testaro does not decide whether those are really the same issue or different issues. That decision belongs to you. The standardization of tool reports by Testaro eliminates some of the drudgery in issue classification, but not any of the judgment required for issue classification.
 
 ### Acts
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "14.6.2",
+  "version": "14.7.0",
   "description": "Automation of accessibility testing",
   "main": "index.js",
   "scripts": {

package/standardize.js

@@ -22,10 +22,15 @@ const cap = rawString => {
 const getIdentifiers = code => {
   let tagName = '';
   let id = '';
+  // If the substring includes a tag of an element:
   if (code && typeof code === 'string' && code.length && /<.+/s.test(code)) {
-    const startTag = code.replace(/^[^<]*<|>.*/sg, '').trim();
+    // Get the first start tag in the substring.
+    const startTag = code.replace(/^.*?<(?!\/)/s, '').replace(/>.*$/s, '').trim();
+    // If it exists:
     if (startTag && startTag.length) {
-      tagName = startTag.replace(/\s.+$/s, '').toUpperCase();
+      // Get its tag name, upper-cased.
+      tagName = startTag.replace(/\s.*$/s, '').toUpperCase();
+      // Get the value of the id attribute of the start tag, if any.
       const idArray = startTag.match(/\sid="([^"<>]+)"/);
       if (idArray && idArray.length === 2) {
         id = idArray[1];
@@ -53,7 +58,7 @@ const doAxe = (result, standardResult, certainty) => {
         const ordinalSeverity = severityWeights[node.impact] + (certainty === 'violations' ? 2 : 0);
         const identifiers = getIdentifiers(node.html);
         const instance = {
-          issueID: rule.id,
+          ruleID: rule.id,
           what: Array.from(whatSet.values()).join('; '), 
           ordinalSeverity,
           tagName: identifiers[0],
@@ -79,7 +84,7 @@ const doHTMLCS = (result, standardResult, severity) => {
         ruleData[what].forEach(item => {
           const {tagName, id, code} = item;
           const instance = {
-            issueID: ruleID,
+            ruleID,
             what,
             ordinalSeverity: ['Warning', '', '', 'Error'].indexOf(severity),
             tagName: tagName.toUpperCase(),
@@ -113,7 +118,7 @@ const doNuVal = (result, standardResult, docType) => {
       }
       // Include the message twice, because in scoring it is likely to be replaced by a pattern.
       const instance = {
-        issueID: item.message,
+        ruleID: item.message,
         what: item.message,
         ordinalSeverity: -1,
         tagName: identifiers[0],
@@ -154,14 +159,14 @@ const doQualWeb = (result, standardResult, ruleClassName) => {
         failed: 3
       }
     };
-    Object.keys(ruleClass.assertions).forEach(rule => {
-      const ruleResult = ruleClass.assertions[rule];
+    Object.keys(ruleClass.assertions).forEach(ruleID => {
+      const ruleResult = ruleClass.assertions[ruleID];
       ruleResult.results.forEach(item => {
         item.elements.forEach(element => {
           const {htmlCode} = element;
           const identifiers = getIdentifiers(htmlCode);
           const instance = {
-            issueID: rule,
+            ruleID,
             what: ruleResult.description,
             ordinalSeverity: severities[ruleClassName][item.verdict],
             tagName: identifiers[0],
@@ -185,8 +190,8 @@ const doWAVE = (result, standardResult, categoryName) => {
   if (result.categories && result.categories[categoryName]) {
     const category = result.categories[categoryName];
     const ordinalSeverity = categoryName === 'alert' ? 0 : 3;
-    Object.keys(category.items).forEach(rule => {
-      category.items[rule].selectors.forEach(selector => {
+    Object.keys(category.items).forEach(ruleID => {
+      category.items[ruleID].selectors.forEach(selector => {
         let tagName = '';
         let id = '';
         if (typeof selector === 'string') {
@@ -201,8 +206,8 @@ const doWAVE = (result, standardResult, categoryName) => {
           }
         }
         const instance = {
-          issueID: rule,
-          what: category.items[rule].description,
+          ruleID,
+          what: category.items[ruleID].description,
           ordinalSeverity,
           tagName,
           id,
@@ -234,16 +239,17 @@ const convert = (toolName, result, standardResult) => {
           tagName = tagNameArray[1];
         }
       }
+      const {rule, target} = item;
       const instance = {
-        issueID: item.rule.ruleID,
-        what: item.rule.ruleSummary,
+        ruleID: rule.ruleID,
+        what: rule.ruleSummary,
         ordinalSeverity: ['cantTell', '', '', 'failed'].indexOf(item.verdict),
         tagName: tagName.toUpperCase() || identifiers[0],
         id: identifiers[1],
         location: {
           doc: 'dom',
           type: 'xpath',
-          spec: item.target.path
+          spec: target.path
         },
         excerpt: cap(code)
       };
@@ -281,7 +287,7 @@ const convert = (toolName, result, standardResult) => {
         }
       }
       const instance = {
-        issueID: item.engineTestId.toString(),
+        ruleID: item.engineTestId.toString(),
         what: item.attributeDetail,
         ordinalSeverity: 3,
         tagName,
@@ -320,7 +326,7 @@ const convert = (toolName, result, standardResult) => {
         }
       }
       const instance = {
-        issueID: item.ruleId,
+        ruleID: item.ruleId,
         what: item.message,
         ordinalSeverity: ['', 'recommendation', '', 'violation'].indexOf(item.level),
         tagName: identifiers[0],
@@ -385,7 +391,7 @@ const convert = (toolName, result, standardResult) => {
         }
       }
       const instance = {
-        issueID: item.tID ? item.tID.toString() : '',
+        ruleID: item.tID ? item.tID.toString() : '',
         what: item.errorTitle || '',
         ordinalSeverity: Math.min(
           3, Math.max(0, Math.round((item.certainty || 0) * (item.priority || 0) / 3333))

package/testaro/focInd.js

@@ -27,15 +27,15 @@ exports.reporter = async (page, withItems, revealAll = false, allowedDelay = 250
       totals: {
         total: 0,
         types: {
-          indicatorMissing: {
+          missing: {
             total: 0,
             tagNames: {}
           },
-          nonOutlinePresent: {
+          nonoutline: {
             total: 0,
             tagNames: {}
           },
-          outlinePresent: {
+          outline: {
             total: 0,
             meanDelay: 0,
             tagNames: {}
@@ -45,16 +45,16 @@ exports.reporter = async (page, withItems, revealAll = false, allowedDelay = 250
     };
     if (withItems) {
       data.items = {
-        indicatorMissing: [],
-        nonOutlinePresent: [],
-        outlinePresent: []
+        missing: [],
+        nonoutline: [],
+        outline: []
       };
     }
     // Adds facts about an element to the result.
     const addElementFacts = (element, status, delay = null) => {
       const type = data.totals.types[status];
       type.total++;
-      if (status === 'outlinePresent') {
+      if (status === 'outline') {
         type.meanDelay = Math.round(((type.total - 1) * type.meanDelay + delay) / type.total);
       }
       const tagName = element.tagName;
@@ -71,7 +71,7 @@ exports.reporter = async (page, withItems, revealAll = false, allowedDelay = 250
           text: (element.textContent.trim() || element.outerHTML.trim()).replace(/\s+/g, ' ')
           .slice(0, 100)
         };
-        if (status === 'outlinePresent') {
+        if (status === 'outline') {
           elementData.delay = delay;
         }
         data.items[status].push(elementData);
@@ -111,7 +111,7 @@ exports.reporter = async (page, withItems, revealAll = false, allowedDelay = 250
           // If a non-transparent outline appeared immediately on focus:
           if (styleDec.outlineWidth !== '0px' && styleDec.outlineColor !== 'rgba(0, 0, 0, 0)') {
             // Add facts about the element to the result.
-            addElementFacts(element, 'outlinePresent', 0);
+            addElementFacts(element, 'outline', 0);
             hasOutline = true;
           }
           // Otherwise, if a wait for an outline is allowed:
@@ -137,7 +137,7 @@ exports.reporter = async (page, withItems, revealAll = false, allowedDelay = 250
             const delay = await outlineDelay;
             if (delay) {
               // Add facts about the element to the result.
-              addElementFacts(element, 'outlinePresent', delay);
+              addElementFacts(element, 'outline', delay);
               hasOutline = true;
             }
           }
@@ -158,7 +158,7 @@ exports.reporter = async (page, withItems, revealAll = false, allowedDelay = 250
           || hasDiffBorder
           || indicatorStyleNames.slice(5).reduce((any, styleName) => any || diff(styleName), false);
           // Add the determination to the result.
-          const status = hasIndicator ? 'nonOutlinePresent' : 'indicatorMissing';
+          const status = hasIndicator ? 'nonoutline' : 'missing';
           addElementFacts(element, status);
         }
       }
@@ -166,17 +166,17 @@ exports.reporter = async (page, withItems, revealAll = false, allowedDelay = 250
     return data;
   }, [allowedDelay, withItems]);
   const {types} = data.totals;
-  const totals = [0, 0, types.nonOutlinePresent.total, types.indicatorMissing.total];
+  const totals = [0, 0, types.nonoutline.total, types.missing.total];
   const standardInstances = [];
   if (data.items) {
-    const issueNames = ['nonOutlinePresent', 'indicatorMissing'];
+    const issueNames = ['nonoutline', 'missing'];
     issueNames.forEach(issueName => {
       data.items[issueName].forEach(item => {
-        const qualifier = issueName === 'nonOutlinePresent' ? 'a non-outline' : 'no';
+        const qualifier = issueName === 'nonoutline' ? 'a non-outline' : 'no';
         standardInstances.push({
           issueID: `focInd-${issueName}`,
           what: `${item.tagName} element has ${qualifier} focus indicator`,
-          ordinalSeverity: issueName === 'nonOutlinePresent' ? 2 : 3,
+          ordinalSeverity: issueName === 'nonoutline' ? 2 : 3,
           tagName: item.tagName,
           id: item.id,
           location: {
@@ -190,11 +190,11 @@ exports.reporter = async (page, withItems, revealAll = false, allowedDelay = 250
     });
   }
   else {
-    if (types.indicatorMissing.total) {
+    if (types.missing.total) {
       standardInstances.push({
         issueID: 'focInd-missing',
         what: 'Elements have missing focus indicators',
-        count: types.indicatorMissing.total,
+        count: types.missing.total,
         ordinalSeverity: 3,
         tagName: '',
         id: '',
@@ -206,11 +206,11 @@ exports.reporter = async (page, withItems, revealAll = false, allowedDelay = 250
         excerpt: ''
       });
     }
-    if (types.nonOutlinePresent.total) {
+    if (types.nonoutline.total) {
       standardInstances.push({
         issueID: 'focInd-nonoutline',
         what: 'Elements have non-outline focus indicators',
-        count: types.nonOutlinePresent.total,
+        count: types.nonoutline.total,
         ordinalSeverity: 2,
         tagName: '',
         id: '',