testaro 26.2.0 → 26.3.0

package/CONTRIBUTING.md

@@ -0,0 +1,73 @@
+# testaro
+
+Ensemble testing for web accessibility
+
+## Contributing
+
+Contributions to Testaro are welcome.
+
+### Testaro rule specification
+
+Testaro integrates 8 accessibility testing tools, and one of them is Testaro itself. The Testaro tool has defined 43 rules, and the other 7 tools have defined about 600 rules in the aggregate. Contributing a new Testaro rule can add value if the new rule will not merely duplicate one of the existing rules. The issue classification (`tic…`) files in the `procs` directory of the [Testilo](https://www.npmjs.com/package/testilo) package can help in this determination.
+
+If you determine that a new Testaro rule would be valuable, the first step is to specify it. This specification has the following parts:
+1. Adding an entry to the `evalRules` or `etcRules` object in the `tests/testaro.js` file.
+1. Adding a validation target directory to the `validation/tests/targets` directory.
+1. Adding at least one validation target to the target directory.
+1. Adding a validation job to the `validation/tests/jobs` directory.
+
+### Testaro rule validation
+
+All Testaro rules have validators, as mentioned above.
+
+The first step in creating a validator is to create at least one HTML file that will be tested against the new rule. A single `index.html` file may suffice. It should contain cases that will pass the test and cases that will fail the test. If appropriate, you can create multiple targets, such as `good.html` and `bad.html`.
+
+The second step is to create a validation job. It launches a browser, navigates to a validation target, and conducts the test. It includes expectations about the results. Typically, the expectations relate to the standard instances included in the results. The `Tests/Expectations` section of the `README.md` file describes the syntax of expectations.
+
+When a rule `xyz` has been defined and the `npm test xyz` statement is executed, the validation job will be run.
+
+### Rule creation
+
+Once you have specified a new rule, you, or somebody else, can develop the rule in accord with your specification. Developing the rule consists of:
+1. Creating a rule-definition file.
+1. Adding the file to the `testaro` directory.
+1. Validating it.
+
+For example, if the new rule has the ID `bigAlt`, it must succeed when the statement `npm test bigAlt` is executed.
+
+### Simple rules
+
+Some existing Testaro rules are simple enough to be defined in JSON files. If a new rule is similarly simple, you can define it with a similar JSON file.
+
+An example is the `titledEl` rule:
+
+```json
+{
+  "ruleID": "titledEl",
+  "selector": "[title]:not(input, button, textarea, select, iframe):visible",
+  "complaints": {
+    "instance": "Ineligible element has a title attribute",
+    "summary": "Ineligible elements have title attributes"
+  },
+  "ordinalSeverity": 2,
+  "summaryTagName": ""
+}
+```
+
+To create a simple rule, you can copy one of the existing JSON files and replace the values with values that are appropriate for the new rule. Testaro uses the properties in the file as follows:
+- Every element matching the selector is treated as a violator of the rule.
+- Testaro describes violations with one of the `complaints` values. If the user has requested itemized results, the `instance` value is used; otherwise the `summary` value is used.
+- Testaro assigns the `ordinalSeverity` value as the ordinal severity of each violation. This is a scale of integers, from 0 to 3.
+- When Testaro itemizes results, it reports the tag name of violating elements. When it only summarizes results, it includes the `summaryTagName`. If every violating element necessarily has one particular tag name, then you should make that (e.g., `BUTTON`) the value of `summaryTagName`.
+
+### Complex rules
+
+Your new rule may not be quite so simple. For example, it may need to examine the selected elements and identify only some of them as rule violators.
+
+You will need to define such a rule in a JavaScript file, rather than a JSON file.
+
+An example of such an existing rule is `filter`. The test for that rule initially selects all elements that descend from the `body` element (or a sample of 100 such elements in the page contains more than 100). Then it examines each selected element to determine whether it violates the rule. Specifically, it determines whether the element has a `filter` style property with a value other than `'none'`. It reports those violations.
+
+To define such a rule, you can copy an existing rule file and replace parts of its code as needed. Examples include:
+- `allSlanted`, `distortion`, `filter`, `miniText`, `zIndex`: violations based on element styles.
+- `linkTitle`: violations based on attributes and text content.

package/README.md

@@ -559,9 +559,9 @@ Thus, when the `rules` argument is omitted, QualWeb will test for all of the rul
 
 If you do not specify rules when using the `testaro` tool, Testaro will test for the rules listed in the `evalRules` object of the `tests/testaro.js` file.
 
-The `rules` property for `testaro` is an array whose first item is either `'y'` or `'n'` and whose remaining items are rule IDs. If `'y'`, then only the specified rules’ tests are performed. If `'n'`, then all the evaluative tests are performed, except for the specified rules.
+The `rules` argument for a `testaro` test act is an array whose first item is either `'y'` or `'n'` and whose remaining items are rule IDs. If `'y'`, then only the specified rules’ tests are performed. If `'n'`, then all the evaluative tests are performed, **except** for the specified rules.
 
-The `testaro` tool (like the `ibm` tool) has a `withItems` property. If you set it to `false`, the `standardResult` object of `testaro` will contain an `instances` property with summaries that identify issues and instance counts. If you set it to `true`, some of the instances will be itemized.
+The `testaro` tool (like the `ibm` tool) has a `withItems` property. If you set it to `false`, the `standardResult` object will contain an `instances` property with summaries that identify issues and instance counts. If you set it to `true`, some of the instances will be itemized.
 
 Unlike any other tool, the `testaro` tool requires a `stopOnFail` property, which specifies whether a failure to conform to any rule (i.e. any value of `totals` other than `[0, 0, 0, 0]`) should terminate the execution of tests for the remaining rules.
 
@@ -569,7 +569,7 @@ Warnings in the `testaro/hover.js`, `testaro/motion.js`, and `procs/visChange.js
 
 Several Testaro tests make use of the `init()` function in the `procs/testaro` module. That function samples elements if the population of elements to be tested is larger than 100. The purpose is to achieve reasonable performance. The sampling overweights elements near the beginning of a page, because of the tendency of that location to have important and atypical elements.
 
-You can add custom rules to the rules of any tool. Testaro provides a template, `data/template.js`, for the definition of a rule to be added. Once you have created a copy of the template with revisions, you can move the copy into the `testaro` directory and add an entry for your custom rule to the `evalRules` object in the `tests/testaro.js` file. Then your custom rule will act as a Testaro rule.
+You can add custom rules to the rules of any tool. Testaro provides a template, `data/template.js`, for the definition of a rule to be added. Once you have created a copy of the template with revisions, you can move the copy into the `testaro` directory and add an entry for your custom rule to the `evalRules` object in the `tests/testaro.js` file. Then your custom rule will act as a Testaro rule. Some `testaro` rules are simple enough to be fully specified in JSON files. You can use any of those as a template if you want to create a sufficiently simple custom rule, namely a rule whose prohibited elements are all and only the elements matching a CSS selector. More details about rule creation are in the `CONTRIBUTING.md` file.
 
 ###### WAVE
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "26.2.0",
+  "version": "26.3.0",
   "description": "Run 650 web accessibility tests from 8 tools and get a standardized report",
   "main": "index.js",
   "scripts": {

package/testaro/allCaps.js

@@ -15,7 +15,7 @@ const {init, report} = require('../procs/testaro');
 // Runs the test and returns the result.
 exports.reporter = async (page, withItems) => {
   // Initialize the locators and result.
-  const all = await init(page, 'body *:not(style):not(script):not(svg)');
+  const all = await init(page, 'body *:not(style, script, svg)');
   // For each locator:
   for (const loc of all.allLocs) {
     // Get whether its element violates the rule.

package/testaro/allSlanted.js

@@ -15,7 +15,7 @@ const {init, report} = require('../procs/testaro');
 // Runs the test and returns the result.
 exports.reporter = async (page, withItems) => {
   // Initialize the locators and result.
-  const all = await init(page, 'body *:not(style):not(script):not(svg)');
+  const all = await init(page, 'body *:not(style, script, svg)');
   // For each locator:
   for (const loc of all.allLocs) {
     // Get whether its element violates the rule.

package/testaro/buttonMenu.js

@@ -130,9 +130,7 @@ exports.reporter = async (page, withItems, trialKeySpecs = []) => {
         // If they were obtained:
         if (extraData) {
           // Get locators for its descendant non-menu menu items.
-          const miLocAll = menuLoc.locator(
-            '[role=menuitem]:not([role=menu]), [role=menuitem]:not([role=menubar])'
-          );
+          const miLocAll = menuLoc.locator('[role=menuitem]:not([role=menu], [role=menubar])');
           // Get which of them are direct descendants.
           const areDirect = await miLocAll.evaluateAll((els, menuID) => {
             return els.map(el => {

package/testaro/captionLoc.json

@@ -0,0 +1,10 @@
+{
+  "ruleID": "captionLoc",
+  "selector": "*:not(table) > caption, caption:not(:first-child)",
+  "complaints": {
+    "instance": "Element is not the first child of a table element",
+    "summary": "caption elements are not the first children of table elements"
+  },
+  "ordinalSeverity": 3,
+  "summaryTagName": "CAPTION"
+}

package/testaro/hover.js

@@ -23,7 +23,7 @@ exports.reporter = async (page, withItems) => {
   );
   const allNonTrigger = await init(
     page,
-    'body *:not([aria-controls]):not([aria-expanded]):not([aria-haspopup]):not([onmouseenter]):not([onmouseover])'
+    'body *:not([aria-controls], [aria-expanded], [aria-haspopup], [onmouseenter], [onmouseover])'
   );
   const populationSize
     = allTrigger.result.data.populationSize + allNonTrigger.result.data.populationSize;

package/testaro/hr.json

@@ -0,0 +1,10 @@
+{
+  "ruleID": "hr",
+  "selector": "body hr",
+  "complaints": {
+    "instance": "Element instead of styling is used for vertical segmentation",
+    "summary": "Elements instead of styling are used for vertical segmentation"
+  },
+  "ordinalSeverity": 0,
+  "summaryTagName": "HR"
+}

package/testaro/linkExt.json

@@ -0,0 +1,10 @@
+{
+  "ruleID": "linkExt",
+  "selector": "a[target=_blank]",
+  "complaints": {
+    "instance": "Link has a target=_blank attribute",
+    "summary": "Links have target=_blank attributes"
+  },
+  "ordinalSeverity": 0,
+  "summaryTagName": "A"
+}

package/testaro/linkTo.json

@@ -0,0 +1,10 @@
+{
+  "ruleID": "linkTo",
+  "selector": "a:not([href]):visible",
+  "complaints": {
+    "instance": "Link has no href attribute",
+    "summary": "Links are missing href attributes"
+  },
+  "ordinalSeverity": 2,
+  "summaryTagName": "A"
+}

package/testaro/nonTable.js

@@ -40,7 +40,7 @@ exports.reporter = async (page, withItems) => {
         // Return misuse.
         return true;
       }
-      // Otherwise, if it contains a table-compatible el:
+      // Otherwise, if it contains a table-compatible element:
       else if (
         el.caption
         || ['grid', 'treegrid'].includes(role)

package/testaro/pseudoP.js

@@ -18,8 +18,8 @@ exports.reporter = async (page, withItems) => {
   const all = await init(page, 'body *', {has: page.locator('br + br')});
   // For each locator:
   for (const loc of all.allLocs) {
-    // Get whether its element violates the rule.
     const isBad = await loc.evaluate(el => {
+      // Get whether it has 2 adjacent br elements with no non-space text between them.
       const childNodes = Array.from(el.childNodes);
       const realChildNodes = childNodes.filter(
         node => node.nodeType !== Node.TEXT_NODE || node.nodeValue.replace(/\s/g, '').length

package/testaro/titledEl.json

@@ -0,0 +1,10 @@
+{
+  "ruleID": "titledEl",
+  "selector": "[title]:not(input, button, textarea, select, iframe):visible",
+  "complaints": {
+    "instance": "Ineligible element has a title attribute",
+    "summary": "Ineligible elements have title attributes"
+  },
+  "ordinalSeverity": 2,
+  "summaryTagName": ""
+}

package/tests/testaro.js

@@ -3,6 +3,13 @@
   This test implements the Testaro evaluative rules.
 */
 
+// ######## IMPORTS
+
+// Module to perform common operations.
+const {init, report} = require('../procs/testaro');
+// Module to handle files.
+const fs = require('fs/promises');
+
 // ######## CONSTANTS
 
 const evalRules = {
@@ -12,6 +19,7 @@ const evalRules = {
   autocomplete: 'name and email inputs without autocomplete attributes',
   bulk: 'large count of visible elements',
   buttonMenu: 'nonstandard keyboard navigation between items of button-controlled menus',
+  captionLoc: 'caption elements that are not first children of table elements',
   distortion: 'distorted text',
   docType: 'document without a doctype property',
   dupAtt: 'elements with duplicate attributes',
@@ -27,6 +35,7 @@ const evalRules = {
   hovInd: 'hover indication nonstandard',
   hr: 'hr element instead of styles used for vertical segmentation',
   labClash: 'labeling inconsistencies',
+  legendLoc: 'legend elements that are not first children of fieldset elements',
   lineHeight: 'text with a line height less than 1.5 times its font size',
   linkExt: 'links that automatically open new windows',
   linkAmb: 'links with identical texts but different destinations',
@@ -37,6 +46,8 @@ const evalRules = {
   motion: 'motion without user request',
   nonTable: 'table elements used for layout',
   opFoc: 'Operable elements that are not Tab-focusable',
+  optRoleSel: 'Non-option elements with option roles that have no aria-selected attributes',
+  phOnly: 'input elements with placeholders but no accessible names',
   pseudoP: 'adjacent br elements suspected of nonsemantically simulating p elements',
   radioSet: 'radio buttons not grouped into standard field sets',
   role: 'invalid and native-replacing explicit roles',
@@ -53,8 +64,26 @@ const etcRules = {
   title: 'page title',
 };
 
-// FUNCTIONS
+// ######## FUNCTIONS
 
+// Conducts a JSON-defined test.
+const jsonTest = async (ruleID, ruleArgs) => {
+  const [page, withItems] = ruleArgs;
+  // Get the rule definition.
+  const ruleJSON = await fs.readFile(`${__dirname}/../testaro/${ruleID}.json`, 'utf8');
+  const ruleObj = JSON.parse(ruleJSON);
+  // Initialize the locators and result.
+  const all = await init(page, ruleObj.selector);
+  all.locs = all.allLocs;
+  // Populate and return the result.
+  const whats = [
+    ruleObj.complaints.instance,
+    ruleObj.complaints.summary
+  ];
+  return await report(
+    withItems, all, ruleObj.ruleID, whats, ruleObj.ordinalSeverity, ruleObj.summaryTagName
+  );
+};
 // Conducts and reports Testaro tests.
 exports.reporter = async (page, options) => {
   const {withItems, stopOnFail, args} = options;
@@ -80,42 +109,55 @@ exports.reporter = async (page, options) => {
     for (const rule of calledRules) {
       // Initialize an argument array.
       const ruleArgs = [page, withItems];
-      // If the rule has extra arguments:
-      if (argRules && argRules.includes(rule)) {
-        // Add them to the argument array.
-        ruleArgs.push(... args[rule]);
-      }
-      // Test the page.
-      const what = evalRules[rule] || etcRules[rule];
-      if (! data.rules[rule]) {
-        data.rules[rule] = {};
-      }
-      data.rules[rule].what = what;
-      console.log(`>>>>>> ${rule} (${what})`);
-      try {
-        const startTime = Date.now();
-        const ruleReport = await require(`../testaro/${rule}`).reporter(... ruleArgs);
-        // Add data from the test to the result.
-        const endTime = Date.now();
-        testTimes.push([rule, Math.round((endTime - startTime) / 1000)]);
-        Object.keys(ruleReport).forEach(key => {
-          data.rules[rule][key] = ruleReport[key];
-        });
-        data.rules[rule].totals = data.rules[rule].totals.map(total => Math.round(total));
-        if (ruleReport.prevented) {
-          data.preventions.push(rule);
+      // If the rule is defined with JavaScript or JSON but not both:
+      const ruleFileNames = await fs.readdir(`${__dirname}/../testaro`);
+      const isJS = ruleFileNames.includes(`${rule}.js`);
+      const isJSON = ruleFileNames.includes(`${rule}.json`);
+      if ((isJS || isJSON) && ! (isJS && isJSON)) {
+        // If with JavaScript and it has extra arguments:
+        if (isJS && argRules && argRules.includes(rule)) {
+          // Add them to the argument array.
+          ruleArgs.push(... args[rule]);
+        }
+        // Test the page.
+        const what = evalRules[rule] || etcRules[rule];
+        if (! data.rules[rule]) {
+          data.rules[rule] = {};
         }
-        // If testing is to stop after a failure and the page failed the test:
-        if (stopOnFail && ruleReport.totals.some(total => total)) {
-          // Stop testing.
-          break;
+        data.rules[rule].what = what;
+        console.log(`>>>>>> ${rule} (${what})`);
+        try {
+          const startTime = Date.now();
+          const ruleReport = isJS
+            ? await require(`../testaro/${rule}`).reporter(... ruleArgs)
+            : await jsonTest(rule, ruleArgs);
+          // Add data from the test to the result.
+          const endTime = Date.now();
+          testTimes.push([rule, Math.round((endTime - startTime) / 1000)]);
+          Object.keys(ruleReport).forEach(key => {
+            data.rules[rule][key] = ruleReport[key];
+          });
+          data.rules[rule].totals = data.rules[rule].totals.map(total => Math.round(total));
+          if (ruleReport.prevented) {
+            data.preventions.push(rule);
+          }
+          // If testing is to stop after a failure and the page failed the test:
+          if (stopOnFail && ruleReport.totals.some(total => total)) {
+            // Stop testing.
+            break;
+          }
+        }
+        // If an error is thrown by the test:
+        catch(error) {
+          // Report this.
+          data.preventions.push(rule);
+          console.log(`ERROR: Test of testaro rule ${rule} prevented (${error.message})`);
         }
       }
-      // If an error is thrown by the test:
-      catch(error) {
+      // Otherwise, i.e. if the rule is undefined or doubly defined:
+      else {
         // Report this.
-        data.preventions.push(rule);
-        console.log(`ERROR: Test of testaro rule ${rule} prevented (${error.message})`);
+        console.log(`ERROR: Rule ${rule} not validly defined`);
       }
     }
     testTimes.sort((a, b) => b[1] - a[1]).forEach(pair => {

package/validation/tests/jobs/captionLoc.json

@@ -0,0 +1,140 @@
+{
+  "id": "captionLoc",
+  "what": "validation of captionLoc test",
+  "strict": true,
+  "timeLimit": 20,
+  "acts": [
+    {
+      "type": "launch",
+      "which": "chromium",
+      "url": "file://validation/tests/targets/captionLoc/index.html",
+      "what": "page with standard and nonstandard caption locations"
+    },
+    {
+      "type": "test",
+      "which": "testaro",
+      "withItems": true,
+      "stopOnFail": true,
+      "expect": [
+        [
+          "standardResult.totals.3",
+          "=",
+          2
+        ],
+        [
+          "standardResult.totals.1",
+          "=",
+          0
+        ],
+        [
+          "standardResult.instances.0.ruleID",
+          "=",
+          "captionLoc"
+        ],
+        [
+          "standardResult.instances.0.what",
+          "=",
+          "Element is not the first child of a table element"
+        ],
+        [
+          "standardResult.instances.0.ordinalSeverity",
+          "=",
+          3
+        ],
+        [
+          "standardResult.instances.0.tagName",
+          "=",
+          "CAPTION"
+        ],
+        [
+          "standardResult.instances.0.location.doc",
+          "=",
+          "dom"
+        ],
+        [
+          "standardResult.instances.0.location.type",
+          "=",
+          "box"
+        ],
+        [
+          "standardResult.instances.0.location.spec.width",
+          ">",
+          0
+        ],
+        [
+          "standardResult.instances.0.excerpt",
+          "=",
+          "Office personnel"
+        ],
+        [
+          "standardResult.instances.1.excerpt",
+          "=",
+          "Farm personnel"
+        ]
+      ],
+      "rules": [
+        "y",
+        "captionLoc"
+      ]
+    },
+    {
+      "type": "test",
+      "which": "testaro",
+      "withItems": false,
+      "stopOnFail": true,
+      "expect": [
+        [
+          "standardResult.totals.3",
+          "=",
+          2
+        ],
+        [
+          "standardResult.totals.1",
+          "=",
+          0
+        ],
+        [
+          "standardResult.instances.length",
+          "=",
+          1
+        ],
+        [
+          "standardResult.instances.0.ruleID",
+          "=",
+          "captionLoc"
+        ],
+        [
+          "standardResult.instances.0.what",
+          "=",
+          "caption elements are not the first children of table elements"
+        ],
+        [
+          "standardResult.instances.0.ordinalSeverity",
+          "=",
+          3
+        ],
+        [
+          "standardResult.instances.0.count",
+          "=",
+          2
+        ],
+        [
+          "standardResult.instances.0.tagName",
+          "=",
+          "CAPTION"
+        ]
+      ],
+      "rules": [
+        "y",
+        "captionLoc"
+      ]
+    }
+  ],
+  "sources": {
+    "script": "",
+    "host": {},
+    "requester": ""
+  },
+  "creationTime": "2013-05-28T12:00:00",
+  "timeStamp": "00000"
+}

package/validation/tests/jobs/legendLoc.json

@@ -0,0 +1,146 @@
+{
+  "id": "legendLoc",
+  "what": "validation of legendLoc test",
+  "strict": true,
+  "timeLimit": 20,
+  "acts": [
+    {
+      "type": "launch",
+      "which": "chromium",
+      "url": "file://validation/tests/targets/legendLoc/index.html",
+      "what": "page with standard and nonstandard legend locations"
+    },
+    {
+      "type": "test",
+      "which": "testaro",
+      "what": "legendLoc",
+      "withItems": true,
+      "stopOnFail": true,
+      "expect": [
+        [
+          "standardResult.totals.3",
+          "=",
+          2
+        ],
+        [
+          "standardResult.totals.1",
+          "=",
+          0
+        ],
+        [
+          "standardResult.instances.0.ruleID",
+          "=",
+          "legendLoc"
+        ],
+        [
+          "standardResult.instances.0.what",
+          "=",
+          "Element is not the first child of a fieldset element"
+        ],
+        [
+          "standardResult.instances.0.ordinalSeverity",
+          "=",
+          3
+        ],
+        [
+          "standardResult.instances.0.tagName",
+          "=",
+          "LEGEND"
+        ],
+        [
+          "standardResult.instances.0.location.doc",
+          "=",
+          "dom"
+        ],
+        [
+          "standardResult.instances.0.location.type",
+          "=",
+          "box"
+        ],
+        [
+          "standardResult.instances.0.location.spec.width",
+          ">",
+          0
+        ],
+        [
+          "standardResult.instances.0.excerpt",
+          "=",
+          "Choose a food"
+        ],
+        [
+          "standardResult.instances.1.ruleID",
+          "=",
+          "legendLoc"
+        ],
+        [
+          "standardResult.instances.1.excerpt",
+          "=",
+          "Choose a gas"
+        ]
+      ],
+      "rules": [
+        "y",
+        "legendLoc"
+      ]
+    },
+    {
+      "type": "test",
+      "which": "testaro",
+      "withItems": false,
+      "stopOnFail": true,
+      "expect": [
+        [
+          "standardResult.totals.3",
+          "=",
+          2
+        ],
+        [
+          "standardResult.totals.1",
+          "=",
+          0
+        ],
+        [
+          "standardResult.instances.length",
+          "=",
+          1
+        ],
+        [
+          "standardResult.instances.0.ruleID",
+          "=",
+          "legendLoc"
+        ],
+        [
+          "standardResult.instances.0.what",
+          "=",
+          "legend elements are not the first children of fieldset elements"
+        ],
+        [
+          "standardResult.instances.0.ordinalSeverity",
+          "=",
+          3
+        ],
+        [
+          "standardResult.instances.0.count",
+          "=",
+          2
+        ],
+        [
+          "standardResult.instances.0.tagName",
+          "=",
+          "LEGEND"
+        ]
+      ],
+      "rules": [
+        "y",
+        "legendLoc"
+      ]
+    }
+  ],
+  "sources": {
+    "script": "",
+    "host": {},
+    "requester": ""
+  },
+  "creationTime": "2013-05-28T12:00:00",
+  "timeStamp": "00000"
+}