testaro 26.3.0 → 26.3.2

package/CONTRIBUTING.md

@@ -1,45 +1,68 @@
-# testaro
+# Contributing to Testaro
 
-Ensemble testing for web accessibility
+## Types of contributions
 
-## Contributing
+Testaro can benefit from contributions of various types, such as:
+- Adding other tools to the tools that it integrates.
+- Improving its execution speed.
+- Improving its own rule implementations.
+- Implementing new rules.
 
-Contributions to Testaro are welcome.
+## Adding tools
 
-### Testaro rule specification
+To come.
 
-Testaro integrates 8 accessibility testing tools, and one of them is Testaro itself. The Testaro tool has defined 43 rules, and the other 7 tools have defined about 600 rules in the aggregate. Contributing a new Testaro rule can add value if the new rule will not merely duplicate one of the existing rules. The issue classification (`tic…`) files in the `procs` directory of the [Testilo](https://www.npmjs.com/package/testilo) package can help in this determination.
+## Improving execution speed
 
-If you determine that a new Testaro rule would be valuable, the first step is to specify it. This specification has the following parts:
-1. Adding an entry to the `evalRules` or `etcRules` object in the `tests/testaro.js` file.
-1. Adding a validation target directory to the `validation/tests/targets` directory.
-1. Adding at least one validation target to the target directory.
-1. Adding a validation job to the `validation/tests/jobs` directory.
+To come.
 
-### Testaro rule validation
+## Improving rule implementations.
 
-All Testaro rules have validators, as mentioned above.
+To come.
 
-The first step in creating a validator is to create at least one HTML file that will be tested against the new rule. A single `index.html` file may suffice. It should contain cases that will pass the test and cases that will fail the test. If appropriate, you can create multiple targets, such as `good.html` and `bad.html`.
+## Implementing new rules
 
-The second step is to create a validation job. It launches a browser, navigates to a validation target, and conducts the test. It includes expectations about the results. Typically, the expectations relate to the standard instances included in the results. The `Tests/Expectations` section of the `README.md` file describes the syntax of expectations.
+Testaro has about 50 of its own rules, in addition to the approximately 600 rules of the other tools that it integrates. According to the issue classifications in the [Testilo](https://www.npmjs.com/package/testilo) package, these 650 or so rules can be classified into about 260 accessibility _issues_, because some rules of some tools at least approximately duplicate some rules of other tools.
 
-When a rule `xyz` has been defined and the `npm test xyz` statement is executed, the validation job will be run.
+However, many other significant accessibility issues exist that are not covered by any of the existing rules. Thus, Testaro welcomes contributions of new rules for such issues.
 
-### Rule creation
+### Step 1
 
-Once you have specified a new rule, you, or somebody else, can develop the rule in accord with your specification. Developing the rule consists of:
-1. Creating a rule-definition file.
-1. Adding the file to the `testaro` directory.
-1. Validating it.
+The first step in contributing a new rule to Testaro is to satisfy yourself that it will not duplicate existing rules. The `procs/score/tic35.js` file in the Testilo package should be helpful here.
 
-For example, if the new rule has the ID `bigAlt`, it must succeed when the statement `npm test bigAlt` is executed.
+### Step 2
+
+The second step is to write a validator for the new rule. A validator is software that defines the correct behavior of the implementation of the rule.
+
+Every Testaro rule has a correspoding validator. A validator has two parts:
+- A job file, in the `validation/tests/jobs` directory. It tells Testaro what tests to perform and what the results should be.
+- A target directory, within the `validation/tests/targets` directory. The target directory contains one or more HTML files that will be tested by the job.
+
+Inspecting some of the jobs and targets in the `validation/tests` directory can help you understand how validators work.
+
+### Step 3
+
+The third step is to add an entry to the `evalRules` or `etcRules` object in the `tests/testaro.js` file.
+
+### Step 4
+
+The fourth step is to implement the new rule by creating a JavaScript or JSON file and saving it in the `testaro` directory.
+
+To optimize quality, it may be wise for one person to perform steps 1, 2, and 3, and then for a second person independently to perform step 4 (“clean-room” development).
+
+At any time after an implementation is attempted or revised, the developer can run the validation on it, simply by executing the statement `npm test xyz` (replacing `xyz` with the name of the new rule). When the implementation fails validation, diagnosis may find fault either with the implementation or with the validator.
+
+Whether a new rule should be implemented in JSON or JavaScript depends on the complexity of the rule. The JSON format is effective for simple rules,and JavaScript is needed for more complex rules.
 
 ### Simple rules
 
-Some existing Testaro rules are simple enough to be defined in JSON files. If a new rule is similarly simple, you can define it with a similar JSON file.
+You can create a JSON-defined rule if a single CSS selector can identify all and only the elements on a page that violate the rule.
+
+Suppose, for example, that you want a rule prohibiting `i` elements (because `i` represents confusingly many different semantic properties). A single CSS selector, namely `"i"`, will identify all the `i` elements on the page, so this rule can be defined with JSON.
+
+ Substantially more complex rules, too, can satisfy this criterion. An example is the `captionLoc` rule, which prohibits `caption` elements that are not the first children of `table` elements. Its CSS selector is `"*:not(table) > caption, caption:not(:first-child)"`. That selector identifies every `caption` element that is the child of any non-`table` element, and also every caption element that is not the first among its sibling elements. The only ways to **avoid** being caught by this selector are (1) not to be a `caption` element and (2) to be a `caption` element that is the first child of a `table` element. Thus, the selector catches all the `caption` elements that are not first children of `table` elements, and only those.
 
-An example is the `titledEl` rule:
+You can copy and revise any of the existing JSON files in the `testaro` directory to implement a new rule. If, for example, you start with a copy of the `titledEl` file, you can change its properties to fit your new rule. In particular:
 
 ```json
 {
@@ -54,20 +77,20 @@ An example is the `titledEl` rule:
 }
 ```
 
-To create a simple rule, you can copy one of the existing JSON files and replace the values with values that are appropriate for the new rule. Testaro uses the properties in the file as follows:
-- Every element matching the selector is treated as a violator of the rule.
-- Testaro describes violations with one of the `complaints` values. If the user has requested itemized results, the `instance` value is used; otherwise the `summary` value is used.
-- Testaro assigns the `ordinalSeverity` value as the ordinal severity of each violation. This is a scale of integers, from 0 to 3.
-- When Testaro itemizes results, it reports the tag name of violating elements. When it only summarizes results, it includes the `summaryTagName`. If every violating element necessarily has one particular tag name, then you should make that (e.g., `BUTTON`) the value of `summaryTagName`.
+- Assign a violation description for a single instance to `complaints.instance`.
+- Assign a violation description for a summary instance (when itemization has been turned off) to `complaints.summary`.
+- Assign an integer from 0 through 3 to `ordinalSeverity`.
+- If all instances of violations of the rule necessarily involve elements of the same type, assign its tag name (such as `"BUTTON"`) to `summaryTagName`.
 
 ### Complex rules
 
-Your new rule may not be quite so simple. For example, it may need to examine the selected elements and identify only some of them as rule violators.
-
-You will need to define such a rule in a JavaScript file, rather than a JSON file.
-
-An example of such an existing rule is `filter`. The test for that rule initially selects all elements that descend from the `body` element (or a sample of 100 such elements in the page contains more than 100). Then it examines each selected element to determine whether it violates the rule. Specifically, it determines whether the element has a `filter` style property with a value other than `'none'`. It reports those violations.
-
-To define such a rule, you can copy an existing rule file and replace parts of its code as needed. Examples include:
-- `allSlanted`, `distortion`, `filter`, `miniText`, `zIndex`: violations based on element styles.
+More complex Testaro rules are implemented in JavaScript. You can use any of the existing JavaScript rules, or `data/template.js` file, as an example to begin with. Examples include:
+- `allSlanted`, `distortion`, `filter`, `lineHeight`, `miniText`, `zIndex`: violations based on element styles.
+- `focOp`, `opFoc`, `targetSize`: violations based on attributes
 - `linkTitle`: violations based on attributes and text content.
+- `linkAmb`, `pseudoP`, `radioSet`: violations based on relations among elements and text.
+- `hover`, `tabNav`: violations based on performing actions and observing page behavior.
+- `role`: violations based on data about standards.
+- `docType`, `title`: violations based on page properties.
+
+Some utility functions in modules in the `procs` directory are available for support of new rules.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "26.3.0",
+  "version": "26.3.2",
   "description": "Run 650 web accessibility tests from 8 tools and get a standardized report",
   "main": "index.js",
   "scripts": {

package/testaro/linkOldAtt.json

@@ -0,0 +1,10 @@
+{
+  "ruleID": "linkOldAtt",
+  "selector": "a[charset], a[coords], a[name], a[rev], a[shape]",
+  "complaints": {
+    "instance": "Element has a deprecated attribute",
+    "summary": "Links have deprecated attributes"
+  },
+  "ordinalSeverity": 1,
+  "summaryTagName": "A"
+}

package/testaro/linkTo.json

@@ -2,7 +2,7 @@
   "ruleID": "linkTo",
   "selector": "a:not([href]):visible",
   "complaints": {
-    "instance": "Link has no href attribute",
+    "instance": "Element has no href attribute",
     "summary": "Links are missing href attributes"
   },
   "ordinalSeverity": 2,

package/testaro/nonTable.js

@@ -15,7 +15,7 @@ exports.reporter = async (page, withItems) => {
   const all = await init(page, 'table');
   // For each locator:
   for (const loc of all.allLocs) {
-    // Get whether its el violates the rule.
+    // Get whether its element violates the rule.
     const isBad = await loc.evaluate(el => {
       const role = el.getAttribute('role');
       // If it contains another table:

package/tests/testaro.js

@@ -39,6 +39,7 @@ const evalRules = {
   lineHeight: 'text with a line height less than 1.5 times its font size',
   linkExt: 'links that automatically open new windows',
   linkAmb: 'links with identical texts but different destinations',
+  linkOldAtt: 'links with deprecated attributes',
   linkTitle: 'links with title attributes repeating text content',
   linkTo: 'links without destinations',
   linkUl: 'missing underlines on inline links',

package/validation/tests/jobs/linkOldAtt.json

@@ -0,0 +1,130 @@
+{
+  "id": "linkOldAtt",
+  "what": "validation of linkOldAtt test",
+  "strict": true,
+  "timeLimit": 20,
+  "acts": [
+    {
+      "type": "launch",
+      "which": "chromium",
+      "url": "file://validation/tests/targets/linkOldAtt/index.html",
+      "what": "page with links with and without deprecated attributes"
+    },
+    {
+      "type": "test",
+      "which": "testaro",
+      "withItems": true,
+      "stopOnFail": true,
+      "expect": [
+        [
+          "standardResult.totals.1",
+          "=",
+          2
+        ],
+        [
+          "standardResult.totals.2",
+          "=",
+          0
+        ],
+        [
+          "standardResult.instances.0.ruleID",
+          "=",
+          "linkOldAtt"
+        ],
+        [
+          "standardResult.instances.0.what",
+          "i",
+          "Element has a deprecated attribute"
+        ],
+        [
+          "standardResult.instances.0.ordinalSeverity",
+          "=",
+          1
+        ],
+        [
+          "standardResult.instances.0.tagName",
+          "=",
+          "A"
+        ],
+        [
+          "standardResult.instances.0.location.doc",
+          "=",
+          "dom"
+        ],
+        [
+          "standardResult.instances.0.location.type",
+          "=",
+          "box"
+        ],
+        [
+          "standardResult.instances.0.location.spec.y",
+          ">",
+          0
+        ],
+        [
+          "standardResult.instances.0.excerpt",
+          "=",
+          "French information"
+        ]
+      ],
+      "rules": [
+        "y",
+        "linkOldAtt"
+      ]
+    },
+    {
+      "type": "test",
+      "which": "testaro",
+      "withItems": false,
+      "stopOnFail": true,
+      "expect": [
+        [
+          "standardResult.totals.1",
+          "=",
+          2
+        ],
+        [
+          "standardResult.totals.3",
+          "=",
+          0
+        ],
+        [
+          "standardResult.instances.0.ruleID",
+          "=",
+          "linkOldAtt"
+        ],
+        [
+          "standardResult.instances.0.what",
+          "i",
+          "Links have deprecated attributes"
+        ],
+        [
+          "standardResult.instances.0.tagName",
+          "=",
+          "A"
+        ],
+        [
+          "standardResult.instances.0.ordinalSeverity",
+          "=",
+          1
+        ],
+        [
+          "standardResult.instances.0.count",
+          "=",
+          2
+        ]
+      ],
+      "rules": [
+        "y",
+        "linkOldAtt"
+      ]
+    }
+  ],
+  "sources": {
+    "script": "",
+    "host": {},
+    "requester": ""
+  },
+  "creationTime": "2013-05-28T12:00:00",
+  "timeStamp": "00000"
+}

package/validation/tests/jobs/linkTo.json

@@ -34,7 +34,7 @@
         [
           "standardResult.instances.0.what",
           "i",
-          "Link has no"
+          "Element has no"
         ],
         [
           "standardResult.instances.0.ordinalSeverity",

package/validation/tests/targets/linkOldAtt/index.html

@@ -0,0 +1,17 @@
+<!DOCTYPE html>
+<html lang="en-US">
+  <head>
+    <meta charset="utf-8">
+    <title>Page with links with current and deprecated attributes</title>
+    <meta name="description" content="tester">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+  </head>
+  <body>
+    <main>
+      <h1>Page with links with current and deprecated attributes</h1>
+      <p>This is a link to <a href="https://en.wikipedia.org" hreflang="en">English information</a>.</p>
+      <p>Defect 1: This is a link to <a href="https://fr.wikipedia.org" hreflang="fr" charset="utf-8">French information</a>.</p>
+      <p>Defect 2: This is a link to <a href="https://de.wikipedia.org" name="dewiki">German information</a>.</p>
+    </main>
+  </body>
+</html>