testilo 2.0.3 → 3.1.0

package/README.md

@@ -1,93 +1,55 @@
 # testilo
-Runner of Testaro tests
+Scorer and digester of Testaro reports
 
 ## Introduction
 
-This application is designed to be installed on a Windows or Macintosh host and to operate as a runner of Testaro jobs.
-
-[Testaro](https://www.npmjs.com/package/testaro) is a dependency that performs digital accessibility tests on Web resources.
+This application enriches [Testaro](https://www.npmjs.com/package/testaro) reports. Testaro performs digital accessibility tests on Web resources and creates reports in JSON format. To make those reports more useful, this application, Testilo, computes scores and converts the scored reports to human-readable web pages (_digests_).
 
 ## Dependencies
 
 The `dotenv` dependency lets you set environment variables in an untracked `.env` file.
 
-The Testaro dependency has some dependencies in the @siteimprove scope that are Github Packages. In order to execute `npm install` successfully, you need the `.npmrc` file in your project directory with this content, unless an `.npmrc` file in your home directory or elsewhere provides the same content:
-
-```bash
-@siteimprove:registry=https://npm.pkg.github.com
-//npm.pkg.github.com/:username=abc
-//npm.pkg.github.com/:_authToken=def
-```
-
-In this content, replace `abc` with your Github username and `def` with a Github personal access token that has `read:packages` scope.
-
-## Operation
-
-### General
-
-Testilo orders a Testaro job by calling Testaro’s `handleRequest` function with an object argument. The argument has this structure:
-
-```javascript
-{
-  id,
-  script: {…},
-  log: [],
-  acts: []
-}
-```
-
-The `script` property has a Testaro script as its value. See the Testaro `README.md` file for documentation on scripts.
-
-If a script is represented as JSON in a file `scripts/scriptX.json`, you can incorporate it into the options object of a Testaro call by executing the statement
-
-```javascript
-node index scriptX
-```
-
-### Batches
+## Architecture
 
-You may wish to have Testaro perform the same sequence of tests on multiple web pages. In that case, you can create a _batch_, with the following structure:
+The routines that perform scoring and digesting are _procs_ and are located in the `procs` directory.
 
-```javascript
-{
-  what: 'Web leaders',
-  hosts: {
-    id: 'w3c',
-    which: 'https://www.w3.org/',
-    what: 'W3C'
-  },
-  {
-    id: 'wikimedia',
-    which: 'https://www.wikimedia.org/',
-    what: 'Wikimedia'
-  }
-}
-```
+To score a report, Testilo needs to be told where to find the report, and also which scoring procedure (or _score proc_) to use. There could be multiple score procs differently aggregating the same report into scores.
 
-With a batch, you can execute a single statement to call Testaro multiple times, one per host. On each call, Testilo takes one of the hosts in the batch and substitutes it for each host specified in a `url` command of the script. Testilo waits for each Testaro job to finish before calling the next Testaro job.
+Similarly, once a report has been scored, Testilo can digest it, provided that Testilo is told where to find the scored report and which _digest proc_ to use. Different digest procs could produce different human-oriented explanations of the same scored report, such as for different audiences.
 
-If a batch is represented as a JSON file `batches/batchY.json`, you can use it to call a set of Testaro jobs with the statement
+Testilo includes some score procs and digest procs. You can add others.
 
-```javascript
-node index scriptX batchY
-```
+## Execution
 
-Given that statement, Testilo replaces the hosts in the script with the first host in the batch and calls Testaro. When Testaro finishes performing that script, Testilo replaces the script hosts with the second batch host and calls Testaro again. And so on.
+### Scoring
 
-### Reports
+To score a Testaro report, execute the statement `node score abc xyz`, replacing (here and below) `abc` with the base of the name of the file containing the report and `xyz` with the base of the name of the score proc.
 
-When you execute a `node index …` statement, Testilo begins populating the object argument by giving its `id` property a value. If there is no batch, the value of that property is a string encoding the date and time when you executed the statement (e.g., `eh9q7r`). If there is a batch, the value is the same, except that it is suffixed with a hyphen-minus character followed by the `id` value of the host (e.g., `eh9q7r-wikimedia`).
+This procedure has some preconditions:
+- The score proc is compatible with the report.
+- The full name of the report file is `abc.json`.
+- The report file is located in the directory whose relative path (relative to the project directory of Testilo) is the value of the `REPORTDIR_RAW` environment variable.
+- Testilo can read and write in the `REPORTDIR_RAW` directory.
+- Testilo can write in the `REPORTDIR_SCORED` directory.
+- The `procs/score` directory contains a file named `xyz.js`.
 
-Testaro delivers its results by populating the `log` and `acts` arrays of the object argument. Testilo waits for Testaro to finish performing the script and then saves the object argument in JSON format as a file in the `reports` directory. The name of the file is the `id` value of the object, suffixed with `.json`.
+Thus, the script that Testaro ran in order to produce the report must be one that Testilo has a scoring algorithm (_score proc_) for. If so, Testilo can score the report.
 
-## Configuration
+When Testilo scores a report, Testilo saves the scored report in the directory whose relative path is the value of the `REPORTDIR_SCORED`. The scored report file has the same name as the original.
 
-### `ibm` test
+### Digesting
 
-Testaro can perform the `ibm` test. That test requires the `aceconfig.js` configuration file in the root directory of the Testilo project.
+To make a scored Testaro report more useful for humans, Testilo can create a digest of the report. This is an HTML document (a web page) summarizing and explaining the findings.
 
-### Environment variables
+To make Testilo digest a report, execute the statement `node digest abc xyz`, replacing `abc` (here and below) with the base of the name of the file containing the scored report and `xyz` with the base of the name of the digest proc.
 
-If a `wave` test is included in a script, an environment variable named `TESTARO_WAVE_KEY` must exist, with your WAVE API key as its value.
+This procedure has some preconditions:
+- The digest proc is compatible with the report.
+- The full name of the report file is `abc.json`.
+- The report file is located in the directory whose relative path (relative to the project directory of Testilo) is the value of the `REPORTDIR_SCORED` environment variable.
+- Testilo can read and write in the `REPORTDIR_SCORED` directory.
+- Testilo can write in the `REPORTDIR_DIGESTED` directory.
+- The `procs/digest` directory contains a subdirectory named `xyz`, which in turn contains files named `index.html` and `index.js`.
+- You have copied the `reports/digested/style.css` file into the `REPORTDIR_DIGESTED` directory.
 
-Before executing a Testaro script, you can optionally also set the environment variables `TESTARO_DEBUG` (to `'true'` or anything else) and/or `TESTARO_WAITS` (to a non-negative integer). The effects of these variables are described in the Testaro `index.js` file.
+When Testilo digests a report, Testilo saves the digest in the directory whose relative path is the value of the `REPORTDIR_DIGESTED`. The digest has the name `abc.html`.

package/digest.js

@@ -0,0 +1,40 @@
+/*
+  digest.js
+  Testilo digesting script.
+*/
+
+// ########## IMPORTS
+
+// Module to keep secrets.
+require('dotenv').config();
+// Module to read and write files.
+const fs = require('fs').promises;
+
+// ########## CONSTANTS
+
+const reportDirScored = process.env.REPORTDIR_SCORED || 'reports/scored';
+const reportDirDigested = process.env.REPORTDIR_DIGESTED || 'reports/digested';
+const reportID = process.argv[2];
+const digesterID = process.argv[3];
+
+// ########## FUNCTIONS
+
+// Replaces the placeholders in content with eponymous query parameters.
+const replaceHolders = (content, query) => content
+.replace(/__([a-zA-Z]+)__/g, (ph, qp) => query[qp]);
+// Creates a digest.
+const digest = async () => {
+  const reportJSON = await fs.readFile(`${__dirname}/${reportDirScored}/${reportID}.json`, 'utf8');
+  const report = JSON.parse(reportJSON);
+  const {makeQuery} = require(`${__dirname}/procs/digest/${digesterID}/index.js`);
+  const query = {};
+  makeQuery(report, query);
+  const template = await fs.readFile(`${__dirname}/procs/digest/${digesterID}/index.html`, 'utf8');
+  const digest = replaceHolders(template, query);
+  await fs.writeFile(`${__dirname}/${reportDirDigested}/${reportID}.html`, digest);
+  console.log(`Report ${reportID} digested and saved`);
+};
+
+// ########## OPERATION
+
+digest();

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "testilo",
-  "version": "2.0.3",
-  "description": "Client that runs Testaro tests to fulfill Aorta jobs",
+  "version": "3.1.0",
+  "description": "Client that scores and digests Testaro reports",
   "main": "index.js",
   "scripts": {
     "test": "echo \"Error: no test specified\" && exit 1"
@@ -18,8 +18,7 @@
   },
   "homepage": "https://github.com/jrpool/testilo",
   "dependencies": {
-    "dotenv": "*",
-    "testaro": "*"
+    "dotenv": "*"
   },
   "devDependencies": {
     "eslint": "*"

package/procs/digest/tsp09a/index.html

@@ -0,0 +1,126 @@
+<!DOCTYPE HTML>
+<html lang="en-US">
+  <head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <meta name="author" content="__testerName__">
+    <meta name="creator" content="__testerName__">
+    <meta name="publisher" name="__testerName__">
+    <meta name="description" content="report on results of __scoreProc__ procedure">
+    <meta name="keywords" content="accessibility a11y web testing">
+    <title>Accessibility test digest</title>
+    <link rel="icon" href="favicon.png">
+    <link rel="stylesheet" href="style.css">
+  </head>
+  <body>
+    <main>
+      <header>
+        <h1>Accessibility test digest</h1>
+        <p class="summary bold">__org__</p>
+        <p class="summary">Testilo, <code>tsp09</code> procedure</p>
+        <p class="summary">Score: __totalScore__</p>
+      </header>
+      <h2>Introduction</h2>
+      <p>The <code>tsp09</code> accessibility testing procedure was executed by <a href="https://www.npmjs.com/package/testaro">Testaro</a> on the web page at <a href="__url__">__url__</a>, belonging to __org__, on __dateSlash__. The procedure performed 628 tests. Of these, 16 tests are custom tests defined by Testaro, and the others belong to packages of tests created by others. Testaro produced a report enumerating the test results.</p>
+      <p>These tests&mdash;like all tests&mdash;are fallible. The failures described below merit investigation as <strong>potential</strong> opportunities for improved accessibility. But some reported faults may not actually harm accessibility, and some accessibility faults can escape detection by any of these tests.</p>
+      <p>Given the report produced by Testaro, another procedure, <code>tsp09a</code>, assigned weights to the tests and computed a total score for the page, __totalScore__ (where 0 is the best possible score). Just as tests are fallible, any scoring procedure is subjective, and different reasonable procedures could assign different scores on the basis of the same report. This digest explains how <code>tsp09a</code> computed a score for the page.</p>
+      <h2>Summary</h2>
+      <p>The packages&rsquo; and custom tests&rsquo; contributions to the score were:</p>
+      <table class="allBorder secondCellRight">
+        <caption>Score components</caption>
+        <tbody class="headersLeft">
+        __scoreRows__
+      </tbody></table>
+      <h2>Test packages</h2>
+      <p>Most of the tests belong to the following five accessibility test packages created by various groups of specialists.</p>
+      <h3><code>aatt</code></h3>
+      __aattResult__
+      <p><a href="https://github.com/paypal/AATT">AATT</a> is an open-source package sponsored by PayPal. The <code>aatt</code> test performs all 98 WCAG 2.1 AA tests in the HTML CodeSniffer ruleset.</p>
+      <h3><code>alfa</code></h3>
+      __alfaResult__
+      <p><a href="https://github.com/Siteimprove/alfa">Alfa</a> is an open-source package sponsored by Siteimprove. The <code>alfa</code> test performs all 103 tests in the Alfa package.</p>
+      <h3><code>axe</code></h3>
+      __axeResult__
+      <p><a href="https://github.com/dequelabs/axe-core">Axe</a> is an open-source package sponsored by accessibility consulting firm Deque. The <code>axe</code> test performs all 138 default tests in the Axe package.</p>
+      <h3><code>ibm</code></h3>
+      __ibmResult__
+      <p><a href="https://github.com/IBMa/equal-access">Equal Access</a> is an open-source package sponsored by IBM Corporation. The <code>ibm</code> test performs all 163 default tests in the Equal Access package.</p>
+      <h3><code>wave</code></h3>
+      __waveResult__
+      <p><a href="https://wave.webaim.org/">WAVE</a> is a proprietary package owned by webAIM, a program of the Institute for Disability Research, Policy, and Practice at Utah State University. The <code>wave</code> test performs all 110 default tests in the WAVE package.</p>
+      <h2>Custom tests</h2>
+      <p>The tests in the above packages are designed to detect some, not all, accessibility problems. The procedure includes the following 16 custom tests that supplement the tests of the packages.</p>
+      <h3><code>bulk</code></h3>
+      __bulkResult__
+      <p>The <code>bulk</code> test counts the initially visible elements in a page. A page with a large count tends to be complex and busy, frustrating some users, especially if they have visual or motor disabilities, as they try to determine what the page is about, whether it is relevant, and how to find a specific thing in it.</p>
+      <p>When the count exceeds 250, the procedure begins to assign a non-zero score.</p>
+      <h3><code>embAc</code></h3>
+      __embAcResult__
+      <p>The <code>embAc</code> test detects improper embedding of interactive elements (links, buttons, inputs, and select lists) within links or buttons. Such embedding violates the HTML standard, complicates user interaction, and creates risks of error. It becomes non-obvious what a user will activate with a click.</p>
+      <h3><code>focAll</code></h3>
+      __focAllResult__
+      <p>The <code>focAll</code> test detects discrepancies between focusable and Tab-focused element counts. Navigating with the Tab key normally moves the focus and does nothing else. If it also adds or fails to focus focusable elements, this complicates navigation and may make the page unusable for people who must use only a keyboard (not a mouse) or a keyboard-emulating assistive device to navigate.</p>
+      <h3><code>focInd</code></h3>
+      __focIndResult__
+      <p>The <code>focInd</code> test detects focusable elements without standard focus indicators. An outline is the standard and most recognizable focus indicator; as you repeatedly press the Tab key, the outline moves through the page. Other focus indicators are more likely to be misunderstood. For example, underlines may be mistaken for selection indicators or links. An absent focus indicator prevents the user from knowing what a keyboard action will act on.</p>
+      <h3><code>focOp</code></h3>
+      __focOpResult__
+      <p>The <code>focOp</code> test detects descrepancies between Tab-focusability and operability. The standard practice is to make focusable elements operable and vice versa. If focusable elements are not operable, users are likely to be surprised that nothing happens when they try to operate such elements. If operable elements are not focusable, users depending on keyboard navigation are prevented from operating those elements. The test considers an element operable if it has a non-inherited pointer cursor and is not a <code>LABEL</code> element, or has an operable tag name (<code>A</code>, <code>BUTTON</code>, <code>IFRAME</code>, <code>INPUT</code>, <code>SELECT</code>, or <code>TEXTAREA</code>), or has an <code>onclick</code> attribute. The test considers an element Tab-focusable if its <code>tabIndex</code> property has the value 0.</p>
+      <h3><code>hover</code></h3>
+      __hoverResult__
+      <p>The <code>hover</code> test detects unexpected effects of hovering. The normal purpose of hovering is to show the user which element is currently being actually or effectively hovered over and would therefore be the target of a mouse click. When hovering does more than that, the additional effects can confuse or startle users, especially those without precise mouse control. The test detects whether hovering makes elements visible, changes the opacities of elements, affects the opacities of elements by changing the opacities of their ancestors, and fails to reach elements. Only visible elements that have <code>A</code>, <code>BUTTON</code>, and <code>LI</code> tag names or have <code>onmouseenter</code> or <code>onmouseover</code> attributes are considered as triggers of such effects when hovered over. The effects of hovering are inspected for the descendants of the grandparent of the trigger if the trigger has the tag name <code>A</code> or <code>BUTTON</code>, or otherwise the descendants of the trigger. The only elements counted as being made visible by hovering are those with tag names <code>A</code>, <code>BUTTON</code>, <code>INPUT</code>, and <code>SPAN</code>, and those with <code>role="menuitem"</code> attributes.</p>
+      <h3><code>labClash</code></h3>
+      __labClashResult__
+      <p>The <code>labClash</code> test detects defects in the labeling of buttons, non-hidden inputs, select lists, and text areas. The defects include missing labels and redundant labels. Redundant labels are labels that are superseded by other labels. Explicit and implicit (wrapped) labels are additive, not conflicting.</p>
+      <h3><code>linkUl</code></h3>
+      __linkUlResult__
+      <p>The <code>linkUl</code> test detects failures to underline inline links. Underlining and color are the traditional style properties that identify links. Collections of links in blocks can sometimes be recognized without underlines, but inline links are difficult or impossible to distinguish visually from surrounding text if not underlined. Underlining inline links only on hover provides an indicator valuable only to mouse users, and even they must traverse the text with a mouse merely to discover which passages are links.</p>
+      <p>Warning: This test classifies links as inline or block. Some links classified as inline may not look like inline links to users.</p>
+      <h3><code>log</code></h3>
+      __logResult__
+      <p>The <code>log</code> test detects problems with the behavior of the page or the server. Indicators of such problems are the number of messages logged by the browser, the aggregate size of those messages in characters, the number of rejections with abnormal HTTP statuses, the number of <q>prohibited</q> HTTP statuses, and the number of times the browser timed out trying to reach the page. Although log messages do not always indicate page defects, they mostly do.</p>
+      <h3><code>menuNav</code></h3>
+      __menuNavResult__
+      <p>The <code>menuNav</code> test detects nonstandard keyboard navigation among menu items in menus that manage the focus of their menu items. Menus that use pseudofocus with the <code>aria-activedescendant</code> attribute are not tested. The test is based on <a href="https://www.w3.org/TR/wai-aria-practices-1.1/#menu">WAI-ARIA recommendations</a>.</p>
+      <h3><code>motion</code></h3>
+      __motionResult__
+      <p>The <code>motion</code> test detects unrequested motion in a page. Accessibility standards minimally require motion to be brief, or else stoppable by the user. But stopping motion can be difficult or impossible, and, by the time a user manages to stop motion, the motion may have caused annoyance or harm. For superior accessibility, a page contains no motion until and unless the user authorizes it. The test compares five screen shots of the initially visible part of the page and assigns a score based on:</p>
+      <ol>
+        <li>bytes: an array of the sizes of the screen shots, in bytes</li>
+        <li>localRatios: an array of the ratios of bytes of the larger to the smaller of adjacent pairs of screen shots</li>
+        <li>meanLocalRatio: the mean of the ratios in the localRatios array</li>
+        <li>maxLocalRatio: the greatest of the ratios in the localRatios array</li>
+        <li>globalRatio: the ratio of bytes of the largest to the smallest screen shot</li>
+        <li>pixelChanges: an array of counts of differing pixels between adjacent pairs of screen shots</li>
+        <li>meanPixelChange: the mean of the counts in the pixelChanges array</li>
+        <li>maxPixelChange: the greatest of the counts in the pixelChanges array</li>
+        <li>changeFrequency: what fraction of the adjacent pairs of screen shots has pixel differences</li>
+      </ol>
+      <p>Warning: This test waits 2.4 seconds before making its first screen shot. If a page loads more slowly than that, the test may treat it as exhibiting motion.</p>
+      <h3><code>radioSet</code></h3>
+      __radioSetResult__
+      <p>The <code>radioSet</code> test detects nonstandard groupings of radio buttons. It defines the standard to require that two or more radio buttons with the same name, and no other radio buttons, be grouped in a <code>fieldset</code> element with a valid <code>legend</code> element.</p>
+      <h3><code>role</code></h3>
+      __roleResult__
+      <p>The <code>role</code> test detects nonstandard and confusing role assignments. It is inspired by the <a href="https://www.w3.org/TR/wai-aria/#roles_categorization">WAI-ARIA recommendations on roles</a> and their <a href="https://www.w3.org/TR/html-aria/">authoring rules</a>. Abstract roles and roles that are implicit in HTML elements fail the test. The <code>math</code> role has been removed, because of poor adoption and exclusion from HTML5. The <code>img</code> role <a href="https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Roles/img_role">has accessibility uses</a>, so does not fail the test, although implicit in the HTML <code>img</code> element.</p>
+      <h3><code>styleDiff</code></h3>
+      __styleDiffResult__
+      <p>The <code>styleDiff</code> test detects style inconsistencies among inline links, block links, buttons, and all 6 levels of headings. The principle of consistent identification requires using styles to help users classify content. For example, level-2 headings look the same, and they look different from level-1 headings. Ideally, then, for each of these element types, there would be exactly 1 style. The test considers the style properties <code>borderStyle</code>, <code>borderWidth</code>, <code>fontStyle</code>, <code>fontWeight</code>, <code>lineHeight</code>, <code>maxHeight</code>, <code>maxWidth</code>, <code>minHeight</code>, <code>minWidth</code>, <code>opacity</code>, <code>outlineOffset</code>, <code>outlineStyle</code>, <code>outlineWidth</code>, <code>textDecorationLine</code>, <code>textDecorationStyle</code>, and <code>textDecorationThickness</code>. For headings, it also considers the <code>fontSize</code> style property.</p>
+      <h3><code>tabNav</code></h3>
+      __tabNavResult__
+      <p>The <code>tabNav</code> test detects <a href="https://www.w3.org/TR/wai-aria-practices-1.1/#tabpanel">nonstandard keyboard navigation</a> among tab elements in tab lists. Tab lists let users choose which of several content blocks to display. The Tab key moves the focus into and out of a tab list, but the arrow, Home, and End keys move the focus from tab to tab.</p>
+      <h3><code>zIndex</code></h3>
+      __zIndexResult__
+      <p>The <code>zIndex</code> test detects elements with non-default z indexes. Pages present difficulty for some users when they require users to perceive a third dimension (depth) in the two-dimensional display. Layers, popups, and dialogs that cover other content make it difficult for some users to interpret the content and know what parts of the content can be acted on. Layering also complicates accessibility testing. Tests for visibility of focus, for example, may fail if a focused element is covered by another element.</p>
+      <h2>Test prevention</h2>
+      <p>Some pages prevent some of the tests in this procedure from being performed. This may occur, for example, when a page tries to block any non-human visitor. The procedure estimates high scores when pages prevent tests, because preventing accessibility testing is itself an accessibility deficiency. Specifically:</p>
+      <ul>
+        <li>Measuring success is a prerequisite for achieving success, so interfering with accessibility measurement interferes with accessibility.</li>
+        <li>Users with disabilities often rely on assistive technologies to mediate between them and web applications. Applications that interfere with automated testing are at risk of interfering with some assistive technologies, too.</li>
+      </ul>
+      <footer>
+        <p class="date">Produced <time itemprop="datePublished" datetime="__dateISO__">__dateSlash__</time></p>
+      </footer>
+    </main>
+  </body>
+</html>