testaro 28.2.4 → 29.0.0

package/CONTRIBUTING.md

@@ -12,7 +12,6 @@ Testaro can benefit from contributions of various types, such as:
 
 Tools that may merit consideration include:
 - Arc (by TPGi). Costs $0.05 per page tested, and requires each user to have an account with the tool maker and to allow the tool maker to charge a charge account for payment.
-- Aslint (by Essential Accessibility). Apparently requires using a bundle file that is in the repository but is also listed in the `.gitignore` file and is missing from the published package, so would need to be physically included in the Testaro code rather than referenced as a dependency.
 
 ## Improving execution speed
 
@@ -24,13 +23,13 @@ To come.
 
 ## Implementing new rules
 
-Testaro has about 50 of its own rules, in addition to the approximately 600 rules of the other tools that it integrates. According to the issue classifications in the [Testilo](https://www.npmjs.com/package/testilo) package, these 650 or so rules can be classified into about 260 accessibility _issues_, because some rules of some tools at least approximately duplicate some rules of other tools.
+Testaro has about 50 of its own rules, in addition to the approximately 900 rules of the other tools that it integrates. According to the issue classifications in the [Testilo](https://www.npmjs.com/package/testilo) package, these 950 or so rules can be classified into about 290 accessibility _issues_, because some rules of some tools at least approximately duplicate some rules of other tools.
 
 However, many other significant accessibility issues exist that are not covered by any of the existing rules. Thus, Testaro welcomes contributions of new rules for such issues.
 
 ### Step 1
 
-The first step in contributing a new rule to Testaro is to satisfy yourself that it will not duplicate existing rules. The `procs/score/tic35.js` file in the Testilo package should be helpful here.
+The first step in contributing a new rule to Testaro is to satisfy yourself that it will not duplicate existing rules. The `procs/score/tic36.js` file in the Testilo package should be helpful here.
 
 ### Step 2
 

package/README.md

@@ -221,7 +221,7 @@ Testaro helps overcome this format diversity by offering to represent the main f
 
 In the conceptual scheme underlying the format standardization of Testaro, each tool has its own set of _rules_, where a rule is an algorithm for evaluating a target and determining whether instances of some kind of problem exist in it. With standardization, Testaro reports, in a uniform way, the outcomes from the application of rules by tools to a target.
 
-If the `STANDARD` environment variable has the value `also` (which it has by default) or `only`, Testaro converts some data in each tool report to the standard format. That permits you to ignore the format idiosyncrasies of the tools. If `STANDARD` has the value `also`, the job report includes both formats. If the value is `only`, the job report includes only the standard format. If the value is `no`, the job report includes only the original format of each tool report.
+Each job can specify how Testaro is to handle report standardization. A job can contain a `standard` property. If the value of that property is `'also'` or `'only'`, Testaro converts some data in each tool report to the standard format. That permits you to ignore the format idiosyncrasies of the tools. If `standard` has the value `'also'`, the job report includes both formats. If the value is `'only'`, the job report includes only the standard format. If the value is `'no'` (or anything else, or there is no `standard` property), the job report includes only the original format of each tool report.
 
 The standard format of each tool report has these properties:
 - `prevented`: `true` if the tool failed to run on the page, or otherwise omitted.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "28.2.4",
+  "version": "29.0.0",
   "description": "Run 960 web accessibility tests from 9 tools and get a standardized report",
   "main": "index.js",
   "scripts": {

package/procs/tellServer.js

@@ -0,0 +1,28 @@
+/*
+  tellServer
+  Notify the server.
+*/
+
+// CONSTANTS
+
+const httpClient = require('http');
+const httpsClient = require('https');
+const agent = process.env.AGENT;
+
+// FUNCTIONS
+
+exports.tellServer = (report, messageParams, logMessage) => {
+  const observer = report.sources.sendReportTo.replace(/report$/, 'granular');
+  const whoParams = `agent=${agent}&jobID=${report.id || ''}`;
+  const wholeURL = `${observer}?${whoParams}&${messageParams}`;
+  const client = wholeURL.startsWith('https://') ? httpsClient : httpClient;
+  const request = client.request(wholeURL);
+  // If the notification threw an error:
+  request.on('error', error => {
+    // Report the error.
+    const errorMessage = 'ERROR notifying the server';
+    console.log(`${errorMessage} (${error.message})`);
+  });
+  request.end();
+  console.log(`${logMessage} (server notified)`);
+};

package/run.js

@@ -13,6 +13,8 @@ const {actSpecs} = require('./actSpecs');
 const {browserClose, getNonce, goTo, launch} = require('./procs/nav');
 // Module to standardize report formats.
 const {standardize} = require('./procs/standardize');
+// Module to send a notice to the server.
+const {tellServer} = require('./procs/tellServer');
 
 // ########## CONSTANTS
 
@@ -20,8 +22,6 @@ const {standardize} = require('./procs/standardize');
 const debug = process.env.DEBUG === 'true';
 // Set WAITS environment variable to a positive number to insert delays (in ms).
 const waits = Number.parseInt(process.env.WAITS) || 0;
-// Set STANDARD environment variable to also if not set.
-process.env.STANDARD ??= 'also';
 // CSS selectors for targets of moves.
 const moves = {
   button: 'button, [role=button], input[type=submit]',
@@ -45,10 +45,6 @@ const tests = {
   testaro: 'Testaro',
   wave: 'WAVE',
 };
-// Observation items.
-const httpClient = require('http');
-const httpsClient = require('https');
-const agent = process.env.AGENT;
 
 // ########## VARIABLES
 
@@ -466,26 +462,12 @@ const doActs = async (report, actIndex, page) => {
           actInfo = act.which;
         }
       }
-      const whichSuffix = actInfo ? ` (${actInfo})` : '';
       // If granular reporting has been specified:
-      let granularSuffix = '';
       if (report.observe) {
         // Notify the server of the act.
-        const observer = report.sources.sendReportTo.replace(/report$/, 'granular');
-        const whoParams = `agent=${agent}&jobID=${report.id || ''}`;
-        const actParams = `act=${act.type}&which=${act.which || ''}`;
-        const wholeURL = `${observer}?${whoParams}&${actParams}`;
-        const client = wholeURL.startsWith('https://') ? httpsClient : httpClient;
-        const request = client.request(wholeURL);
-        // If the notification threw an error:
-        request.on('error', error => {
-          // Report the error.
-          const errorMessage = 'ERROR notifying the server of an act';
-          console.log(`${errorMessage} (${error.message})`);
-        });
-        request.end();
-        granularSuffix = ' with notice to server';
-        console.log(`>>>> ${act.type}${whichSuffix}${granularSuffix}`);
+        const whichParam = act.which ? `&which=${act.which}` : '';
+        const messageParams = `act=${act.type}${whichParam}`;
+        tellServer(report, messageParams, `>>>> ${act.type}: ${actInfo}`);
       }
       // Increment the count of acts performed.
       actCount++;
@@ -709,7 +691,9 @@ const doActs = async (report, actIndex, page) => {
             act.what = tests[act.which];
             // Initialize the options argument.
             const options = {
+              report,
               act,
+              granular: report.observe,
               scriptNonce: report.jobData.lastScriptNonce || ''
             };
             // Add any specified arguments to it.
@@ -746,7 +730,7 @@ const doActs = async (report, actIndex, page) => {
             const resultCount = Object.keys(toolReport.result).length;
             act.result = resultCount ? toolReport.result : {success: false};
             // If a standard-format result is to be included in the report:
-            const standard = process.env.STANDARD;
+            const standard = report.standard || 'only';
             if (['also', 'only'].includes(standard)) {
               // Initialize it.
               act.standardResult = {
@@ -757,7 +741,11 @@ const doActs = async (report, actIndex, page) => {
               standardize(act);
               // If the original-format result is not to be included in the report:
               if (standard === 'only') {
-                // Remove it.
+                // Remove it, except any important property.
+                if (act.result.important) {
+                  act.data = act.result.important;
+                  console.log('>>>>>> Important result data protected from deletion');
+                }
                 delete act.result;
               }
               // If the test has expectations:

package/tests/testaro.js

@@ -9,21 +9,27 @@
 const {init, report} = require('../procs/testaro');
 // Module to handle files.
 const fs = require('fs/promises');
+// Module to send a notice to the server.
+const {tellServer} = require('../procs/tellServer');
 
 // ######## CONSTANTS
 
-const futureEvalRules = {
-  adbID: 'elements with ambiguous or missing referenced descriptions',
+/*
+const futureEvalRulesTraining = {
   altScheme: 'img elements with alt attributes having URLs as their entire values',
   captionLoc: 'caption elements that are not first children of table elements',
   datalistRef: 'elements with ambiguous or missing referenced datalist elements',
+  secHeading: 'headings that violate the logical level order in their sectioning containers',
+  textSem: 'semantically vague elements i, b, and/or small'
+};
+const futureEvalRulesCleanRoom = {
+  adbID: 'elements with ambiguous or missing referenced descriptions',
   imageLink: 'links with image files as their destinations',
   legendLoc: 'legend elements that are not first children of fieldset elements',
   optRoleSel: 'Non-option elements with option roles that have no aria-selected attributes',
-  phOnly: 'input elements with placeholders but no accessible names',
-  secHeading: 'headings that violate the logical level order in their sectioning containers',
-  textSem: 'semantically vague elements i, b, and/or small',
+  phOnly: 'input elements with placeholders but no accessible names'
 };
+*/
 const evalRules = {
   allCaps: 'leaf elements with entirely upper-case text longer than 7 characters',
   allHidden: 'page that is entirely or mostly hidden',
@@ -95,15 +101,17 @@ const jsonTest = async (ruleID, ruleArgs) => {
 };
 // Conducts and reports Testaro tests.
 exports.reporter = async (page, options) => {
-  const {withItems, stopOnFail, args} = options;
+  const {withItems, stopOnFail, granular, args} = options;
   const argRules = args ? Object.keys(args) : null;
   const rules = options.rules || ['y', ... Object.keys(evalRules)];
   // Initialize the data.
   const data = {
     rules: {},
-    preventions: [],
-    invalid: [],
-    testTimes: {}
+    important: {
+      preventions: [],
+      invalid: [],
+      testTimes: {}
+    }
   };
   // If the rule specification is valid:
   if (
@@ -129,13 +137,21 @@ exports.reporter = async (page, options) => {
           // Add them to the argument array.
           ruleArgs.push(... args[rule]);
         }
-        // Test the page.
+        // If granular reporting is specified:
         const what = evalRules[rule] || etcRules[rule];
+        if (granular) {
+          // Report the rule to the server.
+          tellServer(
+            options.report,
+            `act=test&which=testaro&rule=${rule}&ruleWhat=${what}`,
+            `>>>>>> ${rule} (${what})`
+          );
+        }
+        // Test the page.
         if (! data.rules[rule]) {
           data.rules[rule] = {};
         }
         data.rules[rule].what = what;
-        console.log(`>>>>>> ${rule} (${what})`);
         try {
           const startTime = Date.now();
           const ruleReport = isJS
@@ -149,7 +165,7 @@ exports.reporter = async (page, options) => {
           });
           data.rules[rule].totals = data.rules[rule].totals.map(total => Math.round(total));
           if (ruleReport.prevented) {
-            data.preventions.push(rule);
+            data.important.preventions.push(rule);
           }
           // If testing is to stop after a failure and the page failed the test:
           if (stopOnFail && ruleReport.totals.some(total => total)) {
@@ -160,19 +176,19 @@ exports.reporter = async (page, options) => {
         // If an error is thrown by the test:
         catch(error) {
           // Report this.
-          data.preventions.push(rule);
+          data.important.preventions.push(rule);
           console.log(`ERROR: Test of testaro rule ${rule} prevented (${error.message})`);
         }
       }
       // Otherwise, i.e. if the rule is undefined or doubly defined:
       else {
         // Report this.
-        data.invalid.push(rule);
+        data.important.invalid.push(rule);
         console.log(`ERROR: Rule ${rule} not validly defined`);
       }
     }
     testTimes.sort((a, b) => b[1] - a[1]).forEach(pair => {
-      data.testTimes[pair[0]] = pair[1];
+      data.important.testTimes[pair[0]] = pair[1];
     });
   }
   // Otherwise, i.e. if the rule specification is invalid: