testaro 45.0.10 → 45.0.12

package/README.md

@@ -44,35 +44,37 @@ Testaro uses:
 - [pixelmatch](https://www.npmjs.com/package/pixelmatch) to measure motion
 
 Testaro performs tests of these _tools_:
-- [accessibility-checker](https://www.npmjs.com/package/accessibility-checker) (IBM)
-- [alfa](https://alfa.siteimprove.com/) (Siteimprove)
-- [aslint](https://www.npmjs.com/package/@essentialaccessibility/aslint) (eSSENTIAL Accessibility)
-- [axe-playwright](https://www.npmjs.com/package/axe-playwright) (Deque)
+- [Accessibility Checker](https://www.npmjs.com/package/accessibility-checker) (IBM)
+- [Alfa](https://alfa.siteimprove.com/) (Siteimprove)
+- [ASLint](https://www.npmjs.com/package/@essentialaccessibility/aslint) (eSSENTIAL Accessibility)
+- [Axe](https://www.npmjs.com/package/axe-playwright) (Deque)
 - [Editoria11y](https://github.com/itmaybejj/editoria11y) (Princeton University)
 - [HTML CodeSniffer](https://www.npmjs.com/package/html_codesniffer) (Squiz Labs)
 - [Nu Html Checker](https://github.com/validator/validator) (World Wide Web Consortium)
-- [QualWeb core](https://www.npmjs.com/package/@qualweb/core) (University of Lisbon)
+- [QualWeb](https://www.npmjs.com/package/@qualweb/core) (University of Lisbon)
 - [Testaro](https://www.npmjs.com/package/testaro) (CVS Health)
-- [WAVE API](https://wave.webaim.org/api/) (WebAIM)
+- [WallyAX](https://www.npmjs.com/package/@wally-ax/wax-dev) (Wally Solutions)
+- [WAVE](https://wave.webaim.org/api/) (WebAIM)
 
 Some of the tests of Testaro are designed to act as approximate alternatives to tests of vulnerable, restricted, or no longer available tools. In all such cases the Testaro rules are independently designed and implemented, without reference to the code of the tests that inspired them.
 
 ## Rules
 
-Each tool accessed with Testaro defines _rules_ and tests _targets_ for compliance with its rules. In total, the nine tools define about a thousand rules. The latest tabulation of tool rules is:
+Each tool accessed with Testaro defines _rules_ and tests _targets_ for compliance with its rules. In total, the eleven tools define more than a thousand rules. The latest tabulation of tool rules is:
 
 ```
-alfa: 59
-aslint: 136
-axe: 80
-ed11y: 24
-htmlcs: 115
-ibm: 132
-nuVal: 215
-qualWeb: 131
-testaro: 40
-wave: 58
-total: 990
+Accessibility Checker: 132
+Alfa: 59
+ASLint: 136
+Axe: 80
+Editoria11y: 24
+HTML CodeSniffer: 115
+Nu Html Checker: 215
+QualWeb: 131
+Testaro: 40
+WallyAX: 27
+WAVE: 58
+total: 1017
 ```
 
 Some of the tools are under active development, and their rule counts change over time.
@@ -119,7 +121,7 @@ To run Testaro after installation, provide the environment variables described b
 
 ## Payment
 
-All of the tests that Testaro can perform are free of cost, except those performed by the WAVE tool. The owner of that tool gives new registrants a free allowance of credits before it becomes necessary to pay for use of the API of the tool. The required environment variable for authentication and payment is described below under “Environment variables”.
+All of the tests that Testaro can perform are free of cost, except those performed by the WallyAX and WAVE tools. The owners of those tools issue API keys. A free initial allowance of usage may be granted to you with a new API key. Before using Testaro to perform their tests, get your API keys for [WallyAX](mailto:technology@wallyax.com) and [WAVE](https://wave.webaim.org/api/). Then use those API keys to define environment variables, as described below under “Environment variables”.
 
 ## Jobs
 
@@ -459,7 +461,7 @@ The changes in `htmlcs/HTMLCS.js` are:
 >       );
 ```
 
-#### IBM Equal Access
+#### Accessibility Checker
 
 The `ibm` tests require the `aceconfig.js` file.
 
@@ -532,6 +534,12 @@ Several Testaro tests make use of the `init()` function in the `procs/testaro` m
 
 You can add custom rules to the rules of any tool. Testaro provides a template, `data/template.js`, for the definition of a rule to be added. Once you have created a copy of the template with revisions, you can move the copy into the `testaro` directory and add an entry for your custom rule to the `evalRules` object in the `tests/testaro.js` file. Then your custom rule will act as a Testaro rule. Some `testaro` rules are simple enough to be fully specified in JSON files. You can use any of those as a template if you want to create a sufficiently simple custom rule, namely a rule whose prohibited elements are all and only the elements matching a CSS selector. More details about rule creation are in the `CONTRIBUTING.md` file.
 
+#### WallyAX
+
+If a `wax` test act is included in the job, an environment variable named `WAX_KEY` must exist, with your WallyAX API key as its value. You can get it from [WallyAX](mailto:technology@wallyax.com).
+
+The `wax` tool imposes a limit on the size of a page to be tested. If the page exceeds the limit, Testaro treats the page as preventing `wax` from performing its tests. The limit is less than 500,000 characters.
+
 #### WAVE
 
 If a `wave` test act is included in the job, an environment variable named `WAVE_KEY` must exist, with your WAVE API key as its value. You can get it from [WebAIM](https://wave.webaim.org/api/).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "45.0.10",
+  "version": "45.0.12",
   "description": "Run 1000 web accessibility tests from 10 tools and get a standardized report",
   "main": "index.js",
   "scripts": {
@@ -32,6 +32,7 @@
     "@siteimprove/alfa-playwright": "*",
     "@siteimprove/alfa-rules": "*",
     "@siteimprove/alfa-scraper": "*",
+    "@wally-ax/wax-dev": "*",
     "accessibility-checker": "*",
     "aslint-testaro": "*",
     "axe-playwright": "*",

package/procs/job.js

@@ -47,6 +47,7 @@ const tools = exports.tools = {
   nuVal: 'Nu Html Checker',
   qualWeb: 'QualWeb',
   testaro: 'Testaro',
+  wax: 'WallyAX',
   wave: 'WAVE',
 };
 
@@ -270,7 +271,14 @@ exports.doBy = async function(timeLimit, obj, fnName, fnArgs, noticePrefix) {
   // Start the function execution.
   const fnPromise = new Promise(async function(resolve) {
     fnResolver = resolve;
-    resolve(await obj[fnName](... fnArgs));
+    let fnResult;
+    try {
+      fnResult = await obj[fnName](... fnArgs);
+    }
+    catch(error) {
+      fnResult = `failed (error: ${error.message})`;
+    };
+    resolve(fnResult);
   });
   // Start a timer.
   const timerPromise = new Promise(resolve => {

package/procs/standardize.js

@@ -640,6 +640,49 @@ const convert = (toolName, data, result, standardResult) => {
       });
     }
   }
+  // wax
+  else if (toolName === 'wax' && result.violations && result.violations.length) {
+    // For each violation:
+    result.violations.forEach(violation => {
+      // Get its standard instance properties.
+      const element = violation.element.replace(/\s+/g, ' ');
+      const {message, description, severity} = violation;
+      const ordinalSeverity = ['Minor', 'Moderate', 'Major', 'Severe', 'Critical'].indexOf(severity);
+      const tagNameCandidate = element.replace(/^<| .*$/g, '');
+      const tagName = /^[a-zA-Z0-9]+$/.test(tagNameCandidate) ? tagNameCandidate.toUpperCase() : '';
+      let id = '';
+      const location = {};
+      if (tagName) {
+        const idTerm = element
+        .replace(/>.*$/, '')
+        .split(' ')
+        .slice(1)
+        .find(term => term.startsWith('id="'));
+        if (idTerm) {
+          const idCandidate = idTerm.slice(4, -1);
+          if (! idCandidate.includes('"')) {
+            id = idCandidate;
+            location.doc = 'dom';
+            location.type = 'selector';
+            location.spec = `#${id}`;
+          }
+        }
+      }
+      const instance = {
+        ruleID: message,
+        what: description,
+        ordinalSeverity,
+        tagName,
+        id,
+        location,
+        excerpt: element
+      };
+      // Add its ordinal severity to the standard result totals.
+      standardResult.totals[ordinalSeverity]++;
+      // Add the instance to the standard result.
+      standardResult.instances.push(instance);
+    });
+  }
   // wave
   else if (
     toolName === 'wave'

package/tests/wax.js

@@ -0,0 +1,93 @@
+/*
+  © 2024 CVS Health and/or one of its affiliates. All rights reserved.
+
+  Permission is hereby granted, free of charge, to any person obtaining a copy
+  of this software and associated documentation files (the "Software"), to deal
+  in the Software without restriction, including without limitation the rights
+  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+  copies of the Software, and to permit persons to whom the Software is
+  furnished to do so, subject to the following conditions:
+
+  The above copyright notice and this permission notice shall be included in all
+  copies or substantial portions of the Software.
+
+  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+  SOFTWARE.
+*/
+
+/*
+  wax
+  This test implements the WallyAX WAX Dev Testing Framework ruleset.
+*/
+
+// IMPORTS
+
+// Module to handle files.
+const fs = require('fs/promises');
+// Utility module.
+const {doBy} = require('../procs/job');
+// WAX
+const runWax = require('@wally-ax/wax-dev');
+const waxDev = {runWax};
+
+// FUNCTIONS
+
+// Conducts and reports the WAX tests.
+exports.reporter = async (page, report, actIndex, timeLimit) => {
+  // Initialize the act report.
+  let data = {};
+  let result = {};
+  // Run WAX.
+  const act = report.acts[actIndex];
+  const rules = act.rules || [];
+  const pageCode = await page.content();
+  const waxOptions = {
+    rules,
+    apiKey: process.env.WAX_KEY || ''
+  };
+  const actReport = await doBy(
+    timeLimit, waxDev, 'runWax', [pageCode, waxOptions], 'wax report retrieval'
+  );
+  // If WAX failed:
+  if (typeof actReport === 'string') {
+    // If the failure was a timeout:
+    if (actReport === 'timedOut') {
+      // Report this.
+      data.prevented = true;
+      data.error = 'Retrieval of result timed out';
+    }
+    // Otherwise, i.e. if the failure was not a timeout:
+    else {
+      // Report this.
+      data.prevented = true;
+      data.error = actReport;
+    }
+  }
+  // Otherwise, i.e. if WAX succeeded:
+  else {
+    // Populate the act report.
+    result = {
+      violations: actReport
+    }
+  }
+  // Return the results.
+  try {
+    JSON.stringify(data);
+  }
+  catch(error) {
+    const message = `ERROR: WAX result cannot be made JSON (${error.message.slice(0, 200)})`;
+    data = {
+      prevented: true,
+      error: message
+    };
+  }
+  return {
+    data,
+    result
+  };
+};