testaro 70.1.0 → 71.0.0

package/AGENTS.md

@@ -13,9 +13,9 @@
 
 - **Main dirs**: `/tests` (tool test definitions), `/procs` (shared procedures), `/validation` (test validators), root (core modules)
 - **Key files**: `run.js` (main executor), `actSpecs.js` (act specifications), `call.js` (CLI entry), `tests/testaro.js` (Testaro tool rules)
-- **Tools**: Integrates 12 accessibility tools (Axe, Alfa, IBM Checker, QualWeb, ASLint, WAVE, WallyAX, Ed11y, HTML CodeSniffer, Nu Validator API, Nu Html Checker, Testaro)
+- **Tools**: Integrates these accessibility tools: Axe, Alfa, IBM Checker, QualWeb, ASLint, WAVE, Ed11y, HTML CodeSniffer, Nu Html Checker API, Nu Html Checker, Testaro
 - **Data flow**: Jobs (JSON) → run.js → tool tests → reports (with standardized results)
-- **Env vars**: Required for WallyAX (`WAX_KEY`), WAVE (`WAVE_KEY`); optional `DEBUG`, `WAITS`, `JOBDIR`, `REPORTDIR`, `AGENT`
+- **Env vars**: Required for WAVE (`WAVE_KEY`); optional `DEBUG`, `WAITS`, `JOBDIR`, `REPORTDIR`, `AGENT`
 
 ## Code Style
 

package/README.md

@@ -57,12 +57,11 @@ Testaro can perform tests of these _tools_:
 - [Nu Html Checker](https://github.com/validator/validator) (World Wide Web Consortium)
 - [QualWeb](https://www.npmjs.com/package/@qualweb/core) (University of Lisbon)
 - [Testaro](https://www.npmjs.com/package/testaro) (CVS Health)
-- [WallyAX](https://www.npmjs.com/package/@wally-ax/wax-dev) (Wally Solutions)
 - [WAVE](https://wave.webaim.org/api/) (WebAIM)
 
 For the tools that are open-source, the identified organizations are their principal or original sponsors.
 
-As shown, Testaro is not only an integrator but also one of the 11 integrated tools. That is because it provides about 50 tests of its own, mostly to complement tests provided by the other 10 tools. Some of those Testaro tests are designed to act as approximate alternatives to tests of vulnerable, restricted, or no longer available tools. In all such cases the Testaro tests are independently designed and implemented, without reference to the code of the tests that inspired them.
+As shown, Testaro is not only an integrator but also one of the integrated tools. That is because it provides about 50 tests of its own, mostly to complement tests provided by the other tools. Some of those Testaro tests are designed to act as approximate alternatives to tests of vulnerable, restricted, or no longer available tools. In all such cases the Testaro tests are independently designed and implemented, without reference to the code of the tests that inspired them.
 
 ## Concepts and terms
 
@@ -121,8 +120,6 @@ PUPPETEER_DISABLE_HEADLESS_WARNING=true
 WAITS=0
 # API key to enable the WAVE tool.
 WAVE_KEY=yourwavekey (get it from [WebAim](https://wave.webaim.org/api/)).
-# `proTestKit` API key to enable the `npm Package` of the WallyAX tool.
-WAX_KEY=yourwaxkey (get it from [WallyAX](https://account.wallyax.com/?ref_app=Developer&app_type=npm)).
 #----------------------------
 # When Testaro listens for new jobs in a directory:
 # Directory where it listens for them.
@@ -332,7 +329,7 @@ Testaro uses the following techniques to make the tools calculate XPaths:
 - `alfa` and `aslint`: They report XPaths, so Testaro needs only to normalize them.
 - `ed11y`: Testaro adds it and a `window.getXPath` method to the page; when the tool reports an element, Testaro computes its XPath.
 - `wave`: It reports a selector for each element; Testaro finds each element in the page via its selector and executes `window.getXPath` on the element.
-- `axe`, `htmlcs`, `ibm`, `nuVal`, `nuVnu`, `qualWeb`, `wax`: Testaro adds `data-xpath` attributes to all elements; the tools include code excerpts, with the `data-expath` attributes, in the reported violations.
+- `axe`, `htmlcs`, `ibm`, `nuVal`, `nuVnu`, `qualWeb`: Testaro adds `data-xpath` attributes to all elements; the tools include code excerpts, with the `data-expath` attributes, in the reported violations.
 - `testaro`: Testaro designs each of its own tests to report element XPaths.
 
 By attaching a catalog entry to each reported element, Testaro allows an application that uses Testaro to tell users, for any particular HTML element, which tools ascribed violations of which rules to that element. An application could, for example, use a screenshot or a text-fragment link or could ask the user to paste the XPath into a browser developer tool.
@@ -507,12 +504,6 @@ Unlike any other tool, the `testaro` tool requires a `stopOnFail` property, whic
 
 Tests of the `testaro` tests (i.e. _validation_) can be performed as documented in the `VALIDATION.md` file.
 
-### WallyAX
-
-If a `wax` test act is included in the job, an environment variable named `WAX_KEY` must exist, with your WallyAX API key as its value. You can obtain it from [WallyAX](https://account.wallyax.com/?ref_app=Developer&app_type=npm).
-
-The `wax` tool imposes a limit on the size of a page to be tested. If the page exceeds the limit, Testaro treats the page as preventing `wax` from performing its tests. The limit is less than 500,000 characters.
-
 ### WAVE
 
 If a `wave` test act is included in the job, the WAVE tests will be performed either by the subscription API or by the stand-alone API.

package/UPGRADES.md

@@ -789,7 +789,7 @@ testaro-monorepo/
 
 Comments on the above suggestions:
 
-The `nuVal` tool is one of three (also `wave` and `wax`) that operate as APIs, with the substantial disadvantage that they can test only publicly reachable pages. The `nuVal` tool could be brought into the package (see How to to run your own copy of the [Nu Html Checker](@https://validator.w3.org/nu/about.html)). Until then, it seems to me that it avoids presenting incompatibility problems, so I believe another tool would be better to start with. But that decision could await the start of the upgrade project.
+The `nuVal` tool is one of two (also `wave`) that operate as APIs, with the substantial disadvantage that they can test only publicly reachable pages. The `nuVal` tool could be brought into the package (see How to to run your own copy of the [Nu Html Checker](@https://validator.w3.org/nu/about.html)). Until then, it seems to me that it avoids presenting incompatibility problems, so I believe another tool would be better to start with. But that decision could await the start of the upgrade project.
 
 I forgot to mention the modernization of validation. I am using that term in place of unit, integration, and end-to-end testing, because the entire project is a testing project and it gets confusing when the word “test” can refer both to the primary functionality of the project and the verification that the project works correctly. Currently validation is limited to the Testaro tool’s 55 or so tests. The idea behind that is that this project treats the 10 external tools like oracles that make claims, but these claims are inherently suspect, so for any tool there are 10 others that may perform their own version of any test. When tools disagree, Testaro does not try to decide which is correct. Instead, it uses Testilo to find similar tests of other tools and collect the results together so users can experience the disagreements and decide for themselves which tools are correct. Such user judgments can be collected by the UI and lead to deprecations by Testilo so that dubious tests are disregarded. Validation of the Testaro tests is currently exposed in a nonstandard way, and I want to consider converting it to a widely accepted interface.
 
@@ -809,7 +809,6 @@ You're absolutely right about `nuVal`'s limitations. Since it requires publicly
 
 1. **`htmlcs`** - Pure JavaScript, no external dependencies, works on any HTML
 2. **`wave`** - Also API-based, so same limitations as `nuVal`
-3. **`wax`** - API-based as well
 4. **`axe`** - Self-contained, works on any page, excellent for establishing patterns
 
 **I recommend starting with `axe`** because:
@@ -844,7 +843,7 @@ interface ValidationSuite {
 
 // Tool-specific validation
 interface ToolValidation extends ValidationSuite {
-  tool: 'axe' | 'wave' | 'htmlcs' | 'nuVal' | 'wax' | 'alfa' | 'qualWeb' | 'ibm' | 'ed11y' | 'aslint';
+  tool: 'axe' | 'wave' | 'htmlcs' | 'nuVal' | 'alfa' | 'qualWeb' | 'ibm' | 'ed11y' | 'aslint';
   rules: RuleDefinition[];
 }
 ```
@@ -1050,7 +1049,7 @@ interface ValidationSuite {
 
 // Tool-specific validation
 interface ToolValidation extends ValidationSuite {
-  tool: 'axe' | 'wave' | 'htmlcs' | 'nuVal' | 'wax' | 'alfa' | 'qualWeb' | 'ibm' | 'ed11y' | 'aslint';
+  tool: 'axe' | 'wave' | 'htmlcs' | 'nuVal' | 'alfa' | 'qualWeb' | 'ibm' | 'ed11y' | 'aslint';
   rules: RuleDefinition[];
 }
 ```
@@ -1636,7 +1635,6 @@ testaro-monorepo/
 │   │   ├── ed11y/                      # Editoria11y package
 │   │   ├── nuval/                      # Nu Html Checker package
 │   │   ├── wave/                       # WAVE package
-│   │   ├── wax/                        # WallyAX package
 │   │   └── testaro-rules/              # Testaro custom rules package
 │   ├── validation/                     # Validation framework
 │   └── custom-tools/                   # Customer-specific tools
@@ -1936,7 +1934,6 @@ testaro-monorepo/
 │   │   ├── ed11y/                      # Editoria11y package
 │   │   ├── nuval/                      # Nu Html Checker package
 │   │   ├── wave/                       # WAVE package
-│   │   ├── wax/                        # WallyAX package
 │   │   └── testaro-rules/              # Testaro custom rules package
 │   ├── validation/                     # Validation framework
 │   └── custom-tools/                   # Customer-specific tools
@@ -2968,7 +2965,6 @@ One example of the elapsed times (in seconds) for tool execution on a relatively
   "alfa": 13,
   "qualWeb": 13,
   "aslint": 7,
-  "wax": 7,
   "axe": 6,
   "ed11y": 5,
   "nuVal": 4,

package/netWatch.js

@@ -76,7 +76,7 @@ exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) =
       // Configure the next check.
       const logStart = `Requested job from ${jobHost} and got `;
       // Perform it.
-      await new Promise(resolve => {
+      await new Promise(async resolve => {
         try {
           const client = jobURL.protocol === 'https:' ? httpsClient : httpClient;
           // Request a job.
@@ -116,7 +116,7 @@ exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) =
                   await wait(1000 * intervalInSeconds);
                   resolve(true);
                 }
-                // Otherwise, if a job was received:
+                // Otherwise, if it is a job:
                 else if (id) {
                   // Check it for validity.
                   const jobValidity = isValidJob(contentObj);
@@ -128,6 +128,8 @@ exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) =
                       error: jobValidity.error
                     }, response);
                     console.log(`${logStart}invalid job (${jobValidity.error})`);
+                    // Wait for the specified interval.
+                    await wait(1000 * intervalInSeconds);
                     resolve(true);
                   }
                   // Otherwise, i.e. if it is valid:
@@ -164,8 +166,8 @@ exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) =
                         .on('error', async error => {
                           // Report this.
                           console.log(`${reportLogStart}error message ${error.message}\n`);
-                          // Stop checking.
-                          abort = true;
+                          // Wait for the specified interval.
+                          await wait(1000 * intervalInSeconds);
                           resolve(true);
                         })
                         // If the response delivers data:
@@ -194,6 +196,8 @@ exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) =
                           // Free the memory used by the job and the report.
                           contentObj = {};
                           responseJSON = '';
+                          // Wait for the specified interval.
+                          await wait(1000 * intervalInSeconds);
                           resolve(true);
                         });
                       })
@@ -205,8 +209,8 @@ exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) =
                         console.log(
                           `ERROR ${error.code} in report submission: ${reportLogStart}error message ${error.message}\n`
                         );
-                        // Stop checking.
-                        abort = true;
+                        // Wait for the specified interval.
+                        await wait(1000 * intervalInSeconds);
                         resolve(true);
                       })
                       // Finish submitting the report.
@@ -214,8 +218,8 @@ exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) =
                     }
                     catch(error) {
                       console.log(`ERROR performing job ${id} (${error.message})`);
-                      // Stop checking.
-                      abort = true;
+                      // Wait for the specified interval.
+                      await wait(1000 * intervalInSeconds);
                       resolve(true);
                     }
                   }
@@ -224,6 +228,8 @@ exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) =
                 else {
                   // Report it.
                   console.log(`${logStart}${JSON.stringify(contentObj, null, 2)}`);
+                  // Wait for the specified interval.
+                  await wait(1000 * intervalInSeconds);
                   resolve(true);
                 }
               }
@@ -231,8 +237,8 @@ exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) =
               catch(error) {
                 // Report this.
                 console.log(`ERROR: ${logStart}status ${response.statusCode}, error message ${error.message}, and non-JSON response ${content.slice(0, 1000)}\n`);
-                // Stop checking.
-                abort = true;
+                // Wait for the specified interval.
+                await wait(1000 * intervalInSeconds);
                 resolve(true);
               };
             });
@@ -243,30 +249,24 @@ exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) =
             if (error.code && error.code.includes('ECONNREFUSED')) {
               // Report this.
               console.log(`${logStart}no connection`);
-              // Stop checking.
-              abort = true;
             }
             // Otherwise, if it was a DNS failure:
             else if (error.code && error.code.includes('ENOTFOUND')) {
               // Report this.
               console.log(`${logStart}no domain name resolution`);
-              // Stop checking.
-              abort = true;
             }
             // Otherwise, if it was any other error with a message:
             else if (error.message) {
               // Report this.
               console.log(`ERROR: ${logStart}got error message ${error.message.slice(0, 200)}`);
-              // Stop checking.
-              abort = true;
             }
             // Otherwise, i.e. if it was any other error with no message:
             else {
               // Report this.
               console.log(`ERROR: ${logStart}got an error with no message`);
-              // Stop checking.
-              abort = true;
             }
+            // Wait for the specified interval.
+            await wait(1000 * intervalInSeconds);
             resolve(true);
           })
           // Finish sending the job request.
@@ -278,8 +278,8 @@ exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) =
         catch(error) {
           // Report this.
           console.log(`ERROR requesting a network job (${error.message})`);
-          // Stop checking.
-          abort = true;
+          // Wait for the specified interval.
+          await wait(1000 * intervalInSeconds);
           resolve(true);
         }
       });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "70.1.0",
+  "version": "71.0.0",
   "description": "Run 1000 web accessibility tests from 11 tools and get a standardized report",
   "main": "index.js",
   "scripts": {
@@ -38,7 +38,6 @@
     "@siteimprove/alfa-act": "*",
     "@siteimprove/alfa-playwright": "*",
     "@siteimprove/alfa-rules": "*",
-    "@wally-ax/wax-dev": "npm:@jrpool/wax-dev@^1.0.0",
     "accessibility-checker": "*",
     "aria-query": "*",
     "aslint-testaro": "*",

package/procs/doActs.js

@@ -56,8 +56,7 @@ const timeLimits = {
   nuVal: 30,
   nuVnu: 20,
   qualWeb: 45,
-  testaro: 150 + Math.round(6 * waits / 1000),
-  wax: 35
+  testaro: 150 + Math.round(6 * waits / 1000)
 };
 // Timeout multiplier.
 const timeoutMultiplier = Number.parseFloat(process.env.TIMEOUT_MULTIPLIER) || 1;

package/procs/doTestAct.js

@@ -54,8 +54,7 @@ const xPathNeeds = {
   nuVal: 'attribute',
   nuVnu: 'attribute',
   qualWeb: 'attribute',
-  wave: 'script',
-  wax: 'attribute'
+  wave: 'script'
 };
 const accessibleNameNeeders = ['testaro'];
 

package/procs/job.js

@@ -35,7 +35,6 @@ const tools = exports.tools = {
   nuVnu: 'Html Checker',
   qualWeb: 'QualWeb',
   testaro: 'Testaro',
-  wax: 'WallyAX',
   wave: 'WAVE',
 };
 

package/patches/@wally-ax+wax-dev+0.2.62.patch

@@ -1,28 +0,0 @@
-diff --git a/node_modules/@wally-ax/wax-dev/src/runnerHtml.js b/node_modules/@wally-ax/wax-dev/src/runnerHtml.js
-index 1d84a0b..d9c3fc5 100644
---- a/node_modules/@wally-ax/wax-dev/src/runnerHtml.js
-+++ b/node_modules/@wally-ax/wax-dev/src/runnerHtml.js
-@@ -34,8 +34,8 @@ const runner = function (code, options) {
-           description,
-           severity,
-           code,
--          groupData = {}
-         } = item;
-+        const groupData = item.groupData || {};
- 
-         const cleanedItem = {};
-         if (element) cleanedItem.element = element;
-diff --git a/node_modules/@wally-ax/wax-dev/src/runnerUrl.js b/node_modules/@wally-ax/wax-dev/src/runnerUrl.js
-index 0e020a0..3c6850f 100644
---- a/node_modules/@wally-ax/wax-dev/src/runnerUrl.js
-+++ b/node_modules/@wally-ax/wax-dev/src/runnerUrl.js
-@@ -33,8 +33,8 @@ const runner = function (code, options) {
-             selector,
-             component,
-             issue_category_name,
--            groupData = {}
-           } = item;
-+          const groupData = item.groupData || {};
- 
-           const cleanedItem = {};
- 

package/tests/wax.js

@@ -1,133 +0,0 @@
-/*
-  © 2024 CVS Health and/or one of its affiliates. All rights reserved.
-  © 2025–2026 Jonathan Robert Pool.
-
-  Licensed under the MIT License. See LICENSE file at the project root or
-  https://opensource.org/license/mit/ for details.
-
-  SPDX-License-Identifier: MIT
-*/
-
-/*
-  wax
-  This test implements the WallyAX WAX Dev Testing Framework ruleset.
-*/
-
-// IMPORTS
-
-const {getAttributeXPath, getXPathCatalogIndex} = require('../procs/xPath');
-const runWax = require('@wally-ax/wax-dev');
-const waxDev = {runWax};
-
-// FUNCTIONS
-
-// Conducts and reports the WAX tests.
-exports.reporter = async (page, report, actIndex) => {
-  // Initialize the act report.
-  const data = {};
-  const result = {
-    nativeResult: {},
-    standardResult: {}
-  };
-  const standard = report.standard !== 'no';
-  // If standard results are to be reported:
-  if (standard) {
-    // Initialize the standard result.
-    result.standardResult = {
-      prevented: false,
-      totals: [0, 0, 0, 0],
-      instances: []
-    };
-  }
-  const act = report.acts[actIndex];
-  const rules = act.rules || [];
-  const pageCode = await page.content();
-  const waxOptions = {
-    rules,
-    apiKey: process.env.WAX_KEY || ''
-  };
-  try {
-    // Run WAX.
-    const actReport = await waxDev.runWax(pageCode, waxOptions);
-    // If WAX failed with a string report:
-    if (typeof actReport === 'string') {
-      // Report this.
-      data.prevented = true;
-      data.error = actReport;
-    }
-    // Otherwise, if the report is an array:
-    else if (Array.isArray(actReport)) {
-      // If it is an error report:
-      if (actReport.length === 1 && typeof actReport[0] === 'object' && actReport[0].error) {
-        // Report this.
-        data.prevented = true;
-        const {error} = actReport[0];
-        data.error = error;
-        console.log(`ERROR running wax: ${error}`);
-      }
-      // Otherwise, i.e. if it is a successful report:
-      else {
-        // Populate the native result with it.
-        result.nativeResult = actReport;
-        // If standard results are to be reported:
-        if (standard) {
-          const {standardResult} = result;
-          const {instances, totals} = standardResult;
-          actReport.forEach(violation => {
-            const ordinalSeverity = ['Minor', 'Moderate', '', 'Severe'].indexOf(violation.severity);
-            // Increment the applicable total of the standard result.
-            totals[ordinalSeverity]++;
-            // Initialize a standard instance.
-            const instance = {
-              ruleID: violation.message,
-              what: violation.description || violation.message,
-              ordinalSeverity,
-              count: 1
-            };
-            const {element} = violation;
-            // Get the path ID of the element from its data-xpath attribute.
-            const pathID = getAttributeXPath(element);
-            // If the acquisition succeeded:
-            if (pathID) {
-              // Add the catalog index to the standard instance.
-              instance.catalogIndex = getXPathCatalogIndex(report.catalog, pathID);
-            }
-            // Add the standard instance to the standard result.
-            instances.push(instance);
-          });
-        }
-      }
-    }
-    // Otherwise, if the report is a non-array object:
-    else if (typeof actReport === 'object') {
-      // If the response status was a system error:
-      if(actReport.responseCode === 500) {
-        // Report this.
-        data.prevented = true;
-        data.error = actReport.message || 'response status code 500';
-      }
-      // Otherwise, if there was an error:
-      else if (actReport.error) {
-        // Report this.
-        data.prevented = true;
-        data.error = actReport.error;
-      }
-      // In any other case:
-      else {
-        // Report a prevention.
-        data.prevented = true;
-        data.error = 'wax failure';
-      }
-    }
-  }
-  catch(error) {
-    const message = `ERROR running WallyAX (${error.message})`;
-    data.prevented = true;
-    data.error = message;
-    console.log(message);
-  }
-  return {
-    data,
-    result
-  };
-};