testaro 60.18.2 → 61.1.0

package/AGENTS.md

@@ -13,7 +13,7 @@
 
 - **Main dirs**: `/tests` (tool test definitions), `/procs` (shared procedures), `/validation` (test validators), root (core modules)
 - **Key files**: `run.js` (main executor), `actSpecs.js` (act specifications), `call.js` (CLI entry), `tests/testaro.js` (Testaro tool rules)
-- **Tools**: Integrates 11 a11y tools (Axe, Alfa, IBM Checker, QualWeb, ASLint, WAVE, WallyAX, Ed11y, HTML CodeSniffer, Nu Validator, Testaro)
+- **Tools**: Integrates 12 accessibility tools (Axe, Alfa, IBM Checker, QualWeb, ASLint, WAVE, WallyAX, Ed11y, HTML CodeSniffer, Nu Validator API, Nu Html Checker, Testaro)
 - **Data flow**: Jobs (JSON) → run.js → tool tests → reports (with standardized results)
 - **Env vars**: Required for WallyAX (`WAX_KEY`), WAVE (`WAVE_KEY`); optional `DEBUG`, `WAITS`, `JOBDIR`, `REPORTDIR`, `AGENT`
 

package/README.md

@@ -342,9 +342,9 @@ That means that a test act (i.e. an act with a `type` property having the value
 
 If a particular test act either must have or may have any other properties, those properties are specified in the `tools` property in `actSpecs.js`.
 
-When you include a `rules` property, you limit the tests of the tool that are performed or reported. For some tools (`alfa`, `axe`, `htmlcs`, `qualWeb`, `testaro`, and `wax`), only the specified tests are performed. Other tools (`aslint`, `ed11y`, `ibm`, `nuVal`, and `wave`) do not allow such a limitation, so, for those tools, all tests are performed but results are reported from only the specified tests.
+When you include a `rules` property, you limit the tests of the tool that are performed or reported. For some tools (`alfa`, `axe`, `htmlcs`, `qualWeb`, `testaro`, and `wax`), only the specified tests are performed. Other tools (`aslint`, `ed11y`, `ibm`, `nuVal`, `nuVnu`, and `wave`) do not allow such a limitation, so, for those tools, all tests are performed but results are reported from only the specified tests.
 
-The `nuVal`, `qualWeb`, and `testaro` tools require specific formats for the `rules` property. Those formats are described below in the sections about those tools.
+The `nuVal`, `nuVnu`, `qualWeb`, and `testaro` tools require specific formats for the `rules` property. Those formats are described below in the sections about those tools.
 
 ##### Examples of test acts
 
@@ -490,9 +490,9 @@ Experimentation indicates that the `ibm` tool emits untrappable errors for some
 
 #### Nu Html Checker
 
-The `nuVal` tool performs the tests of the Nu Html Checker.
+The `nuVal` and `nuVnu` tools perform the tests of the Nu Html Checker.
 
-Its `rules` argument is **not** an array of rule IDs, but instead is an array of rule _specifications_. A rule specification for `nuVal` is a string with the format `=ruleID` or `~ruleID`. The `=` prefix indicates that the rule ID is invariable. The `~` prefix indicates that the rule ID is variable, in which case the `ruleID` part of the specification is a matching regular expression, rather than the exact text of a message. This `rules` format arises from the fact that `nuVal` generates customized messages and does not accompany them with rule identifiers.
+Its `rules` argument is **not** an array of rule IDs, but instead is an array of rule _specifications_. A rule specification for `nuVal` or `nuVnu` is a string with the format `=ruleID` or `~ruleID`. The `=` prefix indicates that the rule ID is invariable. The `~` prefix indicates that the rule ID is variable, in which case the `ruleID` part of the specification is a matching regular expression, rather than the exact text of a message. This `rules` format arises from the fact that `nuVal` and `nuVnu` generate customized messages and do not accompany them with rule identifiers.
 
 #### QualWeb
 
@@ -731,7 +731,7 @@ This instance says that a `button` element violates a rule named `rule01`.
 
 The element has no `id` attribute to distinguish it from other `button` elements, but the tool describes its location. This tool uses an XPath to do that. Tools use various methods for location description, namely:
 
-- `line` (line number in the code of the page): Nu Html Checker
+- `code` (line, starting column, and ending column): Nu Html Checker (API and installed)
 - `selector` (CSS selector): Axe, QualWeb, WAVE
 - `xpath`: Alfa, ASLint, Equal Access
 - `box` (coordinates, width, and height of the element box): Editoria11y, Testaro
@@ -771,7 +771,7 @@ This standard format reflects some judgments. For example:
 
 - The `ordinalSeverity` property of an instance involves interpretation. Tools may report severity, certainty, priority, or some combination of those. They may use ordinal or metric quantifications. If they quantify ordinally, their scales may have more or fewer than 4 ranks. Testaro coerces each tool’s severity, certainty, and/or priority classification into a 4-rank ordinal classification. This classification is deemed to express the most common pattern among the tools.
 - The `tagName` property of an instance may not always be obvious, because in some cases the rule being tested for requires a relationship among more than one element (e.g., “An X element may not have a Y element as its parent”).
-- The `ruleID` property of an instance is a matching rule if the tool issues a message but no rule identifier for each instance. The `nuVal` tool does this. In this case, Testaro is classifying the messages into rules.
+- The `ruleID` property of an instance is a matching rule if the tool issues a message but no rule identifier for each instance. The `nuVal` and `nuVnu` tools do this. In this case, Testaro is classifying the messages into rules.
 - The `ruleID` property of an instance may reclassify tool rules. For example, if a tool rule covers multiple situations that are dissimilar, that rule may be split into multiple rules with distinct `ruleID` properties.
 
 You are not dependent on the judgments incorporated into the standard format, because Testaro can give you the original reports from the tools as the `result` property of a `test` act.

package/{VALIDATION_README.md → VALIDATION.md}

@@ -1,4 +1,4 @@
-# Validation README
+# Validation
 
 Quick notes to run the Testaro validation tests locally (Windows PowerShell):
 

package/actSpecs.js

@@ -133,7 +133,7 @@ exports.actSpecs = {
       {
         which: [true, 'string', 'isTest', 'tool name'],
         launch: [false, 'object', '', 'new target, browserID, and/or what, if any'],
-        rules: [false, 'array', 'areStrings', 'rule IDs or (for testaro or nuVal) specifications, if not all']
+        rules: [false, 'array', 'areStrings', 'rule IDs or (for testaro, nuVal, or nuVnu) specifications, if not all']
       }
     ],
     text: [

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "60.18.2",
+  "version": "61.1.0",
   "description": "Run 1000 web accessibility tests from 11 tools and get a standardized report",
   "main": "index.js",
   "scripts": {
@@ -48,7 +48,8 @@
     "playwright-dompath": "*",
     "playwright-extra": "*",
     "playwright-extra-plugin-stealth": "*",
-    "puppeteer-extra-plugin-stealth": "*"
+    "puppeteer-extra-plugin-stealth": "*",
+    "vnu-jar": "*"
   },
   "devDependencies": {
     "dom-accessibility-api": "*",

package/procs/job.js

@@ -31,7 +31,8 @@ const tools = exports.tools = {
   ed11y: 'Editoria11y',
   htmlcs: 'HTML CodeSniffer WCAG 2.1 AA ruleset',
   ibm: 'IBM Accessibility Checker',
-  nuVal: 'Nu Html Checker',
+  nuVal: 'Nu Html Checker API',
+  nuVnu: 'Nu Html Checker',
   qualWeb: 'QualWeb',
   testaro: 'Testaro',
   wax: 'WallyAX',

package/procs/standardize.js

@@ -30,31 +30,24 @@ const cap = rawString => {
 };
 // Returns whether an id attribute value is valid without character escaping.
 const isBadID = id => /[^-\w]|^\d|^--|^-\d/.test(id);
-// Returns the tag name and the value of an id attribute from a substring of HTML code.
+// Returns a tag name and the value of an id attribute from a substring of HTML code.
 const getIdentifiers = code => {
-  let tagName = '';
-  let id = '';
-  // If the substring includes the start tag of an element:
-  if (code && typeof code === 'string' && code.length && /<\s*[a-zA-Z]/.test(code)) {
-    // Get the first start tag in the substring.
-    const startTag = code.replace(/^.*?<(?=[a-zA-Z])/s, '').replace(/[^a-zA-Z0-9].*$/s, '').trim();
-    // If it exists:
-    if (startTag && startTag.length) {
-      // Get its tag name, upper-cased.
-      tagName = startTag.replace(/\s.*$/s, '').toUpperCase();
-      // Get the value of the id attribute of the start tag, if any.
-      const idArray = startTag.match(/\sid="([^"<>]+)"/);
-      if (idArray && idArray.length === 2) {
-        id = idArray[1];
-      }
-      // If the id value is invalid without character escaping:
-      if (isBadID(id)) {
-        // Remove it.
-        id = '';
-      }
-    }
+  // Normalize the code.
+  code = code.replace(/\s+/g, ' ').replace(/\\"/g, '"');
+  // Get the first start tag of an element, if any.
+  const startTagData = code.match(/^.*?< ?([^>]*)/);
+  // If there is any:
+  if (startTagData) {
+    // Get the tag name.
+    const tagNameData = startTagData[1].match(/^[A-Za-z]+/);
+    const tagName = tagNameData ? tagNameData[0].toUpperCase() : '';
+    // Get the value of the id attribute, if any.
+    const idData = startTagData[1].match(/ id="([^"]+)"/);
+    const id = idData ? idData[1] : '';
+    // Return the tag name and the value of the id attribute, if any.
+    return [tagName, id];
   }
-  return [tagName, id];
+  return ['', ''];
 };
 /*
   Differentiates some rule IDs of aslint.
@@ -221,30 +214,77 @@ const doNuVal = (result, standardResult, docType) => {
   const items = result[docType] && result[docType].messages;
   if (items && items.length) {
     items.forEach(item => {
-      const identifiers = getIdentifiers(item.extract);
-      if (! identifiers[0] && item.message) {
-        const tagNameLCArray = item.message.match(
+      const {extract, firstColumn, lastColumn, lastLine, message, subType, type} = item;
+      const identifiers = getIdentifiers(extract);
+      if (! identifiers[0] && message) {
+        const tagNameLCArray = message.match(
           /^Element ([^ ]+)|^An (img) element| (meta|script) element| element (script)| tag (script)/
         );
         if (tagNameLCArray && tagNameLCArray[1]) {
           identifiers[0] = tagNameLCArray[1].toUpperCase();
         }
       }
-      // Include the message twice. A scoring procedure may replace the ruleID with a pattern.
+      let spec = '';
+      const locationSegments = [lastLine, firstColumn, lastColumn];
+      if (locationSegments.every(segment => typeof segment === 'number')) {
+        spec = locationSegments.join(':');
+      }
       const instance = {
-        ruleID: item.message,
-        what: item.message,
+        ruleID: message,
+        what: message,
         ordinalSeverity: -1,
         tagName: identifiers[0],
         id: identifiers[1],
         location: {
           doc: docType === 'pageContent' ? 'dom' : 'source',
-          type: 'line',
-          spec: item && item.lastLine && item.lastLine.toString() || ''
+          type: 'code',
+          spec
         },
-        excerpt: cap(item.extract)
+        excerpt: cap(extract)
+      };
+      if (type === 'info' && subType === 'warning') {
+        instance.ordinalSeverity = 0;
+      }
+      else if (type === 'error') {
+        instance.ordinalSeverity = subType === 'fatal' ? 3 : 2;
+      }
+      standardResult.instances.push(instance);
+    });
+  }
+};
+// Converts issue instances from a nuVnu document type.
+const doNuVnu = (result, standardResult, docType) => {
+  const items = result[docType] && result[docType].messages;
+  if (items && items.length) {
+    items.forEach(item => {
+      const {extract, firstColumn, lastColumn, lastLine, message, subType, type} = item;
+      const identifiers = getIdentifiers(extract);
+      if (! identifiers[0] && message) {
+        const tagNameLCArray = message.match(
+          /^Element ([^ ]+)|^An (img) element| (meta|script) element| element (script)| tag (script)/
+        );
+        if (tagNameLCArray && tagNameLCArray[1]) {
+          identifiers[0] = tagNameLCArray[1].toUpperCase();
+        }
+      }
+      let spec = '';
+      const locationSegments = [lastLine, firstColumn, lastColumn];
+      if (locationSegments.every(segment => typeof segment === 'number')) {
+        spec = locationSegments.join(':');
+      }
+      const instance = {
+        ruleID: message,
+        what: message,
+        ordinalSeverity: -1,
+        tagName: identifiers[0],
+        id: identifiers[1],
+        location: {
+          doc: docType === 'pageContent' ? 'dom' : 'source',
+          type: 'code',
+          spec
+        },
+        excerpt: cap(extract)
       };
-      const {type, subType} = item;
       if (type === 'info' && subType === 'warning') {
         instance.ordinalSeverity = 0;
       }
@@ -568,6 +608,15 @@ const convert = (toolName, data, result, standardResult) => {
       doNuVal(result, standardResult, 'rawPage');
     }
   }
+  // nuVnu
+  else if (toolName === 'nuVnu' && (result.pageContent || result.rawPage)) {
+    if (result.pageContent) {
+      doNuVnu(result, standardResult, 'pageContent');
+    }
+    if (result.rawPage) {
+      doNuVnu(result, standardResult, 'rawPage');
+    }
+  }
   // qualWeb
   else if (
     toolName === 'qualWeb'

package/run.js

@@ -83,7 +83,7 @@ const timeLimits = {
   alfa: 20,
   ed11y: 30,
   ibm: 30,
-  testaro: 200 + Math.round(6 * waits / 1000)
+  testaro: 150 + Math.round(6 * waits / 1000)
 };
 // Timeout multiplier.
 const timeoutMultiplier = Number.parseFloat(process.env.TIMEOUT_MULTIPLIER) || 1;

package/tests/nuVal.js

@@ -9,13 +9,8 @@
 
 /*
   nuVal
-  This tool subjects a page and its source to the Nu Html Checker, thereby testing scripted
-  content found only in the loaded page and erroneous content before the browser corrects it.
-  The API erratically replaces left and right double quotation marks with invalid UTF-8, which
-  appears as 2 or 3 successive instances of the replacement character (U+fffd). Therefore, this
-  test removes all such quotation marks and the replacement character. That causes
-  'Bad value “” for' to become 'Bad value  for'. Since the corruption of quotation marks is
-  erratic, no better solution is known.
+  This tool subjects a page and its source to the Nu Html Checker, thereby testing scripted content found only in the loaded page and erroneous content before the browser corrects it. The API erratically replaces left and right double quotation marks with invalid UTF-8, which appears as 2 or 3 successive instances of the replacement character (U+fffd). Therefore, this test removes all such quotation marks and the replacement character. That causes 'Bad value “” for' to become 'Bad value  for'. Since the corruption of quotation marks is erratic, no better solution is known.
+  This tool is the API version of the Nu Html Checker. It is an alternative to the vnu tool, which uses the same validator as an installed dependency. Each tool has advantages and disadvantages. The main advantage of nuVal is that it does not require the Testaro host to provide a Java virtual machine. The main disadvantage of nuVal is that it returns 502 status errors when the content being uploaded is larger than about 80,000 bytes. The main advantage of the vnu tool is that it can evaluate pages larger than about 80,000 bytes and pages reachable from the host that Testaro runs on even if not reachable from the public Internet.
 */
 
 // IMPORTS

package/tests/nuVnu.js

@@ -0,0 +1,131 @@
+/*
+  © 2022–2024 CVS Health and/or one of its affiliates. All rights reserved.
+  © 2025 Jonathan Robert Pool.
+
+  Licensed under the MIT License. See LICENSE file at the project root or
+  https://opensource.org/license/mit/ for details.
+
+  SPDX-License-Identifier: MIT
+*/
+
+/*
+  This tool subjects a page and its source to the Nu Html Checker, thereby testing scripted content found only in the loaded page and erroneous content before the browser corrects it. The API erratically replaces left and right double quotation marks with invalid UTF-8, which appears as 2 or 3 successive instances of the replacement character (U+fffd). Therefore, this test removes all such quotation marks and the replacement character. That causes 'Bad value “” for' to become 'Bad value  for'. Since the corruption of quotation marks is erratic, no better solution is known.
+  This tool is the installed version of the Nu Html Checker. It is an alternative to the nuVal tool, which uses the same validator as a web service of the World Wide Web Consortium (W3C). Each tool has advantages and disadvantages. The main advantage of the nuVnu tool is that it can evaluate pages larger than about 80,000 bytes and pages reachable from the host that Testaro runs on even if not reachable from the public Internet. The main advantage of nuVal is that it does not require the Testaro host to provide a Java virtual machine.
+*/
+
+// IMPORTS
+
+// Module to perform file operations.
+const fs = require('fs/promises');
+// Module to define the operating-system temporary-file directory.
+const os = require('os');
+// Module to run tests.
+const {vnu} = require('vnu-jar');
+// Module to get the document source.
+const {getSource} = require('../procs/getSource');
+
+// CONSTANTS
+
+const tmpDir = os.tmpdir();
+
+// FUNCTIONS
+
+// Conducts and reports the Nu Html Checker tests.
+exports.reporter = async (page, report, actIndex) => {
+  const act = report.acts[actIndex];
+  const {rules} = act;
+  // Get the browser-parsed page.
+  const pageContent = await page.content();
+  const pagePath = `${tmpDir}/nuVnu-page-${report.id}.html`;
+  // Save it.
+  await fs.writeFile(pagePath, pageContent);
+  // Get the source.
+  const sourceData = await getSource(page);
+  const data = {
+    docTypes: {
+      pageContent: {},
+      rawPage: {}
+    }
+  };
+  const result = {};
+  // If the source was not obtained:
+  if (sourceData.prevented) {
+    // Report this.
+    data.prevented = true;
+    data.error = sourceData.error;
+  }
+  // Otherwise, i.e. if it was obtained:
+  else {
+    const sourcePath = `${tmpDir}/nuVnu-source-${report.id}.html`;
+    // Save the source.
+    await fs.writeFile(sourcePath, sourceData.source);
+    const pageTypes = [['pageContent', pagePath], ['rawPage', sourcePath]];
+    // For each page type:
+    for (const page of pageTypes) {
+      let nuResult;
+      try {
+        // Get Nu Html Checker output on it.
+        const nuOutput = await vnu.check(['--format', 'json', '--stdout', page[1]]);
+        // Consider the JSON output to be the result.
+        nuResult = nuOutput;
+      }
+      // If any error was thrown:
+      catch (error) {
+        let errorMessage = error.message;
+        // If it was due to an incompatible Java version:
+        if (errorMessage.includes('Unsupported major.minor version')) {
+          // Revise the error messageand report this.
+          errorMessage = `Installed version of Java is incompatible. Details: ${errorMessage}`;
+          data.docTypes[page[0]].prevented = true;
+          data.docTypes[page[0]].error = errorMessage;
+        }
+        // Otherwise, i.e. if it was not due to an incompatible Java version:
+        else {
+          try {
+            // Treat the output as a JSON result reporting rule violations.
+            nuResult = JSON.parse(error.message);
+            // But, if parsing it as JSON fails:
+          } catch (error) {
+            // Report this.
+            data.docTypes[page[0]].prevented = true;
+            data.docTypes[page[0]].error = `Error getting result (${error.message.slice(0, 300)});`;
+          }
+        }
+      }
+      // Delete the temporary file.
+      await fs.unlink(page[1]);
+      // If a result with or without reported violations was obtained:
+      if (nuResult) {
+        // Delete left and right quotation marks and their erratic invalid replacements.
+        const nuResultClean = JSON.parse(JSON.stringify(nuResult).replace(/[\u{fffd}“”]/ug, ''));
+        result[page[0]] = nuResultClean;
+        // If there is a report and rules were specified:
+        if (! result[page[0]].error && rules && Array.isArray(rules) && rules.length) {
+          // Remove all messages except those specified.
+          result[page[0]].messages = result[page[0]].messages.filter(message => rules.some(rule => {
+            if (rule[0] === '=') {
+              return message.message === rule.slice(1);
+            }
+            else if (rule[0] === '~') {
+              return new RegExp(rule.slice(1)).test(message.message);
+            }
+            else {
+              console.log(`ERROR: Invalid nuVal rule ${rule}`);
+              return false;
+            }
+          }));
+        }
+      }
+    }
+    // If both page types prevented testing:
+    if (pageTypes.every(pageType => data.docTypes[pageType[0]].prevented)) {
+      // Report this.
+      data.prevented = true;
+      data.error = 'Both doc types prevented';
+    }
+  }
+  return {
+    data,
+    result
+  };
+};