testaro 75.0.0 → 75.1.0

package/README.md

@@ -41,7 +41,7 @@ Testaro uses:
 - [Playwright](https://playwright.dev/) to launch browsers, perform user actions in them, and perform tests
 - [playwright-extra](https://www.npmjs.com/package/playwright-extra) and [puppeteer-extra-plugin-stealth](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to make a Playwright-controlled browser more indistinguishable from a human-operated browser and thus make its requests more likely to succeed
 - [playwright-dompath](https://www.npmjs.com/package/playwright-dompath) to retrieve XPaths of elements
-- [BlazeDiff](https://blazediff.dev/) to measure motion
+- [pixelmatch](https://www.npmjs.com/package/pixelmatch) to measure motion
 - [dotenv](https://www.npmjs.com/package/dotenv) to load environment variables
 
 Testaro can perform tests of these _tools_:
@@ -142,7 +142,8 @@ Here is a sample job, showing properties that you can set:
   id: 'healthcheck2611', // Job identifier
   what: 'monthly health check', // Job description
   strict: true, // Whether to reject redirections from the target URL
-  standard: 'also', // or 'only' or 'no' (whether to report a standard result)
+  standard: 'only', // Report native (no), standard (only), or both (also) results
+  imageColor: 0, // Color type (0, 2, 4, 6) of the page image, if one is to be created along with a catalog
   device: { // Device to emulate
     id: 'iPhone 8',
     windowOptions: {
@@ -288,13 +289,18 @@ A report is a job with information about the results of the performance of the j
 
 ### Whole-job data
 
-As Testaro performs a job, information about the job as a whole is inserted into the job. That information is organized into one or two properties:
+As Testaro performs a job, information about the job as a whole is inserted into the job. That information is organized into one, two, or three properties:
 
 - `jobData`: Facts about the performance of the job
 - `catalog`: A collection of data about the HTML elements of the target that are relevant to any test failures
+- `images`: A collection of page images captured during the job
+
+#### `jobData`
 
 Testaro inserts the `jobData` property into every job.
 
+#### `catalog`
+
 Testaro inserts the `catalog` property only into jobs that instruct Testaro to produce standard results. The catalog is an inventory of HTML elements in the DOM of the target.
 
 The `catalog` property has an object value. Here is an example:
@@ -336,6 +342,12 @@ In some cases no catalog entry can be found. The reasons may include:
 - The element is inside a `noscript` element and therefore not considered an element in the DOM.
 - The violation is not ascribed to a single element.
 
+#### `images`
+
+Testaro inserts an `images` array property if necessary to store page images in the report. If the job has an `imageColor` property with `0`, `2`, `4`, or `6` as its value and Testaro will insert a `catalog` property, then Testaro also creates a page image with that color type and makes its base64-encoded PNG the first item in the `images` array.
+
+There is a `shoot` act type that can be used to make additional page images during a job.
+
 ### Act data
 
 As Testaro performs the acts of a job, information about the result of each act is inserted into that act. For acts of type `test`, the added properties are:
@@ -542,6 +554,8 @@ Test targets employ mechanisms to prevent scraping, multiple requests within a s
 
 Some targets prohibit the execution of alien scripts unless the client can demonstrate that it is the requester of the page. Failure to provide that evidence results in the script being blocked and an error message being logged, saying “Refused to execute a script because its hash, its nonce, or unsafe-inline does not appear in the script-src directive of the Content Security Policy”. This mechanism affects tools that insert scripts into a target in order to test it. To comply with this requirement, Testaro obtains a _nonce_ from the response that serves the target. Then the file that runs the tool adds that nonce to the script as the value of a `nonce` attribute when it inserts its script into the target.
 
+Some targets have been found erratically to prevent the creation of page images. When page images have been created, during the `motion` test in `testaro` some targets have been found to prevent their comparison by BlazeDiff, but comparison by `pixelmatch` has succeeded. For this reason, although reportedly slower, `pixelmatch` is the library used for image comparison.
+
 ### Tool duplicativity
 
 Tools sometimes do redundant testing, in that two or more tools test for the same defects, although such duplications are not necessarily perfect. This fact creates problems:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "75.0.0",
+  "version": "75.1.0",
   "description": "Run 1300 web accessibility tests from 10 tools and get a standardized report",
   "main": "index.js",
   "scripts": {
@@ -30,7 +30,6 @@
   },
   "homepage": "https://github.com/jrpool/testaro#readme",
   "dependencies": {
-    "@blazediff/core": "*",
     "@qualweb/core": "*",
     "@qualweb/act-rules": "*",
     "@qualweb/wcag-techniques": "*",
@@ -43,6 +42,7 @@
     "aslint-testaro": "*",
     "axe-playwright": "*",
     "dotenv": "*",
+    "pixelmatch": "*",
     "playwright": "*",
     "playwright-dompath": "*",
     "playwright-extra": "*",

package/procs/catalog.js

@@ -25,6 +25,7 @@
 
 // Module to close and launch browsers.
 const {browserClose, launch} = require('./launch');
+const {shoot} = require('./shoot');
 
 // FUNCTIONS
 
@@ -43,7 +44,18 @@ exports.getCatalog = async report => {
     });
     // If the launch and navigation succeeded:
     if (page) {
+      // If a page image is required:
+      if ([0, 2, 4, 6].includes(report.imageColor)) {
+        // Create one and add it to the report.
+        console.log('Creating page image');
+        await shoot(page, report, {
+          exclusionSelector: '',
+          colorType: report.imageColor,
+          action: 'report'
+        });
+      }
       // Get a catalog of the elements in the page.
+      console.log('Creating catalog');
       const catalog = await page.evaluate(() => {
         const elements = Array.from(document.querySelectorAll('*'));
         // Initialize a catalog.

package/run.js

@@ -126,7 +126,9 @@ exports.doJob = async (job, opts = {}) => {
     });
     // If the job specifies a browser ID and a target and requires standardization:
     if (job.browserID && job.target && job.standard !== 'no') {
-      // Create a catalog of the target and add it to the report.
+      // Initialize a catalog so it precedes any page images in the report.
+      report.catalog = {};
+      // Add a catalog of the target, and a page image if required, to the report.
       report.catalog = await getCatalog(report);
     }
     // Perform the acts and revise the report.

package/testaro/motion.js

@@ -5,7 +5,7 @@
 
 /*
   motion
-  This test reports motion in a page by comparing the first and last of the screenshots previously made by the shoot0 and shoot1 tests.
+  This test reports motion in a page by making a page image and comparing it with the initial one, i.e. the one made by the catalog proc.
 
   For minimal accessibility, standards require motion to be brief, or else stoppable by the user. But stopping motion can be difficult or impossible, and, by the time a user manages to stop motion, the motion may have caused annoyance or harm. For superior accessibility, a page contains no motion until and unless the user authorizes it. The test reports a rule violation if any pixels differ between the screenshots. The larger the change fraction, the greater the ordinal severity.
 
@@ -15,54 +15,69 @@
 // IMPORTS
 
 const {getXPathCatalogIndex} = require('../procs/xPath');
-const fs = require('fs/promises');
-const blazediff = require('@blazediff/core').diff;
+const {shoot} = require('../procs/shoot');
+const pixelmatch = require('pixelmatch').default;
 const {PNG} = require('pngjs');
 
 // FUNCTIONS
 
 // Runs the test and returns the result.
-exports.reporter = async (_0, report, _1, _2, tmpDir) => {
+exports.reporter = async (page, report) => {
   // Initialize the totals and standard instances.
   const data = {};
   const totals = [0, 0, 0, 0];
   const standardInstances = [];
-  let violationWhat = '';
-  let ordinalSeverity = 0;
-  try {
-    // Get the screenshot PNG buffers made by the shoot0 and shoot1 tests.
-    let shoot0PNGBuffer = await fs.readFile(report.jobData.testaroShoot0);
-    let shoot1PNGBuffer = await fs.readFile(report.jobData.testaroShoot1);
-    // Delete the buffer files.
-    await fs.unlink(report.jobData.testaroShoot0);
-    await fs.unlink(report.jobData.testaroShoot1);
-    // If both buffers exist:
-    if (shoot0PNGBuffer && shoot1PNGBuffer) {
-      // Parse them into PNG objects.
-      let shoot0PNG = PNG.sync.read(shoot0PNGBuffer);
-      let shoot1PNG = PNG.sync.read(shoot1PNGBuffer);
-      const {width, height} = shoot0PNG;
+  // If the initial image exists:
+  if (report.images?.length) {
+    let violationWhat = '';
+    let ordinalSeverity = 0;
+    // Make an image with the same color type as the initial one and get its base64 encoding.
+    const png = await shoot(page, report, {
+      exclusionSelector: null,
+      colorType: report.imageColor,
+      action: 'return'
+    });
+    // If this succeeded:
+    if (png) {
+      // Parse both base64 encodings into PNG objects.
+      const initialPNG = PNG.sync.read(Buffer.from(report.images[0], 'base64'));
+      const finalPNG = PNG.sync.read(Buffer.from(png, 'base64'));
       // If their dimensions differ:
-      if (shoot1PNG.width !== width || shoot1PNG.height !== height) {
-        const fromSize = `${width}×${height}`;
-        const toSize = `${shoot1PNG.width}×${shoot1PNG.height}`;
+      if (finalPNG.width !== initialPNG.width || finalPNG.height !== initialPNG.height) {
+        const fromSize = `${initialPNG.width}×${initialPNG.height}`;
+        const toSize = `${finalPNG.width}×${finalPNG.height}`;
         // Describe the violation.
         violationWhat = `Page size changes spontaneously (from ${fromSize} to ${toSize})`;
       }
       // Otherwise, i.e. if their dimensions are identical:
       else {
-        // Get the count of differing pixels between the shots.
-        const pixelChanges = blazediff(shoot0PNG.data, shoot1PNG.data, null, width, height);
-        // Get the ratio of differing to all pixels as a percentage.
-        const changePercent = Math.round(100 * pixelChanges / (width * height));
-        // Free the memory used by screenshots.
-        shoot0PNG = shoot1PNG = shoot0PNGBuffer = shoot1PNGBuffer = null;
-        // If any pixels were changed:
-        if (pixelChanges) {
-          // Describe the violation.
-          violationWhat = `Content changes spontaneously (${changePercent}% of pixels changed)`;
-          // Get the ordinal severity from the fractional pixel change.
-          ordinalSeverity = Math.floor(Math.min(3, 0.4 * Math.sqrt(changePercent)));
+        // Get the count of differing pixels between the images, using the default sensitivity.
+        try {
+          const pixelChanges = pixelmatch(
+            initialPNG.data,
+            finalPNG.data,
+            null,
+            initialPNG.width,
+            initialPNG.height,
+            {
+              threshold: 0.1
+            }
+          );
+          // Get the ratio of differing to all pixels as a percentage.
+          const changePercent = Math.round(
+            100 * pixelChanges / (initialPNG.width * initialPNG.height)
+          );
+          // If any pixels were changed:
+          if (pixelChanges) {
+            // Describe the violation.
+            violationWhat = `Content changes spontaneously (${changePercent}% of pixels changed)`;
+            // Get the ordinal severity from the fractional pixel change.
+            ordinalSeverity = Math.floor(Math.min(3, 0.4 * Math.sqrt(changePercent)));
+          }
+        } catch (err) {
+          console.log(`pixelmatch error: ${err.message}, ${err.stack}`);
+          data.prevented = true;
+          data.error = `Pixel comparison failed: ${err.message}`;
         }
       }
       // If there was a violation:
@@ -79,18 +94,18 @@ exports.reporter = async (_0, report, _1, _2, tmpDir) => {
         });
       }
     }
-    // Otherwise, i.e. if they do not both exist:
+    // Otherwise, i.e. if it failed:
     else {
       // Report this.
       data.prevented = true;
-      data.error = 'At least 1 screenshot missing';
+      data.error = 'Image creation failed';
     }
   }
-  // If getting or deleting either buffer file failed:
-  catch(error) {
+  // Otherwise, i.e. if the initial image does not exist:
+  else {
     // Report this.
     data.prevented = true;
-    data.error = `Screenshot file error (${error.message})`;
+    data.error = 'Initial image missing';
   }
   // Return the result.
   return {

package/tests/testaro.js

@@ -23,15 +23,6 @@ const {launch} = require('../procs/launch');
 
 // Metadata of all rules in default execution order.
 const allRules = [
-  {
-    id: 'shoot0',
-    what: 'first page screenshot',
-    contaminates: false,
-    needsAccessibleName: false,
-    needsTmpDir: true,
-    timeOut: 5,
-    defaultOn: true
-  },
   {
     id: 'adbID',
     what: 'elements with ambiguous or missing referenced descriptions',
@@ -328,21 +319,11 @@ const allRules = [
     timeOut: 5,
     defaultOn: true
   },
-  {
-    id: 'shoot1',
-    what: 'second page screenshot',
-    contaminates: false,
-    needsAccessibleName: false,
-    needsTmpDir: true,
-    timeOut: 5,
-    defaultOn: true
-  },
   {
     id: 'motion',
-    what: 'motion without user request, measured across tests',
+    what: 'motion without user request',
     contaminates: false,
     needsAccessibleName: false,
-    needsTmpDir: true,
     timeOut: 5,
     defaultOn: true
   },

package/testaro/shoot0.js

@@ -1,36 +0,0 @@
-/*
-  © 2025–2026 Jonathan Robert Pool.
-  Licensed under the MIT License. See LICENSE file for details.
-*/
-
-/*
-  shoot0
-  This test makes and saves the first of two screenshots.
-*/
-
-// IMPORTS
-
-const {shoot} = require('../procs/shoot');
-
-// FUNCTIONS
-
-// Makes and saves the first screenshot.
-exports.reporter = async (page, report) => {
-  // Make and save the screenshot.
-  const pngPath = await shoot(page, report, {
-    exclusionSelector: '',
-    colorType: 0,
-    action: 'file'
-  });
-  // If this succeeded:
-  if (pngPath) {
-    // Add the file path to the report.
-    report.jobData.testaroShoot0 = pngPath;
-  }
-  // Return whether the screenshot was prevented.
-  return {
-    data: {
-      prevented: ! pngPath
-    }
-  };
-};

package/testaro/shoot1.js

@@ -1,41 +0,0 @@
-/*
-  © 2025–2026 Jonathan Robert Pool.
-  Licensed under the MIT License. See LICENSE file for details.
-*/
-
-/*
-  shoot1
-  This test makes and saves the second of two screenshots. It aborts if the first screenshot was prevented.
-*/
-
-// IMPORTS
-
-const fs = require('fs/promises');
-const {shoot} = require('../procs/shoot');
-
-// FUNCTIONS
-
-// Make and save the second screenshot.
-exports.reporter = async (page, report) => {
-  let pngPath = '';
-  // If there is a shoot0 file:
-  if (report.jobData.testaroShoot0) {
-    // Make and save the screenshot.
-    pngPath = await shoot(page, report, {
-      exclusionSelector: null,
-      colorType: 0,
-      action: 'file'
-    });
-    // If this succeeded:
-    if (pngPath) {
-      // Add the file path to the report.
-      report.jobData.testaroShoot1 = pngPath;
-    }
-  }
-  // Return whether the screenshot was prevented.
-  return {
-    data: {
-      prevented: ! pngPath
-    }
-  };
-};