testaro 21.0.0 → 23.0.0

package/README.md

@@ -144,14 +144,9 @@ Here is an example of a job:
     {
       type: 'launch',
       which: 'chromium',
+      url: 'https://www.w3c.org',
       what: 'Chromium browser'
     },
-    {
-      type: 'url',
-      which: 'https://www.w3c.org',
-      what: 'World Wide Web Consortium',
-      id: 'w3c'
-    },
     {
       type: 'test',
       which: 'alfa',
@@ -174,9 +169,8 @@ Here is an example of a job:
 }
 ```
 
-This job contains three _acts_, telling Testaro to:
-1. open a page in the Chromium browser
-1. navigate to a specified URL
+This job contains two _acts_, telling Testaro to:
+1. open a page in the Chromium browser and navigate to a specified URL
 1. perform two of the tests of the `alfa` tool (the tests for rules `r25` and `r71`) on that URL
 
 Job properties:
@@ -306,7 +300,7 @@ Each act object has a `type` property and optionally has a `name` property (used
 
 #### Act sequence
 
-The first two acts in any job have the types `launch` and `url`, respectively, as shown in the example above. They launch a browser and then use it to visit a URL.
+The first act in any job has the type `launch`, as shown in the example above. It launches a browser and then uses it to visit a URL.
 
 #### Act types
 
@@ -338,11 +332,9 @@ When the texts of multiple elements of the same type will contain the same `whic
 
 ##### Navigations
 
-An example of a **navigation** is the act of type `url` above.
+An example of a **navigation** is the act of type `launch` above.
 
-Once you have included a `url` act in a job, you do not need to add more `url` acts unless you want the browser to visit a different URL or revisit the same URL.
-
-If any act alters the page, you can restore the page to its original state for the next act by inserting new `launch` and `url` acts (and, if necessary, additional page-specific acts) between them.
+If any act alters the page, you can restore the page to its original state for the next act by inserting a new `launch` act (and, if necessary, additional page-specific acts) between them.
 
 Another navigation example is:
 
@@ -585,7 +577,11 @@ The `rules` property for `testaro` is an array whose first item is either `'y'`
 
 The `testaro` tool (like the `ibm` tool) has a `withItems` property. If you set it to `false`, the `standardResult` object of `testaro` will contain an `instances` property with summaries that identify issues and instance counts. If you set it to `true`, some of the instances will be itemized.
 
-Unlike any other tool, the `testaro` tools requires a `stopOnFail` property, which specifies whether a failure to conform to any rule (i.e. any value of `totals` other than `[0, 0, 0, 0]`) should terminate the execution of tests for the remaining rules.
+Unlike any other tool, the `testaro` tool requires a `stopOnFail` property, which specifies whether a failure to conform to any rule (i.e. any value of `totals` other than `[0, 0, 0, 0]`) should terminate the execution of tests for the remaining rules.
+
+Warnings in the `testaro/hover.js`, `testaro/motion.js`, and `procs/visChange.js` files advise you to avoid launching particular browser types for the performance of particular Testaro tests.
+
+Several Testaro tests make use of the `init()` function in the `procs/testaro` module. That function samples elements if the population of elements to be tested is larger than 100. The purpose is to achieve reasonable performance. The sampling overweights elements near the beginning of a page, because of the tendency of that location to have important and atypical elements.
 
 You can add custom rules to the rules of any tool. Testaro provides a template, `data/template.js`, for the definition of a rule to be added. Once you have created a copy of the template with revisions, you can move the copy into the `testaro` directory and add an entry for your custom rule to the `evalRules` object in the `tests/testaro.js` file. Then your custom rule will act as a Testaro rule.
 
@@ -618,6 +614,26 @@ This act checks the result of the previous act to determine whether its `result.
 
 A `next` act can use a `next` property instead of a `jump` property. The value of the `next` property is an act name. It tells Testaro to continue performing acts starting with the act having that value as the value of its `name` property.
 
+#### Browser types
+
+After any act in a job, you can change the browser type by inserting a `launch` act. One reason for specifying a particular browser type is that particular tests have different results with different browser types. Another is that you may wish to perform tests with more than a single browser type.
+
+The warning comments in the `testaro/hover.js` and `testaro/motion.js` files state that those tests operate correctly only with the `webkit` browser type.
+
+When you want to run some tests of a tool with one browser type and other tests of the same tool with another browser type, you can do so by splitting the rules into two test acts. For example, one test act can specify the rules as
+
+```javascript
+['y', 'r15', 'r54']
+```
+
+and the other test act can specify the rules as
+
+```javascript
+['n', 'r15', 'r54']
+```
+
+Before each test act, you can ensure that the latest `launch` act has specified the browser type to be used in that test act.
+
 #### `actSpecs` file
 
 ##### Introduction

package/actSpecs.js

@@ -27,9 +27,10 @@ exports.actSpecs = {
     launch: [
       'Launch a Playwright browser',
       {
-        which: [true, 'string', 'isBrowserType', '“chromium”, “firefox”, or “webkit”'],
-        what: [false, 'string', 'hasLength', 'comment'],
-        lowMotion: [false, 'boolean', '', 'set reduced-motion option if true']
+        which: [true, 'string', 'isBrowserType', 'chromium, firefox, or webkit'],
+        url: [true, 'string', 'isURL', 'initial URL to navigate to'],
+        lowMotion: [false, 'boolean', '', 'set reduced-motion option if true'],
+        what: [false, 'string', 'hasLength', 'comment']
       }
     ],
     link: [

package/data/template.js

@@ -13,12 +13,12 @@ const {init, report} = require('../procs/testaro');
 // Runs the test and returns the result.
 exports.reporter = async (page, withItems) => {
   // Initialize the locators and result.
-  const all = await init(page, 'body *');
+  const all = await init(page, 'body a');
   // For each locator:
   for (const loc of all.allLocs) {
     // Get whether its element violates the rule.
     const isBad = await loc.evaluate(el => {
-      const isViolator = el.tagName.length > 300;
+      const isViolator = ! el.href;
       return isViolator;
     });
     // If it does:

package/dirWatch.js

@@ -34,11 +34,11 @@ const reWatch = () => {
     }
     if (code === 0) {
       console.log('Watcher exited successfully');
+      reWatch();
     }
     else {
       console.log(`Watcher exited with error code ${code}`);
     }
-    reWatch();
   });
 };
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "21.0.0",
+  "version": "23.0.0",
   "description": "Run 920 web accessibility tests from 9 tools and get a standardized report",
   "main": "index.js",
   "scripts": {

package/procs/nav.js

@@ -0,0 +1,220 @@
+// nav
+
+// ######## IMPORTS
+
+// Playwright package.
+const playwright = require('playwright');
+
+// ######## CONSTANTS
+
+// Strings in log messages indicating errors.
+const errorWords = [
+  'but not used',
+  'content security policy',
+  'deprecated',
+  'error',
+  'exception',
+  'expected',
+  'failed',
+  'invalid',
+  'missing',
+  'non-standard',
+  'not supported',
+  'refused',
+  'requires',
+  'sorry',
+  'suspicious',
+  'unrecognized',
+  'violates',
+  'warning'
+];
+
+// ######## VARIABLES
+
+let browser;
+
+// ######## FUNCTIONS
+
+// Returns a string with any final slash removed.
+const deSlash = string => string.endsWith('/') ? string.slice(0, -1) : string;
+// Visits a URL and returns the response of the server.
+const goTo = async (report, page, url, timeout, waitUntil) => {
+  if (url.startsWith('file://')) {
+    url = url.replace('file://', `file://${__dirname}/`);
+  }
+  // Visit the URL.
+  const startTime = Date.now();
+  try {
+    const response = await page.goto(url, {
+      timeout,
+      waitUntil
+    });
+    report.jobData.visitLatency += Math.round((Date.now() - startTime) / 1000);
+    const httpStatus = response.status();
+    // If the response status was normal:
+    if ([200, 304].includes(httpStatus) || url.startsWith('file:')) {
+      // If the browser was redirected in violation of a strictness requirement:
+      const actualURL = page.url();
+      if (report.strict && deSlash(actualURL) !== deSlash(url)) {
+        // Return an error.
+        console.log(`ERROR: Visit to ${url} redirected to ${actualURL}`);
+        return {
+          exception: 'badRedirection'
+        };
+      }
+      // Otherwise, i.e. if no prohibited redirection occurred:
+      else {
+        // Press the Escape key to dismiss any modal dialog.
+        await page.keyboard.press('Escape');
+        // Return the response.
+        return response;
+      }
+    }
+    // Otherwise, i.e. if the response status was abnormal:
+    else {
+      // Return an error.
+      console.log(`ERROR: Visit to ${url} got status ${httpStatus}`);
+      report.jobData.visitRejectionCount++;
+      return {
+        error: 'badStatus'
+      };
+    }
+  }
+  catch(error) {
+    console.log(`ERROR visiting ${url} (${error.message.slice(0, 200)})`);
+    return {
+      error: 'noVisit'
+    };
+  }
+};
+// Closes the current browser.
+const browserClose = async () => {
+  if (browser) {
+    const browserType = browser.browserType().name();
+    let contexts = browser.contexts();
+    for (const context of contexts) {
+      await context.close();
+      contexts = browser.contexts();
+    }
+    await browser.close();
+    browser = null;
+    console.log(`${browserType} browser closed`);
+  }
+};
+// Launches a browser, navigates to a URL, and returns the status.
+const launch = async (report, typeName, url, debug, waits, isLowMotion = false) => {
+  // If the specified browser type exists:
+  const browserType = playwright[typeName];
+  if (browserType) {
+    // Close the current browser, if any.
+    await browserClose();
+    // Launch a browser of the specified type.
+    const browserOptions = {
+      logger: {
+        isEnabled: () => false,
+        log: (name, severity, message) => console.log(message.slice(0, 100))
+      }
+    };
+    if (debug) {
+      browserOptions.headless = false;
+    }
+    if (waits) {
+      browserOptions.slowMo = waits;
+    }
+    browser = await browserType.launch(browserOptions)
+    // If the launch failed:
+    .catch(async error => {
+      healthy = false;
+      console.log(`ERROR launching browser (${errorStart(error)})`);
+      // Return this.
+      return false;
+    });
+    // Open a context (i.e. browser tab), with reduced motion if specified.
+    const options = {reduceMotion: isLowMotion ? 'reduce' : 'no-preference'};
+    browserContext = await browser.newContext(options);
+    // When a page (i.e. browser tab) is added to the browser context (i.e. browser window):
+    browserContext.on('page', async page => {
+      // Make the page current.
+      currentPage = page;
+      // If it emits a message:
+      page.on('console', msg => {
+        const msgText = msg.text();
+        let indentedMsg = '';
+        // If debugging is on:
+        if (debug) {
+          // Log a summary of the message on the console.
+          const parts = [msgText.slice(0, 75)];
+          if (msgText.length > 75) {
+            parts.push(msgText.slice(75, 150));
+            if (msgText.length > 150) {
+              const tail = msgText.slice(150).slice(-150);
+              if (msgText.length > 300) {
+                parts.push('...');
+              }
+              parts.push(tail.slice(0, 75));
+              if (tail.length > 75) {
+                parts.push(tail.slice(75));
+              }
+            }
+          }
+          indentedMsg = parts.map(part => `    | ${part}`).join('\n');
+          console.log(`\n${indentedMsg}`);
+        }
+        // Add statistics on the message to the report.
+        const msgTextLC = msgText.toLowerCase();
+        const msgLength = msgText.length;
+        report.jobData.logCount++;
+        report.jobData.logSize += msgLength;
+        if (errorWords.some(word => msgTextLC.includes(word))) {
+          report.jobData.errorLogCount++;
+          report.jobData.errorLogSize += msgLength;
+        }
+        const msgLC = msgText.toLowerCase();
+        if (
+          msgText.includes('403') && (msgLC.includes('status')
+          || msgLC.includes('prohibited'))
+        ) {
+          report.jobData.prohibitedCount++;
+        }
+      });
+    });
+    // Open the first page of the context.
+    currentPage = await browserContext.newPage();
+    try {
+      // Wait until it is stable.
+      await currentPage.waitForLoadState('domcontentloaded', {timeout: 5000});
+      // Navigate to the specified URL.
+      const navResult = await goTo(report, currentPage, url, 15000, 'domcontentloaded');
+      // If the navigation failed:
+      if (navResult.error) {
+        // Return this.
+        return false;
+      }
+      // Otherwise, i.e. if it succeeded:
+      else if (! navResult.error) {
+        // Update the name of the current browser type and store it in the page.
+        currentPage.browserTypeName = typeName;
+        // Return the browser context and the page.
+        return {
+          browserContext,
+          currentPage
+        };
+      }
+    }
+    // If it fails to become stable by the deadline:
+    catch(error) {
+      // Return this.
+      console.log(`ERROR: Blank page load in new tab timed out (${error.message})`);
+      return false;
+    }
+  }
+  // Otherwise, i.e. if it does not exist:
+  else {
+    // Return this.
+    console.log(`ERROR: Browser of type ${typeName} could not be launched`);
+    return false;
+  }
+};
+exports.browserClose = browserClose;
+exports.goTo = goTo;
+exports.launch = launch;

package/procs/sample.js

@@ -10,7 +10,7 @@ exports.getSample = (population, sampleSize) => {
   const popSize = population.length;
   // If the sample is smaller than the population:
   if (sampleSize < popSize) {
-    // Assign to each trigger a priority randomly decreasing with its index.
+    // Assign to each individual a priority randomly decreasing with its index.
     const WeightedPopulation = population.map((item, index) => {
       const weight = 1 + Math.sin(Math.PI * index / popSize + Math.PI / 2);
       const priority = weight * Math.random();
@@ -20,7 +20,7 @@ exports.getSample = (population, sampleSize) => {
     const sortedPopulation = WeightedPopulation.sort((a, b) => b[1] - a[1]);
     const sample = sortedPopulation.slice(0, sampleSize);
     const domOrderSample = sample.sort((a, b) => a[0] - b[0]);
-    return domOrderSample.map(trigger => trigger[0]);
+    return domOrderSample.map(individual => individual[0]);
   }
   // Otherwise, i.e. if the sample is at least as large as the population:
   else {

package/{standardize.js → procs/standardize.js}

@@ -400,6 +400,20 @@ const convert = (toolName, result, standardResult) => {
         console.log(`ERROR: Testaro rule ${rule} result has no standardInstances property`);
       }
     });
+    const preventionCount = result.preventions && result.preventions.length;
+    if (preventionCount) {
+      standardResult.totals[3] += preventionCount;
+      standardResult.instances.push({
+        ruleID: 'testPrevention',
+        what: 'Page prevented tests from being performed',
+        ordinalSeverity: 3,
+        count: preventionCount,
+        tagName: '',
+        id: '',
+        location: '',
+        excerpt: ''
+      });
+    }
     standardResult.totals = standardResult.totals.map(total => Math.round(total));
   }
   // wave

package/procs/testaro.js

@@ -2,18 +2,32 @@
 
 // ########## IMPORTS
 
+// Module to sample a population.
+const {getSample} = require('../procs/sample');
 // Module to get locator data.
 const {getLocatorData} = require('../procs/getLocatorData');
 
+// ########## CONSTANTS
+
+const sampleMax = 100;
+
 // ########## FUNCTIONS
 
 // Initializes locators and a result.
 exports.init = async (page, locAllSelector, options = {}) => {
   // Get locators for the specified elements.
-  const locAll = page.locator(locAllSelector, options);
-  const allLocs = await locAll.all();
+  const locPop = page.locator(locAllSelector, options);
+  const locPops = await locPop.all();
+  const populationSize = locPops.length;
+  const sampleSize = Math.min(sampleMax, populationSize);
+  const locIndexes = getSample(locPops, sampleSize);
+  const allLocs = locIndexes.map(index => locPops[index]);
   const result = {
-    data: {},
+    data: {
+      populationSize,
+      sampleSize,
+      populationRatio: sampleSize ? populationSize / sampleSize : null
+    },
     totals: [0, 0, 0, 0],
     standardInstances: []
   };
@@ -28,7 +42,7 @@ exports.init = async (page, locAllSelector, options = {}) => {
 // Populates a result.
 exports.report = async (withItems, all, ruleID, whats, ordinalSeverity, tagName = '') => {
   const {locs, result} = all;
-  const {totals, standardInstances} = result;
+  const {data, totals, standardInstances} = result;
   // For each instance locator:
   for (const locItem of locs) {
     // Get data on its element.
@@ -42,7 +56,7 @@ exports.report = async (withItems, all, ruleID, whats, ordinalSeverity, tagName
     }
     const elData = await getLocatorData(loc);
     // Add to the totals.
-    totals[ordinalSeverity]++;
+    totals[ordinalSeverity] += data.populationRatio;
     // If itemization is required:
     if (withItems) {
       // Add a standard instance to the result.
@@ -64,7 +78,7 @@ exports.report = async (withItems, all, ruleID, whats, ordinalSeverity, tagName
       ruleID,
       what: whats[1],
       ordinalSeverity,
-      count: totals[ordinalSeverity],
+      count: Math.round(totals[ordinalSeverity]),
       tagName,
       id: '',
       location: {

package/procs/visChange.js

@@ -2,6 +2,7 @@
   visChange
   This procedure reports a change in the visible content of a page between two times, optionally
   hovering over a locator-defined element immediately after the first time.
+
   WARNING: This test uses the Playwright page.screenshot method, which produces incorrect results
   when the browser type is chromium and is not implemented for the firefox browser type. The only
   browser type usable with this test is webkit.
@@ -23,7 +24,6 @@ const shoot = async (page, exclusion = null) => {
     timeout: 2000
   };
   if (exclusion) {
-    const exclusionText = await exclusion.textContent();
     options.mask = [exclusion];
   }
   return await page.screenshot(options)