testaro 32.3.5 → 32.4.0

package/README.md

@@ -150,7 +150,7 @@ Here is an example of a job:
 
 ```javascript
 {
-  id: 'be76p-ts25-w3c',
+  id: '241213T1200-ts25-w3c',
   what: 'Test W3C with 2 alfa rules',
   strict: true,
   timeLimit: 65,
@@ -178,8 +178,8 @@ Here is an example of a job:
     },
     requester: 'user@domain.org'
   },
-  creationTime: '2023-05-26T14:28',
-  timeStamp: 'be76p'
+  creationTime: '2024-12-10T14:28Z',
+  timeStamp: '241213T1200'
 }
 ```
 
@@ -198,8 +198,8 @@ Job properties:
    - `batch` (optional): a set of targets (URLs) from which the target of this job was drawn.
    - `target` (optional): an object describing the target being tested by this job.
    - `requester` (optional): the email address that should receive a notice of completion of the job.
-- `creationTime`: the time when the job was created.
-- `timeStamp`: a string unique to this job.
+- `creationTime`: the time in ISO 8601 format when the job was created.
+- `timeStamp`: a string representing the date and time before which the job is not to be performed.
 
 ### Reports
 
@@ -595,7 +595,7 @@ When no string pertains to a module, then QualWeb will test for all of the rules
 
 Thus, when the `rules` argument is omitted, QualWeb will test for all of the rules in all of these modules.
 
-**Notice**: As of 2023-09-02, any attempt to perform best-practices tests of QualWeb caused QualWeb to throw an error. This issue was [reported to QualWeb](https://github.com/qualweb/core/issues/29). Until this issue is resolved, Testaro does not try to run any QualWeb best-practices tests, regardless of the value of the `rules` argument.
+The target can be provided to QualWeb either as an existing page or as a URL. Experience indicates that the results can differ between these methods, with each method reporting some rule violations or some instances that the other method does not report.
 
 ###### Testaro
 
@@ -774,7 +774,7 @@ Testaro can watch for a job in a directory, with the `dirWatch` function, which
 ###### By a module
 
 ```javaScript
-const {dirWatch} = require('./watch');
+const {dirWatch} = require('./dirWatch');
 dirWatch(true, 300);
 ```
 
@@ -786,20 +786,12 @@ Testaro creates a report for each job and saves the report in the directory spec
 
 ###### By a user
 
-A user can choose between two methods:
-
 ```javaScript
 node call dirWatch true 300
 ```
 
-```javaScript
-node dirWatch true 300
-```
-
 The arguments and behaviors described above for execution by a module apply here, too.
 
-The second, shorter method spawns a new watch subprocess after each job performance, to decrease the risk of process corruption involving bogus timeout messages from Playwright during jobs. That method requires you to enter `CTRL-c` to stop the watching.
-
 ##### Network watch
 
 Testaro can poll servers for jobs to be performed.
@@ -808,21 +800,27 @@ An instance of Testaro is an _agent_ and has an identifier specified by `process
 
 The URLs polled by Testaro are specified by `process.env.JOB_URLS`. The format of that environment variable is a `+`-delimited list of URLs, including schemes. If one of the URLs is `https://testrunner.org/a11ytest/api/job`, and if a Testaro instance has the agent ID `tester3`, then a job request is a `GET` request to `https://testrunner.org/a11ytest/api/job?agent=tester3`.
 
-Once a Testaro instance obtains a network job, the report is sent in a `POST` request to the URL specified by the `sources.sendReportTo` property of the job.
+Once a Testaro instance obtains a network job, Testaro performs it and adds the result data to the job, which then becomes the job report. Testaro sends the report in a `POST` request to the URL specified by the `sources.sendReportTo` property of the job.
+
+Network watching can be repeated or 1-job. One-job watching stops after 1 job has been performed.
+
+After checking all the URLs in succession without getting a job from any of them, Testaro waits for a prescribed time before continuing to check.
 
 ###### By a module
 
 ```javaScript
-const {netWatch} = require('./watch');
-netWatch(true, 300);
+const {netWatch} = require('./netWatch');
+netWatch(true, 300, true);
 ```
 
 In this example, a module asks Testaro to check the servers for a job every 300 seconds, to perform any jobs obtained from the servers, and then to continue checking until the process is stopped. If the first argument is `false`, Testaro will stop checking after performing 1 job.
 
+The third argument specifies whether Testaro should be certificate-tolerant. A `true` value makes Testaro accept SSL certificates that fail verification against a list of certificate authorities. This allows testing of `https` targets that, for example, use self-signed certificates. If the third argument is omitted, the default for that argument is implemented. The default is `true`.
+
 ###### By a user
 
 ```javaScript
-node call netWatch true 300
+node call netWatch true 300 true
 ```
 
 The arguments and behaviors described above for execution by a module apply here, too. If the first argument is `true`, you can terminate the process by entering `CTRL-c`.

package/actSpecs.js

@@ -145,7 +145,7 @@ exports.actSpecs = {
       'Perform tests of a tool',
       {
         which: [true, 'string', 'isTest', 'tool name'],
-        rules: [false, 'array', 'areStrings', 'rule IDs or specifications, if not all']
+        rules: [false, 'array', 'areStrings', 'rule IDs or (for nuVal) specifications, if not all']
       }
     ],
     text: [

package/call.js

@@ -41,7 +41,8 @@ const fs = require('fs/promises');
 // Function to process a testing request.
 const {doJob} = require('./run');
 // Function to watch for jobs.
-const {dirWatch, netWatch} = require('./watch');
+const {dirWatch} = require('./dirWatch');
+const {netWatch} = require('./netWatch');
 
 // CONSTANTS
 
@@ -81,12 +82,17 @@ const callRun = async jobIDStart => {
   }
 };
 // Starts a directory watch, converting the interval argument to a number.
-const callDirWatch = async (isForever, interval) => {
-  await dirWatch(isForever === 'true', Math.max(5, Number.parseInt(interval, 10)));
+const callDirWatch = async (isForever, intervalInSeconds) => {
+  await dirWatch(isForever === 'true', Math.max(5, Number.parseInt(intervalInSeconds, 10)));
 };
 // Starts a network watch, converting the interval argument to a number.
-const callNetWatch = async(isForever, interval) => {
-  netWatch(isForever === 'true', Number.parseInt(interval, 10));
+const callNetWatch = async (isForever, intervalInSeconds, isCertTolerant) => {
+  await netWatch(
+    isForever === 'true',
+    Number.parseInt(intervalInSeconds, 10),
+    isCertTolerant ? isCertTolerant === 'true' : undefined
+  );
+  console.log('netWatch run');
 };
 
 // OPERATION
@@ -100,11 +106,20 @@ if (fn === 'run' && fnArgs.length < 2) {
   });
 }
 else if (fn === 'dirWatch' && fnArgs.length === 2) {
-  callDirWatch(... fnArgs);
+  callDirWatch(... fnArgs)
+  .then(() => {
+    console.log('Directory watch ended');
+    process.exit(0);
+  });
 }
-else if (fn === 'netWatch' && fnArgs.length === 2) {
-  callNetWatch(... fnArgs);
+else if (fn === 'netWatch' && [2, 3].includes(fnArgs.length)) {
+  callNetWatch(... fnArgs)
+  .then(() => {
+    console.log('Network watch ended');
+    process.exit(0);
+  });
 }
 else {
   console.log('ERROR: Invalid statement');
+  process.exit(1);
 }

package/dirWatch.js

@@ -22,48 +22,137 @@
 
 /*
   dirWatch.js
-  Module for launching a repeated one-time directory watch.
+  Module for watching a directory for jobs.
 */
 
 // ########## IMPORTS
 
-// Module to spawn a child process.
-const {spawn} = require('node:child_process');
+// Module to keep secrets.
+require('dotenv').config();
+// Module to read and write files.
+const fs = require('fs/promises');
+// Module to perform jobs.
+const {doJob} = require('./run');
 
 // ########## CONSTANTS
 
-const interval = process.argv[2];
+const jobDir = process.env.JOBDIR;
+const reportDir = process.env.REPORTDIR;
 
 // ########## FUNCTIONS
 
-// Spawns a one-time directory watch.
-const spawnWatch = (command, args) => spawn(command, args, {stdio: ['inherit', 'inherit', 'pipe']});
-// Repeatedly spawns a one-time directory watch.
-const reWatch = () => {
-  const watcher = spawnWatch('node', ['call', 'watch', 'false', interval]);
-  let error = '';
-  watcher.stderr.on('data', data => {
-    error += data.toString();
+// Gets a segment of a timestamp.
+const tsPart = (timeStamp, startIndex) => timeStamp.slice(startIndex, startIndex + 2);
+// Returns a string representing the date and time.
+const nowString = () => (new Date()).toISOString().slice(0, 16);
+// Gets date of a timestamp.
+const dateOf = ts => {
+  const dateString = `20${tsPart(ts, 0)}-${tsPart(ts, 2)}-${tsPart(ts, 4)}`;
+  const timeString = `${tsPart(ts, 7)}:${tsPart(ts, 9)}:00`;
+  const dateTimeString = `${dateString}T${timeString}Z`;
+  return new Date(dateTimeString);
+};
+// Writes a directory report.
+const writeDirReport = async report => {
+  const jobID = report && report.id;
+  if (jobID) {
+    try {
+      const reportJSON = JSON.stringify(report, null, 2);
+      const reportName = `${jobID}.json`;
+      await fs.mkdir(`${reportDir}/raw`, {recursive: true});
+      await fs.writeFile(`${reportDir}/raw/${reportName}`, `${reportJSON}\n`);
+      console.log(`Report ${jobID} saved in ${reportDir}/raw`);
+    }
+    catch(error) {
+      console.log(`ERROR: Failed to save report ${jobID} in ${reportDir}/raw (${error.message})`);
+    }
+  }
+  else {
+    console.log('ERROR: Job has no ID');
+  }
+};
+// Archives a job.
+const archiveJob = async (job, isFile) => {
+  // Save the job in the done subdirectory.
+  const {id} = job;
+  const jobJSON = JSON.stringify(job, null, 2);
+  await fs.mkdir(`${jobDir}/done`, {recursive: true});
+  await fs.writeFile(`${jobDir}/done/${id}.json`, `${jobJSON}\n`);
+  // If the job had been saved as a file in the todo subdirectory:
+  if (isFile) {
+    // Delete the file.
+    await fs.rm(`${jobDir}/todo/${id}.json`);
+  }
+  console.log(`Job ${id} archived in ${jobDir}/done (${nowString()})`);
+};
+// Waits.
+const wait = ms => {
+  return new Promise(resolve => {
+    setTimeout(() => {
+      resolve('');
+    }, ms);
   });
-  watcher.on('close', async code => {
-    if (error) {
-      if (error.startsWith('Navigation timeout of 30000 ms exceeded')) {
-        console.log('ERROR: Playwright claims 30-second timeout exceeded');
+};
+/* 
+  Checks for a directory job and, when found, performs and reports it.
+  Arguments:
+  0. Whether to continue watching after a job is run.
+  1: interval in seconds from a no-job check to the next check.
+*/
+exports.dirWatch = async (isForever, intervalInSeconds) => {
+  console.log(`Starting to watch directory ${jobDir}/todo for jobs`);
+  let notYetRun = true;
+  // As long as watching as to continue:
+  while (isForever || notYetRun) {
+    try {
+      // If there are any jobs in the watched directory:
+      const toDoFileNames = await fs.readdir(`${jobDir}/todo`);
+      const jobFileNames = toDoFileNames.filter(fileName => fileName.endsWith('.json'));
+      if (jobFileNames.length) {
+        // If the first one is ready to do:
+        const firstJobTimeStamp = jobFileNames[0].replace(/-.+$/, '');
+        if (Date.now() > dateOf(firstJobTimeStamp)) {
+          // Get it.
+          const jobJSON = await fs.readFile(`${jobDir}/todo/${jobFileNames[0]}`, 'utf8');
+          try {
+            const job = JSON.parse(jobJSON);
+            const report = JSON.parse(jobJSON);
+            const {id} = job;
+            console.log(`Directory job ${id} ready to do (${nowString()})`);
+            // Perform it.
+            await doJob(report);
+            console.log(`Job ${id} finished (${nowString()})`);
+            // Report it.
+            await writeDirReport(report);
+            // Archive it.
+            await archiveJob(job, true);
+          }
+          catch(error) {
+            console.log(`ERROR processing directory job (${error.message})`);
+          }
+          notYetRun = false;
+        }
+        // Otherwise, i.e. if the first one is not yet ready to do:
+        else {
+          // Report this.
+          console.log(`All jobs in ${jobDir} not yet ready to do (${nowString()})`);
+          // Wait for the specified interval.
+          await wait(1000 * intervalInSeconds);
+        }
       }
+      // Otherwise, i.e. if there are no jobs in the watched directory:
       else {
-        console.log(`ERROR: ${error.slice(0, 200)}`);
+        console.log(`No job in ${jobDir} (${nowString()})`);
+        // Wait for the specified interval.
+        await wait(1000 * intervalInSeconds);
       }
     }
-    if (code === 0) {
-      console.log('Watcher exited successfully\n');
-      reWatch();
-    }
-    else {
-      console.log(`Watcher exited with error code ${code}`);
+    // If a fatal error was thrown:
+    catch(error) {
+      // Report this.
+      console.log(`ERROR: Directory watching failed (${error.message}); watching aborted`);
+      // Quit watching.
+      break;
     }
-  });
+  }
 };
-
-// ########## OPERATION
-
-reWatch();

package/netWatch-old.js

@@ -0,0 +1,397 @@
+/*
+  © 2022–2023 CVS Health and/or one of its affiliates. All rights reserved.
+
+  Permission is hereby granted, free of charge, to any person obtaining a copy
+  of this software and associated documentation files (the "Software"), to deal
+  in the Software without restriction, including without limitation the rights
+  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+  copies of the Software, and to permit persons to whom the Software is
+  furnished to do so, subject to the following conditions:
+
+  The above copyright notice and this permission notice shall be included in all
+  copies or substantial portions of the Software.
+
+  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+  SOFTWARE.
+*/
+
+/*
+  netWatch.js
+  Module for watching for a network job and running it when found.
+*/
+
+// ########## IMPORTS
+
+// Module to keep secrets.
+require('dotenv').config();
+// Module to read and write files.
+const fs = require('fs/promises');
+// Module to make requests to servers.
+const httpClient = require('http');
+const httpsClient = require('https');
+// Module to perform jobs.
+const {doJob} = require('./run');
+
+// CONSTANTS
+
+const jobDir = process.env.JOBDIR;
+const jobURLs = process.env.JOB_URLS;
+const reportDir = process.env.REPORTDIR;
+const agent = process.env.AGENT;
+
+// ########## FUNCTIONS
+
+// Returns a string representing the date and time.
+const nowString = () => (new Date()).toISOString().slice(0, 19);
+// Waits.
+const wait = ms => {
+  return new Promise(resolve => {
+    setTimeout(() => {
+      resolve('');
+    }, ms);
+  });
+};
+// Serves an object in JSON format.
+const serveObject = (object, response) => {
+  response.setHeader('Content-Type', 'application/json; charset=utf-8');
+  response.end(JSON.stringify(object));
+};
+// Writes a directory report.
+const writeDirReport = async report => {
+  const jobID = report && report.id;
+  if (jobID) {
+    try {
+      const reportJSON = JSON.stringify(report, null, 2);
+      const reportName = `${jobID}.json`;
+      await fs.mkdir(reportDir, {recursive: true});
+      await fs.writeFile(`${reportDir}/${reportName}`, reportJSON);
+      console.log(`Report ${reportName} saved in ${reportDir}`);
+    }
+    catch(error) {
+      console.log(`ERROR: Failed to write report ${jobID} in ${reportDir} (${error.message})`);
+    }
+  }
+  else {
+    console.log('ERROR: Job has no ID');
+  }
+};
+// Archives a job.
+const archiveJob = async (job, isFile) => {
+  // Save the job in the done subdirectory.
+  const {id} = job;
+  const jobJSON = JSON.stringify(job, null, 2);
+  await fs.mkdir(`${jobDir}/done`, {recursive: true});
+  await fs.writeFile(`${jobDir}/done/${id}.json`, jobJSON);
+  // If the job had been saved as a file in the todo subdirectory:
+  if (isFile) {
+    // Delete the file.
+    await fs.rm(`${jobDir}/todo/${id}.json`);
+  }
+};
+// Checks for a directory job and, if found, performs and reports it, once or repeatedly.
+const checkDirJob = async (isForever, interval) => {
+  try {
+    // If there are any jobs in the watched directory:
+    const toDoFileNames = await fs.readdir(`${jobDir}/todo`);
+    const jobFileNames = toDoFileNames.filter(fileName => fileName.endsWith('.json'));
+    if (jobFileNames.length) {
+      // If the first one is ready to do:
+      const firstJobTime = jobFileNames[0].replace(/-.+$/, '');
+      if (Date.now() > dateOf(firstJobTime)) {
+        // Perform it.
+        const jobJSON = await fs.readFile(`${jobDir}/todo/${jobFileNames[0]}`, 'utf8');
+        try {
+          const job = JSON.parse(jobJSON, null, 2);
+          const {id} = job;
+          console.log(`Directory job ${id} found (${nowString()})`);
+          await doJob(job);
+          console.log(`Job ${id} finished (${nowString()})`);
+          // Report it.
+          await writeDirReport(job);
+          // Archive it.
+          await archiveJob(job, true);
+          console.log(`Job ${id} archived in ${jobDir} (${nowString()})`);
+          // If watching is repetitive:
+          if (isForever) {
+            // Wait 2 seconds.
+            await wait(2000);
+            // Check the directory again.
+            checkDirJob(true, interval);
+          }
+        }
+        catch(error) {
+          console.log(`ERROR processing directory job (${error.message})`);
+        }
+      }
+      // Otherwise, i.e. if the first one is not yet ready to do:
+      else {
+        // Report this.
+        console.log(`All jobs in ${jobDir} not yet ready to do (${nowString()})`);
+        // Wait for the specified interval.
+        await wait(1000 * interval);
+        // Check the directory again.
+        await checkDirJob(true, interval);
+      }
+    }
+    // Otherwise, i.e. if there are no more jobs in the watched directory:
+    else {
+      console.log(`No job in ${jobDir} (${nowString()})`);
+      // Wait for the specified interval.
+      await wait(1000 * interval);
+      // Check the directory again.
+      await checkDirJob(true, interval);
+    }
+  }
+  catch(error) {
+    console.log(`ERROR: Directory watching failed (${error.message})`);
+  }
+};
+// Checks servers for a network job.
+const checkNetJob = async (servers, serverIndex, isForever, interval, noJobCount) => {
+  // If all servers are jobless:
+  if (noJobCount === servers.length) {
+    // Wait for the specified interval.
+    await wait(1000 * interval);
+    // Reset the count of jobless servers.
+    noJobCount = 0;
+  }
+  // Otherwise, i.e. if any server may still have a job:
+  else {
+    // Wait 2 seconds.
+    await wait(2000);
+  }
+  // If the last server has been checked:
+  serverIndex = serverIndex % servers.length;
+  if (serverIndex === 0) {
+    // Report this.
+    console.log('--');
+  }
+  // Check the next server.
+  const server = servers[serverIndex];
+  const client = server.startsWith('https://') ? httpsClient : httpClient;
+  const fullURL = `${server}?agent=${agent}`;
+  const logStart = `Requested job from server ${server} and got `;
+  // Tolerate unrecognized certificate authorities if the environment specifies.
+  const ruOpt = process.env.REJECT_UNAUTHORIZED === 'false' ? {rejectUnauthorized: false} : {};
+  client.request(fullURL, ruOpt, response => {
+    const chunks = [];
+    response
+    // If the response to the job request threw an error:
+    .on('error', async error => {
+      // Report it.
+      console.log(`${logStart}error message ${error.message}`);
+      // Check the next server.
+      await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+    })
+    .on('data', chunk => {
+      chunks.push(chunk);
+    })
+    // When the response arrives:
+    .on('end', async () => {
+      const content = chunks.join('');
+      try {
+        // If there was no job to do:
+        let contentObj = JSON.parse(content);
+        if (! Object.keys(contentObj).length) {
+          // Report this.
+          console.log(`No job to do at ${server}`);
+          // Check the next server.
+          await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+        }
+        // Otherwise, i.e. if there was a job or a message:
+        else {
+          const {message, id, sources} = contentObj;
+          // If the server sent a message, not a job:
+          if (message) {
+            // Report it.
+            console.log(`${logStart}${message}`);
+            // Check the next server.
+            await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+          }
+          // Otherwise, if the server sent a valid job:
+          else if (id && sources && sources.target && sources.target.which) {
+            // Add the agent to it.
+            sources.agent = agent;
+            // If the job specifies a report destination:
+            const {sendReportTo} = sources;
+            if (sendReportTo) {
+              // Perform the job, adding result data to it.
+              const testee = sources.target.which;
+              console.log(
+                `${logStart}job ${id} (${nowString()})\n>> It will test ${testee}\n>> It will send report to ${sendReportTo}\n`
+              );
+              await doJob(contentObj);
+              let reportJSON = JSON.stringify(contentObj, null, 2);
+              console.log(`Job ${id} finished (${nowString()})`);
+              // Send the report to the specified server.
+              console.log(`Sending report ${id} to ${sendReportTo}`);
+              const reportClient = sendReportTo.startsWith('https://') ? httpsClient : httpClient;
+              const reportLogStart = `Sent report ${id} to ${sendReportTo} and got `;
+              reportClient.request(sendReportTo, {method: 'POST'}, repResponse => {
+                const chunks = [];
+                repResponse
+                // If the response to the report threw an error:
+                .on('error', async error => {
+                  // Report this.
+                  console.log(`${reportLogStart}error message ${error.message}\n`);
+                  // Check the next server.
+                  await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+                })
+                .on('data', chunk => {
+                  chunks.push(chunk);
+                })
+                // When the response arrives:
+                .on('end', async () => {
+                  const content = chunks.join('');
+                  try {
+                    // If the server sent a message, as expected:
+                    const ackObj = JSON.parse(content);
+                    const {message} = ackObj;
+                    if (message) {
+                      // Report it.
+                      console.log(`${reportLogStart}${message}\n`);
+                      // Free the memory used by the report.
+                      reportJSON = '';
+                      contentObj = {};
+                      // Check the next server.
+                      await checkNetJob(servers, serverIndex + 1, isForever, interval, 0);
+                    }
+                    // Otherwise, i.e. if the server sent anything else:
+                    else {
+                      // Report it.
+                      console.log(
+                        `ERROR: ${reportLogStart}status ${repResponse.statusCode} and error message ${JSON.stringify(ackObj, null, 2)}\n`
+                      );
+                      // Check the next server, disregarding the failed job.
+                      await checkNetJob(
+                        servers, serverIndex + 1, isForever, interval, noJobCount + 1
+                      );
+                    }
+                  }
+                  // If processing the report threw an error:
+                  catch(error) {
+                    // Report it.
+                    console.log(
+                      `ERROR: ${reportLogStart}status ${repResponse.statusCode}, error message ${error.message}, and response ${content.slice(0, 1000)}\n`
+                    );
+                    // Check the next server, disregarding the failed job.
+                    await checkNetJob(
+                      servers, serverIndex + 1, isForever, interval, noJobCount + 1
+                    );
+                  }
+                });
+              })
+              // If the report submission throws an error:
+              .on('error', async error => {
+                // Report this.
+                console.log(`ERROR: ${reportLogStart}error message ${error.message}\n`);
+                // Check the next server, disregarding the failed job.
+                await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+              })
+              // Finish submitting the report.
+              .end(reportJSON);
+            }
+            // Otherwise, i.e. if the job specifies no report destination:
+            else {
+              // Report this.
+              const message = `ERROR: ${logStart}job with no report destination`;
+              serveObject({message}, response);
+              console.log(message);
+              // Check the next server, disregarding the defective job.
+              await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+            }
+          }
+          // Otherwise, if the server sent an invalid job:
+          else {
+            // Report this.
+            const message
+            = `ERROR: ${logStart}invalid job:\n${JSON.stringify(contentObj, null, 2)}`;
+            console.log(message);
+            serveObject({message}, response);
+            // Check the next server, disregarding the defective job.
+            await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+          }
+        }
+      }
+      // If an error is thrown:
+      catch(error) {
+        // Report this.
+        console.log(`ERROR: ${error.message} (response ${content.slice(0, 1000)})`);
+        // Check the next server.
+        await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+      }
+    });
+  })
+  // If the job request throws an error:
+  .on('error', async error => {
+    // If it was a refusal to connect:
+    const {message} = error;
+    if (message.includes('ECONNREFUSED')) {
+      // Report this.
+      console.log(`${logStart}no connection`);
+      // Check the next server.
+      await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+    }
+    // Otherwise, if it was a DNS failure:
+    else if (message.includes('ENOTFOUND')) {
+      // Report this.
+      console.log(`${logStart}no domain name resolution`);
+      // Check the next server.
+      await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+    }
+    // Otherwise, i.e. if it was any other error:
+    else {
+      // Report this.
+      console.log(
+        `ERROR: ${logStart}no response, but got error message ${error.message.slice(0, 200)}`
+      );
+      // Check the next server.
+      await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+    }
+  })
+  // Finish sending the request.
+  .end();
+};
+// Composes an interval description.
+const intervalSpec = interval => {
+  if (interval > -1) {
+    return `repeatedly, with ${interval}-second intervals `;
+  }
+  else {
+    return '';
+  }
+};
+// Checks for a directory job, performs it, and submits a report, once or repeatedly.
+exports.dirWatch = async (isForever, interval = 300) => {
+  console.log(`Directory watching started ${intervalSpec(interval)}(${nowString()})\n`);
+  // Start the checking.
+  await checkDirJob(isForever, interval);
+};
+// Checks for a network job, performs it, and submits a report, once or repeatedly.
+exports.netWatch = async (isForever, interval = 300) => {
+  console.log('Starting netWatch');
+  // If the servers to be checked are valid:
+  const servers = jobURLs
+  .split('+')
+  .map(url => [Math.random(), url])
+  .sort((a, b) => a[0] - b[0])
+  .map(pair => pair[1]);
+  if (
+    servers
+    && servers.length
+    && servers
+    .every(server => ['http://', 'https://'].some(prefix => server.startsWith(prefix)))
+  ) {
+    console.log(`Network watching started ${intervalSpec(interval)}(${nowString()})\n`);
+    // Start checking.
+    await checkNetJob(servers, 0, isForever, interval, 0);
+  }
+  else {
+    console.log('ERROR: List of job URLs invalid');
+  }
+};

package/netWatch.js

@@ -0,0 +1,303 @@
+/*
+  © 2022–2023 CVS Health and/or one of its affiliates. All rights reserved.
+
+  Permission is hereby granted, free of charge, to any person obtaining a copy
+  of this software and associated documentation files (the "Software"), to deal
+  in the Software without restriction, including without limitation the rights
+  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+  copies of the Software, and to permit persons to whom the Software is
+  furnished to do so, subject to the following conditions:
+
+  The above copyright notice and this permission notice shall be included in all
+  copies or substantial portions of the Software.
+
+  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+  SOFTWARE.
+*/
+
+/*
+  netWatch.js
+  Module for watching for a network job and running it when found.
+*/
+
+// IMPORTS
+
+// Module to keep secrets.
+require('dotenv').config();
+// Module to read and write files.
+const fs = require('fs/promises');
+// Modules to make requests to servers.
+const httpClient = require('http');
+const httpsClient = require('https');
+// Module to perform jobs.
+const {doJob} = require('./run');
+
+// CONSTANTS
+
+const jobURLSpec = process.env.JOB_URLS;
+const agent = process.env.AGENT;
+
+// FUNCTIONS
+
+// Returns a string representing the date and time.
+const nowString = () => (new Date()).toISOString().slice(2, 15);
+// Waits.
+const wait = ms => {
+  return new Promise(resolve => {
+    setTimeout(() => {
+      resolve('');
+    }, ms);
+  });
+};
+// Serves an object in JSON format.
+const serveObject = (object, response) => {
+  response.setHeader('Content-Type', 'application/json; charset=utf-8');
+  response.end(JSON.stringify(object));
+};
+/*
+  Requests a network job and, when found, performs and reports it.
+  Arguments:
+  0. whether to continue watching after a job is run.
+  1: interval in seconds from a cycle of no-job checks to the next cycle.
+*/
+exports.netWatch = async (isForever, intervalInSeconds, isCertTolerant = true) => {
+  const urls = jobURLSpec
+  .split('+')
+  .map(url => [Math.random(), url])
+  .sort((a, b) => a[0] - b[0])
+  .map(pair => pair[1]);
+  const urlCount = urls.length;
+  // If the job URLs exist and are valid:
+  if (
+    urls
+    && urlCount
+    && urls.every(url => ['http://', 'https://'].some(prefix => url.startsWith(prefix)))
+  ) {
+    // Configure the watch.
+    let cycleIndex = -1;
+    let urlIndex = -1;
+    let noJobYet = true;
+    let abort = false;
+    const certOpt = isCertTolerant ? {rejectUnauthorized: false} : {};
+    const certInfo = `Certificate-${isCertTolerant ? '' : 'in'}tolerant`;
+    const foreverInfo = isForever ? 'repeating' : 'one-job';
+    const intervalInfo = `with ${intervalInSeconds}-second intervals`;
+    console.log(
+      `${certInfo} ${foreverInfo} network watching started ${intervalInfo} (${nowString()})\n`
+    );
+    // As long as watching is to continue:
+    while ((isForever || noJobYet) && ! abort) {
+      // If the cycle is complete:
+      if (cycleIndex === urlCount - 1) {
+        // Wait for the specified interval.
+        await wait(1000 * intervalInSeconds);
+        // Log the start of a cycle.
+        console.log('--');
+      }
+      // Otherwise, i.e. if the cycle is incomplete:
+      else {
+        // Wait briefly.
+        await wait(1000);
+      }
+      // Configure the next check.
+      cycleIndex = ++cycleIndex % urlCount;
+      urlIndex = ++urlIndex % urlCount;
+      const url = urls[urlIndex];
+      const logStart = `Requested job from server ${url} and got `;
+      const fullURL = `${url}?agent=${agent}`;
+      // Perform it.
+      await new Promise(resolve => {
+        try {
+          const client = url.startsWith('https://') ? httpsClient : httpClient;
+          // Request a job.
+          client.request(fullURL, certOpt, response => {
+            const chunks = [];
+            response
+            // If the response throws an error:
+            .on('error', async error => {
+              // Report it.
+              console.log(`${logStart}error message ${error.message}`);
+              resolve(true);
+            })
+            .on('data', chunk => {
+              chunks.push(chunk);
+            })
+            // When the response arrives:
+            .on('end', async () => {
+              const content = chunks.join('');
+              try {
+                // If there was no job to do:
+                let contentObj = JSON.parse(content);
+                if (! Object.keys(contentObj).length) {
+                  // Report this.
+                  console.log(`No job to do at ${url}`);
+                  resolve(true);
+                }
+                // Otherwise, i.e. if there was a job or a message:
+                else {
+                  const {message, id, sources} = contentObj;
+                  // If the server sent a message, not a job:
+                  if (message) {
+                    // Report it.
+                    console.log(`${logStart}${message}`);
+                    resolve(true);
+                  }
+                  // Otherwise, if the server sent a valid job:
+                  else if (id && sources && sources.target && sources.target.which) {
+                    // Restart the cycle.
+                    cycleIndex = -1;
+                    // Prevent further watching, if unwanted.
+                    noJobYet = false;
+                    // Add the agent to the job.
+                    sources.agent = agent;
+                    // If the job specifies a report destination:
+                    const {sendReportTo} = sources;
+                    if (sendReportTo) {
+                      // Perform the job, adding result data to it.
+                      const target = sources.target.which;
+                      console.log(`${logStart}job ${id} (${nowString()}`);
+                      console.log(`>> It will test ${target}`);
+                      console.log(`>> It will send report to ${sendReportTo}`);
+                      await doJob(contentObj);
+                      let reportJSON = JSON.stringify(contentObj, null, 2);
+                      console.log(`Job ${id} finished (${nowString()})`);
+                      // Send the report to the specified server.
+                      console.log(`Sending report ${id} to ${sendReportTo}`);
+                      const reportClient = sendReportTo.startsWith('https://') ? httpsClient : httpClient;
+                      const reportLogStart = `Sent report ${id} to ${sendReportTo} and got `;
+                      reportClient.request(sendReportTo, {method: 'POST'}, repResponse => {
+                        const chunks = [];
+                        repResponse
+                        // If the response to the report threw an error:
+                        .on('error', async error => {
+                          // Report this.
+                          console.log(`${reportLogStart}error message ${error.message}\n`);
+                          resolve(true);
+                        })
+                        .on('data', chunk => {
+                          chunks.push(chunk);
+                        })
+                        // When the response arrives:
+                        .on('end', async () => {
+                          const content = chunks.join('');
+                          try {
+                            // If the server sent a message, as expected:
+                            const ackObj = JSON.parse(content);
+                            const {message} = ackObj;
+                            if (message) {
+                              // Report it.
+                              console.log(`${reportLogStart}message ${message}\n`);
+                              // Free the memory used by the report.
+                              reportJSON = '';
+                              contentObj = {};
+                              resolve(true);
+                            }
+                            // Otherwise, i.e. if the server sent anything else:
+                            else {
+                              // Report it.
+                              console.log(
+                                `ERROR: ${reportLogStart}status ${repResponse.statusCode} and error message ${JSON.stringify(ackObj, null, 2)}\n`
+                              );
+                              resolve(true);
+                            }
+                          }
+                          // If processing the server message throws an error:
+                          catch(error) {
+                            // Report it.
+                            console.log(
+                              `ERROR: ${reportLogStart}status ${repResponse.statusCode}, error message ${error.message}, and response ${content.slice(0, 1000)}\n`
+                            );
+                            resolve(true);
+                          }
+                        });
+                      })
+                      // If the report submission throws an error:
+                      .on('error', async error => {
+                        // Report this.
+                        console.log(`ERROR: ${reportLogStart}error message ${error.message}\n`);
+                        resolve(true);
+                      })
+                      // Finish submitting the report.
+                      .end(reportJSON);
+                    }
+                    // Otherwise, i.e. if the job specifies no report destination:
+                    else {
+                      // Report this.
+                      const message = `ERROR: ${logStart}job with no report destination`;
+                      serveObject({message}, response);
+                      console.log(message);
+                      resolve(true);
+                    }
+                  }
+                  // Otherwise, i.e. if the server sent an invalid job:
+                  else {
+                    // Report this.
+                    const message
+                    = `ERROR: ${logStart}invalid job:\n${JSON.stringify(contentObj, null, 2)}`;
+                    console.log(message);
+                    serveObject({message}, response);
+                    resolve(true);
+                  }
+                }
+              }
+              // If processing the server response throws an error:
+              catch(error) {
+                // Report this.
+                console.log(`ERROR: ${error.message} (response ${content.slice(0, 1000)})`);
+                resolve(true);
+              }
+            });
+          })
+          // If the job request throws an error:
+          .on('error', async error => {
+            // If it is a refusal to connect:
+            if (error.code && error.code.includes('ECONNREFUSED')) {
+              // Report this.
+              console.log(`${logStart}no connection`);
+            }
+            // Otherwise, if it was a DNS failure:
+            else if (error.code && error.code.includes('ENOTFOUND')) {
+              // Report this.
+              console.log(`${logStart}no domain name resolution`);
+            }
+            // Otherwise, if it was any other error with a message:
+            else if (error.message) {
+              // Report this.
+              console.log(`ERROR: ${logStart}got error message ${error.message.slice(0, 200)}`);
+              // Abort the watch.
+              abort = true;
+            }
+            // Otherwise, i.e. if it was any other error with no message:
+            else {
+              // Report this.
+              console.log(`ERROR: ${logStart}got an error with no message`);
+              // Abort the watch.
+              abort = true;
+            }
+            resolve(true);
+          })
+          // Finish sending the job request.
+          .end();
+        }
+        // If requesting a job throws an error:
+        catch(error) {
+          // Report this.
+          console.log(`ERROR requesting a network job (${error.message})`);
+          abort = true;
+          resolve(true);
+        }
+      });
+    }
+    console.log('Watching complete');
+  }
+  // Otherwise, i.e. if the job URLs do not exist or are invalid:
+  else {
+    // Report this.
+    console.log('ERROR: List of job URLs invalid');
+  }
+};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "32.3.5",
+  "version": "32.4.0",
   "description": "Run 900 web accessibility tests from 9 tools and get a standardized report",
   "main": "index.js",
   "scripts": {

package/procs/standardize.js

@@ -282,7 +282,7 @@ const doQualWeb = (result, standardResult, ruleClassName) => {
           const instance = {
             ruleID,
             what: item.description,
-            ordinalSeverity: severities[ruleClassName][item.verdict],
+            ordinalSeverity: severities[ruleClassName][item.verdict] || 0,
             tagName: identifiers[0],
             id: identifiers[1],
             location: {

package/run.js

@@ -277,14 +277,10 @@ const isValidReport = report => {
     if (typeof sources.script !== 'string') {
       return 'Bad source script';
     }
-    if (
-      ! creationTime
-      || typeof creationTime !== 'string'
-      || ! /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}$/.test(creationTime)
-    ) {
+    if (! (creationTime && typeof creationTime === 'string' && Date.parse(creationTime))) {
       return 'bad job creation time';
     }
-    if (! timeStamp || typeof timeStamp !== 'string') {
+    if (! (timeStamp && typeof timeStamp === 'string')) {
       return 'bad report timestamp';
     }
     return '';
@@ -772,12 +768,18 @@ const doActs = async (report, actIndex, page) => {
           actInfo = act.which;
         }
       }
+      const message = `>>>> ${act.type}: ${actInfo}`;
       // If granular reporting has been specified:
       if (report.observe) {
-        // Notify the observer of the act.
+        // Notify the observer of the act and log it.
         const whichParam = act.which ? `&which=${act.which}` : '';
         const messageParams = `act=${act.type}${whichParam}`;
-        tellServer(report, messageParams, `>>>> ${act.type}: ${actInfo}`);
+        tellServer(report, messageParams, message);
+      }
+      // Otherwise, i.e. if granular reporting has not been specified:
+      else {
+        // Log the act.
+        console.log(message);
       }
       // Increment the count of acts performed.
       actCount++;

package/tests/qualWeb.js

@@ -114,15 +114,11 @@ exports.reporter = async (page, options) => {
       else {
         const bestPractices = bestSpec.slice(5).split(',').map(num => `QW-BP${num}`);
         qualWebOptions['best-practices'] = {bestPractices};
-        // qualWebOptions.execute.bp = true;
-        // Temporarily disable best practices, because they crash QualWeb.
-        qualWebOptions.execute.bp = false;
+        qualWebOptions.execute.bp = true;
       }
     }
     else {
-      // qualWebOptions.execute.bp = true;
-      // Temporarily disable best practices, because they crash QualWeb.
-      qualWebOptions.execute.bp = false;
+      qualWebOptions.execute.bp = true;
     }
     // Get the report.
     let actReports = await qualWeb.evaluate(qualWebOptions);

package/watch-old.js

@@ -0,0 +1,77 @@
+/*
+  © 2023 CVS Health and/or one of its affiliates. All rights reserved.
+
+  Permission is hereby granted, free of charge, to any person obtaining a copy
+  of this software and associated documentation files (the "Software"), to deal
+  in the Software without restriction, including without limitation the rights
+  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+  copies of the Software, and to permit persons to whom the Software is
+  furnished to do so, subject to the following conditions:
+
+  The above copyright notice and this permission notice shall be included in all
+  copies or substantial portions of the Software.
+
+  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+  SOFTWARE.
+*/
+
+/*
+  dirWatch.js
+  Module for launching a one-time directory watch.
+  Argument:
+    1: interval in seconds from a no-job check to the next check.
+*/
+
+// ########## IMPORTS
+
+// Module to spawn a child process.
+const {spawn} = require('node:child_process');
+
+// ########## CONSTANTS
+
+const repeat = process.argv[2];
+const interval = process.argv[3];
+
+// ########## FUNCTIONS
+
+// Spawns a one-time directory watch.
+const spawnWatch = () => spawn(
+  'node', 'dirWatch', 'false', interval, {stdio: ['inherit', 'inherit', 'pipe']}
+);
+// Repeatedly spawns a one-time directory watch.
+const reWatch = () => {
+  const watcher = spawnWatch('node', ['call', 'dirWatch', 'false', interval]);
+  let error = '';
+  watcher.stderr.on('data', data => {
+    error += data.toString();
+  });
+  watcher.on('close', async code => {
+    if (error) {
+      if (error.startsWith('Navigation timeout of 30000 ms exceeded')) {
+        console.log('ERROR: Playwright claims 30-second timeout exceeded');
+      }
+      else {
+        console.log(`ERROR watching: ${error.slice(0, 200)}`);
+      }
+    }
+    if (! error && code === 0) {
+      console.log('Watcher exited successfully\n');
+      reWatch();
+    }
+    else if (code) {
+      console.log(`Watcher exited with error code ${code}`);
+    }
+    else {
+      console.log('Watch aborted');
+    }
+  });
+};
+
+// ########## OPERATION
+
+reWatch();

package/watch.js

@@ -96,39 +96,51 @@ const archiveJob = async (job, isFile) => {
 // Checks for a directory job and, if found, performs and reports it, once or repeatedly.
 const checkDirJob = async (isForever, interval) => {
   try {
-    // If there are any jobs to do in the watched directory:
+    // If there are any jobs in the watched directory:
     const toDoFileNames = await fs.readdir(`${jobDir}/todo`);
     const jobFileNames = toDoFileNames.filter(fileName => fileName.endsWith('.json'));
     if (jobFileNames.length) {
-      // Get the first one.
-      const jobJSON = await fs.readFile(`${jobDir}/todo/${jobFileNames[0]}`, 'utf8');
-      try {
-        const job = JSON.parse(jobJSON, null, 2);
-        const {id} = job;
+      // If the first one is ready to do:
+      const firstJobTime = jobFileNames[0].replace(/-.+$/, '');
+      if (Date.now() > dateOf(firstJobTime)) {
         // Perform it.
-        console.log(`Directory job ${id} found (${nowString()})`);
-        await doJob(job);
-        console.log(`Job ${id} finished (${nowString()})`);
-        // Report it.
-        await writeDirReport(job);
-        // Archive it.
-        await archiveJob(job, true);
-        console.log(`Job ${id} archived in ${jobDir} (${nowString()})`);
-        // If watching is repetitive:
-        if (isForever) {
-          // Wait 2 seconds.
-          await wait(2000);
-          // Check the directory again.
-          checkDirJob(true, interval);
+        const jobJSON = await fs.readFile(`${jobDir}/todo/${jobFileNames[0]}`, 'utf8');
+        try {
+          const job = JSON.parse(jobJSON, null, 2);
+          const {id} = job;
+          console.log(`Directory job ${id} found (${nowString()})`);
+          await doJob(job);
+          console.log(`Job ${id} finished (${nowString()})`);
+          // Report it.
+          await writeDirReport(job);
+          // Archive it.
+          await archiveJob(job, true);
+          console.log(`Job ${id} archived in ${jobDir} (${nowString()})`);
+          // If watching is repetitive:
+          if (isForever) {
+            // Wait 2 seconds.
+            await wait(2000);
+            // Check the directory again.
+            checkDirJob(true, interval);
+          }
+        }
+        catch(error) {
+          console.log(`ERROR processing directory job (${error.message})`);
         }
       }
-      catch(error) {
-        console.log(`ERROR processing directory job (${error.message})`);
+      // Otherwise, i.e. if the first one is not yet ready to do:
+      else {
+        // Report this.
+        console.log(`All jobs in ${jobDir} not yet ready to do (${nowString()})`);
+        // Wait for the specified interval.
+        await wait(1000 * interval);
+        // Check the directory again.
+        await checkDirJob(true, interval);
       }
     }
-    // Otherwise, i.e. if there are no more jobs to do in the watched directory:
+    // Otherwise, i.e. if there are no more jobs in the watched directory:
     else {
-      console.log(`No job to do in ${jobDir} (${nowString()})`);
+      console.log(`No job in ${jobDir} (${nowString()})`);
       // Wait for the specified interval.
       await wait(1000 * interval);
       // Check the directory again.