testaro 29.1.1 → 30.0.0

package/README.md

@@ -762,58 +762,67 @@ The `call` module will find the first job file with a matching name, execute the
 
 #### Watch
 
-In watch mode, Testaro periodically checks for a job to run. When such a job exists, Testaro runs it and produces a report. Testaro may continue watching after the first report, or may quit.
+In watch mode, Testaro periodically checks for a job to run and, when a job is obtained, performs it.
 
-##### By a module
+##### Directory watch
 
-```javaScript
-const {cycle} = require('./watch');
-cycle(true, true, 30);
-```
+Testaro can watch for a job in a directory, with the `dirWatch` function, which can be executed by either a module or a user.
 
-##### By a user
+###### By a module
 
 ```javaScript
-node call watch true true 30
+const {dirWatch} = require('./watch');
+dirWatch(true, 300);
 ```
 
-##### Arguments
+In this example, a module asks Testaro to check a directory for a job every 300 seconds, to perform the jobs in the directory if any are found, and then to continue checking. If the first argument is `false`, Testaro will stop checking after performing 1 job. If it is `true`, Testaro continues checking until the process is stopped.
 
-The arguments passed to `cycle` by a module or to `watch` by a user are:
-- whether to watch a directory (`true`) or the network (`false`)
-- whether to continue watching indefinitely after the first report (`true` or `false`)
-- how many seconds to wait after finding no job before checking again (a nonnegative number)
-- optionally, where the watched jobs are located
+The directory where Testaro checks for jobs is specified by `process.env.JOBDIR`. Testaro checks for jobs in its `todo` subdirectory and, when it has performed a job, moves it into the `done` subdirectory.
 
-##### Directory watch
+Testaro creates a report for each job and saves the report in the directory specified by `process.env.REPORTDIR`.
+
+###### By a user
+
+A user can choose between two methods:
+
+```javaScript
+node call dirWatch true 300
+```
 
-With directory watch, Testaro checks whether the `todo` subdirectory of the job directory contains a job. The job directory is given by the fourth passed argument, if present, or, if not, then by `process.env.JOBDIR`.
+```javaScript
+node dirWatch true 300
+```
 
-When Testaro finds one or more jobs to do, the `watch` module runs the first job, saves the report in the `raw` subdirectory of the report directory. The report directory is given by `process.env.REPORTDIR`. Testaro also moves the job file from the `todo` subdirectory to the `done` subdirectory of the job directory.
+The arguments and behaviors described above for execution by a module apply here, too.
 
-Since Testaro runs the first job (i.e. the job whose file name is first in ASCII order), whoever populates the `todo` subdirectory of the job directory with job files has control over the order in which Testaro runs them. For example, to force a new job to be run before the already waiting jobs, one can give it a filename that comes before that of the first waiting job.
+The second, shorter method spawns a new watch subprocess after each job performance, to decrease the risk of process corruption involving bogus timeout messages from Playwright during jobs. That method requires you to enter `CTRL-c` to stop the watching.
 
 ##### Network watch
 
-Network watching is designed for a situation in which:
-- A managing server may be able to give work to multiple workstations that run Testaro.
-- A workstation running Testaro can contact a managing server, but the server may not be able to contact a workstation.
+Testaro can poll servers for jobs to be performed.
 
-With network watch, the initiator of an interaction is Testaro, not the server. When Testaro is available, it requests a job from a server. If the response is a JSON representation of a job, Testaro runs the job and sends the report to the server.
+An instance of Testaro is an _agent_ and has an identifier specified by `process.env.AGENT`. A Testaro instance identifies itself when polling servers, allowing servers to decide whether to give the instance a job to do.
 
-If multiple workstations run Testaro and do work for the same server, the server can assign jobs to specific agents by requiring each instance of Testaro to have a distinct value of `process.env.AGENT`.
+The URLs polled by Testaro are specified by `process.env.JOB_URLS`. The format of that environment variable is a `+`-delimited list of URLs, including schemes. If one of the URLs is `https://testrunner.org/a11ytest/api/job`, and if a Testaro instance has the agent ID `tester3`, then a job request is a `GET` request to `https://testrunner.org/a11ytest/api/job?agent=tester3`.
 
-The URL from which Testaro requests jobs is given by the fourth passed argument, if present, or, if not, then by `process.env.JOB_URL`.
+Once a Testaro instance obtains a network job, the report is sent in a `POST` request to the URL specified by the `sources.sendReportTo` property of the job.
 
-The URL to which Testaro sends reports is given by the `sources.sendReportTo` property of each job, if the property exists, or, if not, by `process.env.REPORT_URL`.
+###### By a module
 
-##### Job isolation
+```javaScript
+const {netWatch} = require('./watch');
+netWatch(true, 300);
+```
+
+In this example, a module asks Testaro to check the servers for a job every 300 seconds, to perform any jobs obtained from the servers, and then to continue checking until the process is stopped. If the first argument is `false`, Testaro will stop checking after performing 1 job.
 
-If you execute a repeating watch and the watch process becomes corrupted, the corruption can damage an indefinite subsequent sequence of job performances. Under some conditions, for example, Playwright has been observed to throw errors with a message starting with “Navigation timeout of 30000 ms exceeded” and ending with the entire HTML content of the web page, even when page navigation has been subjected to shorter time limits. Thereafter, Playwright issues that error message again about every 15 seconds, even if the browser has been closed.
+###### By a user
 
-To counteract such corruption, you can perform repeating directory watches with job isolation. To do this, use the statement `node dirWatch n`, where `n` is the number of seconds Testaro should wait after finding no job before looking again. The `dirWatch` module spawns a new process for each job, and also shrinks Playwright error messages before they are logged on the console.
+```javaScript
+node call netWatch true 300
+```
 
-You can stop a repeating directory watch of this kind by entering `CTRL-c`.
+The arguments and behaviors described above for execution by a module apply here, too. If the first argument is `true`, you can terminate the process by entering `CTRL-c`.
 
 ### Environment variables
 
@@ -824,7 +833,6 @@ Before making Testaro run a job, you can optionally also set `process.env.DEBUG`
 You may store environment variables in an untracked `.env` file if you wish, and Testaro will recognize them. Here is a template for a `.env` file:
 
 ```conf
-URL_INJECT=yes
 WAVE_KEY=yourwavekey
 JOB_URLs=https://yourserver.tld/job+http://localhost:3004/testapp
 JOBDIR=../testing/jobs/ThisWorkstation

package/call-old.js

@@ -0,0 +1,86 @@
+/*
+  call.js
+  Invokes Testaro modules with arguments.
+  This is the universal module for use of Testaro from a command line.
+  Arguments:
+    0. function to execute.
+    1+. arguments to pass to the function.
+  Usage examples:
+    node call run ts25
+    node call watch true true 30
+*/
+
+// ########## IMPORTS
+
+// Module to keep secrets.
+require('dotenv').config();
+// Module to process files.
+const fs = require('fs/promises');
+// Function to process a testing request.
+const {doJob} = require('./run');
+// Function to watch for jobs.
+const {watch} = require('./watch');
+
+// ########## CONSTANTS
+
+const fn = process.argv[2];
+const fnArgs = process.argv.slice(3);
+const jobDir = process.env.JOBDIR;
+const todoDir = `${jobDir}/todo`;
+const reportDir = process.env.REPORTDIR;
+const rawDir = `${reportDir}/raw`;
+
+// ########## FUNCTIONS
+
+// Fulfills a testing request.
+const callRun = async jobIDStart => {
+  // Find the job.
+  const jobDirFileNames = await fs.readdir(todoDir);
+  const jobFileName = jobDirFileNames.find(fileName => fileName.startsWith(jobIDStart));
+  // If it exists:
+  if (jobFileName) {
+    // Get it.
+    const jobJSON = await fs.readFile(`${todoDir}/${jobFileName}`, 'utf8');
+    const report = JSON.parse(jobJSON);
+    // Run it.
+    await doJob(report);
+    // Archive it.
+    await fs.rename(`${todoDir}/${jobFileName}`, `${jobDir}/done/${jobFileName}`);
+    // Save the report.
+    await fs.writeFile(`${rawDir}/${jobFileName}`, JSON.stringify(report, null, 2));
+    console.log(`Job completed and report ${report.id}.json saved in ${rawDir}`);
+  }
+  // Otherwise, i.e. if the job does not exist.
+  else {
+    // Report the error.
+    console.log(`ERROR: No to-do job ID starts with ${jobIDStart}`);
+  }
+};
+// Starts a watch.
+const callWatch = async (isDirWatch, interval) => {
+  const whenType = interval > -1 ? 'repeating' : 'one-time';
+  const whereType = isDirWatch === 'true' ? 'directory' : 'network';
+  console.log(`Starting ${whenType} ${whereType} watch`);
+  await watch(isDirWatch === 'true', Number.parseInt(interval, 10));
+};
+
+// ########## OPERATION
+
+// Execute the requested function.
+if (fn === 'run' && fnArgs.length === 1) {
+  callRun(fnArgs)
+  .then(() => {
+    console.log('Execution completed\n');
+    process.exit(0);
+  });
+}
+else if (fn === 'watch' && fnArgs.length === 2) {
+  callWatch(... fnArgs)
+  .then(() => {
+    console.log('Execution completed\n');
+    process.exit(0);
+  });
+}
+else {
+  console.log('ERROR: Invalid statement');
+}

package/call.js

@@ -1,16 +1,17 @@
+
 /*
   call.js
-  Invokes Testaro modules with arguments.
+  Invokes Testaro modules.
   This is the universal module for use of Testaro from a command line.
   Arguments:
     0. function to execute.
     1+. arguments to pass to the function.
   Usage examples:
-    node call run ts25
-    node call watch true true 30
+    node call run ts99
+    node call dirWatch true 30
 */
 
-// ########## IMPORTS
+// IMPORTS
 
 // Module to keep secrets.
 require('dotenv').config();
@@ -19,9 +20,9 @@ const fs = require('fs/promises');
 // Function to process a testing request.
 const {doJob} = require('./run');
 // Function to watch for jobs.
-const {cycle} = require('./watch');
+const {dirWatch, netWatch} = require('./watch');
 
-// ########## CONSTANTS
+// CONSTANTS
 
 const fn = process.argv[2];
 const fnArgs = process.argv.slice(3);
@@ -30,7 +31,7 @@ const todoDir = `${jobDir}/todo`;
 const reportDir = process.env.REPORTDIR;
 const rawDir = `${reportDir}/raw`;
 
-// ########## FUNCTIONS
+// FUNCTIONS
 
 // Fulfills a testing request.
 const callRun = async jobIDStart => {
@@ -56,15 +57,16 @@ const callRun = async jobIDStart => {
     console.log(`ERROR: No to-do job ID starts with ${jobIDStart}`);
   }
 };
-// Starts a watch.
-const callWatch = async (isDirWatch, isForever, interval, watchee = null) => {
-  const whenType = isForever === 'true' ? 'repeating' : 'one-time';
-  const whereType = isDirWatch === 'true' ? 'directory' : 'network';
-  console.log(`Starting ${whenType} ${whereType} watch`);
-  await cycle(isDirWatch === 'true', isForever === 'true', Number.parseInt(interval, 10), watchee);
+// Starts a directory watch, converting the interval argument to a number.
+const callDirWatch = async (isForever, interval) => {
+  await dirWatch(isForever === 'true', Math.max(5, Number.parseInt(interval, 10)));
+};
+// Starts a network watch, converting the interval argument to a number.
+const callNetWatch = async(isForever, interval) => {
+  netWatch(isForever === 'true', Number.parseInt(interval, 10));
 };
 
-// ########## OPERATION
+// OPERATION
 
 // Execute the requested function.
 if (fn === 'run' && fnArgs.length === 1) {
@@ -74,12 +76,11 @@ if (fn === 'run' && fnArgs.length === 1) {
     process.exit(0);
   });
 }
-else if (fn === 'watch' && fnArgs.length === 3) {
-  callWatch(... fnArgs)
-  .then(() => {
-    console.log('Execution completed\n');
-    process.exit(0);
-  });
+else if (fn === 'dirWatch' && fnArgs.length === 2) {
+  callDirWatch(... fnArgs);
+}
+else if (fn === 'netWatch' && fnArgs.length === 2) {
+  callNetWatch(... fnArgs);
 }
 else {
   console.log('ERROR: Invalid statement');

package/dirWatch.js

@@ -18,7 +18,7 @@ const interval = process.argv[2];
 const spawnWatch = (command, args) => spawn(command, args, {stdio: ['inherit', 'inherit', 'pipe']});
 // Repeatedly spawns a one-time directory watch.
 const reWatch = () => {
-  const watcher = spawnWatch('node', ['call', 'watch', 'true', 'false', interval]);
+  const watcher = spawnWatch('node', ['call', 'watch', 'false', interval]);
   let error = '';
   watcher.stderr.on('data', data => {
     error += data.toString();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testaro",
-  "version": "29.1.1",
+  "version": "30.0.0",
   "description": "Run 960 web accessibility tests from 9 tools and get a standardized report",
   "main": "index.js",
   "scripts": {

package/procs/tellServer.js

@@ -11,6 +11,7 @@ const agent = process.env.AGENT;
 
 // FUNCTIONS
 
+// Send a notification to an observer.
 exports.tellServer = (report, messageParams, logMessage) => {
   const observer = report.sources.sendReportTo.replace(/report$/, 'granular');
   const whoParams = `agent=${agent}&jobID=${report.id || ''}`;

package/run.js

@@ -464,7 +464,7 @@ const doActs = async (report, actIndex, page) => {
       }
       // If granular reporting has been specified:
       if (report.observe) {
-        // Notify the server of the act.
+        // Notify the observer of the act.
         const whichParam = act.which ? `&which=${act.which}` : '';
         const messageParams = `act=${act.type}${whichParam}`;
         tellServer(report, messageParams, `>>>> ${act.type}: ${actInfo}`);
@@ -744,7 +744,7 @@ const doActs = async (report, actIndex, page) => {
                 // Remove it, except any important property.
                 if (act.result.important) {
                   act.data = act.result.important;
-                  console.log('>>>>>> Important result data protected from deletion');
+                  console.log(`>>>>>> Important ${act.which} data included in report`);
                 }
                 delete act.result;
               }

package/test copy.js

@@ -0,0 +1,38 @@
+/*
+  watch.js
+  Module for watching for a job and running it when found.
+*/
+
+// ########## IMPORTS
+
+const httpClient = require('http');
+
+// ########## FUNCTIONS
+
+// Waits.
+const wait = ms => {
+  return new Promise(resolve => {
+    setTimeout(() => {
+      resolve('');
+    }, ms);
+  });
+};
+// Checks servers for a network job.
+const checkNetJob = () => {
+  const request = httpClient.request('http://localhost:3008/testu', response => {
+    const chunks = [];
+    response.on('data', chunk => {
+      chunks.push(chunk);
+    })
+    // When response arrives:
+    .on('end', () => {
+      // Report it.
+      const content = chunks.join('');
+      console.log(content);
+    });
+  });
+  request.end();
+};
+exports.watch = () => {
+  checkNetJob();
+}

package/watch-old.js

@@ -0,0 +1,275 @@
+/*
+  watch.js
+  Module for watching for a job and running it when found.
+*/
+
+// ########## IMPORTS
+
+// Module to keep secrets local.
+require('dotenv').config();
+// Module to read and write files.
+const fs = require('fs/promises');
+// Module to perform tests.
+const {doJob} = require('./run');
+// HTTP and HTTPS clients.
+const httpClient = require('http');
+const httpsClient = require('https');
+
+// ########## CONSTANTS
+
+const jobURLs = process.env.JOB_URLs;
+const agent = process.env.AGENT;
+const jobDir = process.env.JOBDIR;
+const reportDir = process.env.REPORTDIR;
+// Get a randomized array of servers to watch from the environment.
+const servers = jobURLs
+.split('+')
+.map(url => [Math.random(), url])
+.sort((a, b) => a[0] - b[0])
+.map(pair => pair[1]);
+
+// ########## FUNCTIONS
+
+// Returns a string representing the date and time.
+const nowString = () => (new Date()).toISOString().slice(0, 19);
+// Writes a directory report.
+const writeDirReport = async report => {
+  const jobID = report && report.id;
+  if (jobID) {
+    try {
+      const reportJSON = JSON.stringify(report, null, 2);
+      const reportName = `${jobID}.json`;
+      const rawDir = `${reportDir}/raw`;
+      await fs.mkdir(rawDir, {recursive: true});
+      await fs.writeFile(`${rawDir}/${reportName}`, reportJSON);
+      console.log(`Report ${reportName} saved in ${rawDir}`);
+    }
+    catch(error) {
+      console.log(`ERROR: Failed to write report ${jobID} in ${rawDir} (${error.message})`);
+    }
+  }
+  else {
+    console.log('ERROR: Job has no ID');
+  }
+};
+// Waits.
+const wait = ms => {
+  return new Promise(resolve => {
+    setTimeout(() => {
+      resolve('');
+    }, ms);
+  });
+};
+// Archives a job.
+const archiveJob = async job => {
+  const {id} = job;
+  const jobJSON = JSON.stringify(job, null, 2);
+  await fs.writeFile(`${jobDir}/done/${id}.json`, jobJSON);
+  await fs.rm(`${jobDir}/todo/${id}.json`);
+};
+// Checks for a directory job and, if found, performs and reports it, once or repeatedly.
+const checkDirJob = async (interval) => {
+  try {
+    // If there are any jobs to do in the watched directory:
+    const toDoFileNames = await fs.readdir(`${jobDir}/todo`);
+    const jobFileNames = toDoFileNames.filter(fileName => fileName.endsWith('.json'));
+    if (jobFileNames.length) {
+      // Get the first one.
+      const jobJSON = await fs.readFile(`${jobDir}/todo/${jobFileNames[0]}`, 'utf8');
+      try {
+        const job = JSON.parse(jobJSON, null, 2);
+        const {id} = job;
+        // Perform it.
+        console.log(`Directory job ${id} found (${nowString()})`);
+        await doJob(job);
+        console.log(`Job ${id} finished (${nowString()})`);
+        // Report it.
+        await writeDirReport(job);
+        // Archive it.
+        await archiveJob(job);
+        console.log(`Job ${id} archived in ${jobDir} (${nowString()})`);
+        // If watching is repetitive:
+        if (interval > -1) {
+          // Wait for the specified interval.
+          await wait(1000 * interval);
+          // Check the servers again.
+          checkDirJob(interval);
+        }
+      }
+      catch(error) {
+        console.log(`ERROR processing directory job (${error.message})`);
+      }
+    }
+    // Otherwise, i.e. if there are no more jobs to do in the watched directory:
+    else {
+      console.log(`No job to do in ${jobDir} (${nowString()})`);
+      // If checking is repetitive:
+      if (interval > -1) {
+        // Wait for the specified interval.
+        await wait(1000 * interval);
+        // Check the directory again.
+        await checkDirJob(interval);
+      }
+    }
+  }
+  catch(error) {
+    console.log(`ERROR: Directory watching failed (${error.message})`);
+  }
+};
+// Checks servers for a job and, if obtained, performs and reports it, once or repeatedly.
+const checkNetJob = async (serverIndex, interval) => {
+  // If any servers remain to be checked:
+  if (serverIndex < servers.length) {
+    // Request a job from the indexed server.
+    const server = servers[serverIndex];
+    const logStart = `Asked ${server} for a job and got `;
+    const wholeURL = `${server}?agent=${agent}`;
+    let client = server.startsWith('https://') ? httpsClient : httpClient;
+    console.log(`About to make request to ${wholeURL}`);
+    httpClient.request(
+      'http://localhost:8000', response => console.log(JSON.stringify(response, null, 2))
+    ).end();
+    /*
+    const jobRequest = client.request(wholeURL, {timeout: 1000}, response => {
+      console.log('Request made');
+      console.log(`Status: ${response.statusCode}`);
+      const chunks = [];
+      response.on('data', chunk => {
+        chunks.push(chunk);
+      })
+      // When the response is completed:
+      .on('end', async () => {
+        // If the response was JSON-formatted:
+        const responseJSON = chunks.join('');
+        try {
+          const responseObj = JSON.parse(responseJSON);
+          // If the server sent a valid job:
+          const {id, sources} = responseObj;
+          if (id && sources) {
+            const {sendReportTo} = sources;
+            if (sendReportTo) {
+              console.log(`Network job ${id} received from ${server} (${nowString()})`);
+              // Perform it.
+              await doJob(responseObj);
+              console.log(`Job ${id} finished (${nowString()})`);
+              // Send the report to the server and report its response.
+              console.log(`Sending report to ${sendReportTo}`);
+              client = sendReportTo.startsWith('https://') ? httpsClient : httpClient;
+              const request = client.request(sendReportTo, {method: 'POST'}, response => {
+                const chunks = [];
+                response.on('data', chunk => {
+                  chunks.push(chunk);
+                });
+                // When the response arrives:
+                response.on('end', async () => {
+                  const content = chunks.join('');
+                  const logStart = `Sent report to ${sendReportTo} and got `;
+                  try {
+                    const ack = JSON.parse(content);
+                    console.log(`${logStart}${ack}`);
+                    // Archive the job.
+                    await archiveJob(responseObj);
+                    console.log(`Job ${id} archived (${nowString()})`);
+                    // If watching is repetitive:
+                    if (interval > -1) {
+                      // Wait for the specified interval.
+                      await wait(1000 * interval);
+                      // Check the next server.
+                      await checkNetJob(serverIndex + 1, interval);
+                    }
+                  }
+                  catch(error) {
+                    console.log(
+                      `ERROR: ${logStart}status ${response.statusCode}, error message ${error.message}, and body ${content.slice(0,1000)}`
+                    );
+                  }
+                });
+              })
+              .on('error', error => {
+                console.log(`ERROR submitting job report (${error.message})`);
+              });
+              report.jobData.agent = agent;
+              const reportJSON = JSON.stringify(report, null, 2);
+              request.end(reportJSON);
+            }
+            // Otherwise, if the server sent a job without a report destination:
+            else {
+              // Report this.
+              console.log(`ERROR: ${logStart}a job with no report destination`);
+            }
+          }
+          // Otherwise, if the server sent a message instead of a job:
+          else if (responseObj.message) {
+            // Report it.
+            console.log(`${logStart}${responseObj.message}`);
+          }
+          // Otherwise, if the server sent any other JSON response:
+          else {
+            // Report it.
+            console.log(`${logStart} ${JSON.stringify(responseObj, null, 2)}`);
+            // Check the next server.
+            await checkNetJob(serverIndex + 1, interval);
+          }
+        }
+        catch(error) {
+          // Report any error.
+          console.log(
+            `${logStart}status ${response.statusCode} and reply ${responseJSON.slice(0, 1000)}`
+          );
+          // Check the next server.
+          await checkNetJob(serverIndex + 1, interval);
+        }
+      })
+      // If the response throws an error:
+      .on('error', async error => {
+        // Report it.
+        console.log(
+          `${logStart}status code ${response.statusCode} and error message ${error.message}`
+        );
+        // Check the next server.
+        await checkNetJob(serverIndex + 1, interval);
+      });
+    })
+    // If the request throws an error:
+    .on('error', async error => {
+      // Report it.
+      console.log(`${logStart} error ${error.message}`);
+      // Check the next server.
+      await checkNetJob(serverIndex + 1, interval);
+    })
+    // If the request times out:
+    .on('timeout', async () => {
+      // Report this.
+      console.log(`${logStart} a timeout`);
+      // Check the next server.
+      await checkNetJob(serverIndex + 1, interval);
+    });
+    // Close the request.
+    jobRequest.end();
+    console.log(`Request to ${server} ended`);
+  }
+  // Otherwise, i.e. if no servers remain to be checked:
+  else {
+    // If checking is repetitive:
+    if (interval > -1) {
+      // Wait for the specified interval.
+      await wait(1000 * interval);
+      // Check the servers again.
+      await checkNetJob(0, interval);
+    }
+  */
+  }
+};
+// Checks for a job, performs it, and submits a report, once or repeatedly.
+exports.watch = async (isDirWatch, interval = 300) => {
+  const intervalSpec = interval > -1 ? `repeatedly, with ${interval}-second intervals ` : '';
+  console.log(`Watching started ${intervalSpec}(${nowString()})\n`);
+  // Start the checking.
+  if (isDirWatch) {
+    await checkDirJob(interval);
+  }
+  else {
+    await checkNetJob(0, interval);
+  }
+  console.log(`Watching ended (${nowString()})`);
+};

package/watch-temp.js

@@ -0,0 +1,45 @@
+/*
+  watch.js
+  Module for watching for a job and running it when found.
+*/
+
+// ########## IMPORTS
+
+const httpClient = require('http');
+
+// ########## FUNCTIONS
+
+// Returns a string representing the date and time.
+const nowString = () => (new Date()).toISOString().slice(0, 19);
+// Checks servers for a network job.
+const checkNetJob = (serverIndex, interval) => {
+  const request = httpClient.request('http://localhost:3008/testu', response => {
+    console.log('Response fn being defined');
+    const chunks = [];
+    response.on('data', chunk => {
+      chunks.push(chunk);
+    })
+    // When response arrives:
+    .on('end', () => {
+      // Report it.
+      const content = chunks.join('');
+      console.log(content);
+      console.log(`Watching ended (${nowString()})`);
+    });
+  });
+  console.log('About to close request');
+  request.end();
+  console.log('Closed request');
+};
+// Checks for a job, performs it, and submits a report, once or repeatedly.
+exports.watch = async (isDirWatch, interval = 300) => {
+  const intervalSpec = interval > -1 ? `repeatedly, with ${interval}-second intervals ` : '';
+  console.log(`Watching started ${intervalSpec}(${nowString()})\n`);
+  // Start the checking.
+  if (isDirWatch) {
+    await checkDirJob(interval);
+  }
+  else {
+    checkNetJob(0, interval);
+  }
+};

package/watch.js

@@ -5,146 +5,39 @@
 
 // ########## IMPORTS
 
-// Module to keep secrets local.
+// Module to keep secrets.
 require('dotenv').config();
 // Module to read and write files.
 const fs = require('fs/promises');
-// Module to perform tests.
+// Module to make requests to servers.
+const httpClient = require('http');
+const httpsClient = require('https');
+// Module to perform jobs.
 const {doJob} = require('./run');
-// HTTP and HTTPS clients.
-const http = require('http');
-const https = require('https');
 
-// ########## CONSTANTS
+// CONSTANTS
 
-const httpClient = require('http');
-const httpsClient = require('https');
-const jobURLs = process.env.JOB_URLs;
-const agent = process.env.AGENT;
 const jobDir = process.env.JOBDIR;
+const jobURLs = process.env.JOB_URLS;
 const reportDir = process.env.REPORTDIR;
+const agent = process.env.AGENT;
 
 // ########## FUNCTIONS
 
 // Returns a string representing the date and time.
 const nowString = () => (new Date()).toISOString().slice(0, 19);
-// Checks for a directory job.
-const checkDirJob = async watchee => {
-  try {
-    const watchJobDir = watchee || jobDir;
-    const toDoDirFileNames = await fs.readdir(`${watchJobDir}/todo`);
-    const jobFileNames = toDoDirFileNames.filter(fileName => fileName.endsWith('.json'));
-    if (jobFileNames.length) {
-      const jobJSON = await fs.readFile(`${watchJobDir}/todo/${jobFileNames[0]}`, 'utf8');
-      try {
-        const job = JSON.parse(jobJSON, null, 2);
-        return job;
-      }
-      catch(error) {
-        return {
-          error: `ERROR parsing job as JSON (${error.message})`,
-          message: error.message
-        };
-      }
-    }
-    else {
-      console.log(`No job to do (${nowString()})`);
-      return {};
-    }
-  }
-  catch(error) {
-    console.log('ERROR: Directory watching failed');
-    return {};
-  }
+// Waits.
+const wait = ms => {
+  return new Promise(resolve => {
+    setTimeout(() => {
+      resolve('');
+    }, ms);
+  });
 };
-// Checks for and, if obtained, returns a network job.
-const checkNetJob = async watchee => {
-  let watchJobURLs = [watchee];
-  if (! watchJobURLs[0]) {
-    watchJobURLs = jobURLs
-    .split('+')
-    .map(url => [Math.random(), url])
-    .sort((a, b) => a[0] - b[0])
-    .map(pair => pair[1]);
-  }
-  // For each watchee:
-  for (const watchJobURL of watchJobURLs) {
-    const job = await new Promise(resolve => {
-      // Request a job from it.
-      const wholeURL = `${watchJobURL}?agent=${agent}`;
-      const client = wholeURL.startsWith('https://') ? httpsClient : httpClient;
-      const request = client.request(wholeURL, {timeout: 1000}, response => {
-        const chunks = [];
-        response.on('data', chunk => {
-          chunks.push(chunk);
-        })
-        // When response arrives:
-        .on('end', () => {
-          // If the response was JSON-formatted:
-          try {
-            const jobJSON = chunks.join('');
-            const job = JSON.parse(jobJSON);
-            // Make it the response of the watchee.
-            return resolve(job);
-          }
-          // Otherwise, i.e. if the response was not JSON-formatted:
-          catch(error) {
-            // Make an error report the response of the watchee.
-            const errorMessage = `ERROR: Response of ${watchJobURL} was not JSON`;
-            console.log(errorMessage);
-            return resolve({
-              error: errorMessage,
-              message: error.message,
-              status: response.statusCode
-            });
-          }
-        })
-        .on('error', error => {
-          return resolve({
-            error: 'ERROR getting network job',
-            message: error.message,
-            status: response.statusCode
-          });
-        });
-      });
-      // If the check threw an error:
-      request.on('error', error => {
-        // Make an error report the response of the watchee.
-        const errorMessage = `ERROR checking ${watchJobURL} for a network job`;
-        console.log(`${errorMessage} (${error.message})`);
-        return resolve({
-          error: errorMessage,
-          message: error.message
-        });
-      })
-      .on('timeout', () => {
-        const errorMessage = `ERROR: Request to ${watchJobURL} timed out`;
-        console.log(errorMessage);
-        return resolve({
-          error: errorMessage
-        });
-      });
-      request.end();
-    });
-    // If the watchee sent a job:
-    if (job.id) {
-      // Stop checking and return it.
-      console.log(`Network job ${job.id} received from ${watchJobURL} (${nowString()})`);
-      return job;
-    }
-    // Otherwise, if the watchee sent a message:
-    else if (job.message) {
-      // Report it and continue checking.
-      console.log(job.message);
-    }
-    // Otherwise, i.e. if the watchee sent neither a job nor a message:
-    else {
-      // Report this and continue checking.
-      console.log(`No network job at ${watchJobURL} to do (${nowString()})`);
-    }
-  }
-  // If no watchee sent a job, return this.
-  return {};
+// Serves an object in JSON format.
+const serveObject = (object, response) => {
+  response.setHeader('Content-Type', 'application/json; charset=utf-8');
+  response.end(JSON.stringify(object));
 };
 // Writes a directory report.
 const writeDirReport = async report => {
@@ -153,166 +46,302 @@ const writeDirReport = async report => {
     try {
       const reportJSON = JSON.stringify(report, null, 2);
       const reportName = `${jobID}.json`;
-      const rawDir = `${reportDir}/raw`;
-      await fs.writeFile(`${rawDir}/${reportName}`, reportJSON);
-      console.log(`Report ${reportName} saved in ${rawDir}`);
+      await fs.mkdir(reportDir, {recursive: true});
+      await fs.writeFile(`${reportDir}/${reportName}`, reportJSON);
+      console.log(`Report ${reportName} saved in ${reportDir}`);
     }
     catch(error) {
-      console.log(`ERROR: Failed to write report ${jobID} (${error.message})`);
+      console.log(`ERROR: Failed to write report ${jobID} in ${reportDir} (${error.message})`);
     }
   }
   else {
     console.log('ERROR: Job has no ID');
   }
 };
-// Submits a network report to a server and returns the server response.
-const writeNetReport = async report => {
-  const ack = await new Promise(resolve => {
-    if (report.sources) {
-      // If the report specifies where to send it:
-      const destination = report.sources.sendReportTo;
-      if (destination) {
-        // Send it.
-        const client = destination.startsWith('https://') ? https : http;
-        const request = client.request(destination, {method: 'POST'}, response => {
-          const chunks = [];
-          response.on('data', chunk => {
-            chunks.push(chunk);
-          });
-          response.on('end', () => {
-            const content = chunks.join('');
-            try {
-              resolve(JSON.parse(content));
-            }
-            catch(error) {
-              resolve({
-                error: 'ERROR: Response was not JSON',
-                message: error.message,
-                status: response.statusCode,
-                content: content.slice(0, 3000)
-              });
-            }
-          });
-        });
-        report.jobData.agent = agent;
-        request.on('error', error => {
-          console.log(`ERROR submitting job report (${error.message})`);
-          resolve({
-            error: 'ERROR: Job report submission failed',
-            message: error.message
-          });
-        });
-        const reportJSON = JSON.stringify(report, null, 2);
-        request.end(reportJSON);
-        console.log(`Report ${report.id} submitted (${nowString()})`);
+// Archives a job.
+const archiveJob = async job => {
+  const {id} = job;
+  const jobJSON = JSON.stringify(job, null, 2);
+  await fs.writeFile(`${jobDir}/done/${id}.json`, jobJSON);
+  await fs.rm(`${jobDir}/todo/${id}.json`);
+};
+// Checks for a directory job and, if found, performs and reports it, once or repeatedly.
+const checkDirJob = async (isForever, interval) => {
+  try {
+    // If there are any jobs to do in the watched directory:
+    const toDoFileNames = await fs.readdir(`${jobDir}/todo`);
+    const jobFileNames = toDoFileNames.filter(fileName => fileName.endsWith('.json'));
+    if (jobFileNames.length) {
+      // Get the first one.
+      const jobJSON = await fs.readFile(`${jobDir}/todo/${jobFileNames[0]}`, 'utf8');
+      try {
+        const job = JSON.parse(jobJSON, null, 2);
+        const {id} = job;
+        // Perform it.
+        console.log(`Directory job ${id} found (${nowString()})`);
+        await doJob(job);
+        console.log(`Job ${id} finished (${nowString()})`);
+        // Report it.
+        await writeDirReport(job);
+        // Archive it.
+        await archiveJob(job);
+        console.log(`Job ${id} archived in ${jobDir} (${nowString()})`);
+        // If watching is repetitive:
+        if (isForever) {
+          // Wait 2 seconds.
+          await wait(2000);
+          // Check the directory again.
+          checkDirJob(true, interval);
+        }
       }
-      // Otherwise, i.e. if the report does not specify where to send it:
-      else {
-        // Report this.
-        console.log('ERROR: Report specifies no submission destination');
+      catch(error) {
+        console.log(`ERROR processing directory job (${error.message})`);
       }
     }
+    // Otherwise, i.e. if there are no more jobs to do in the watched directory:
     else {
-      console.log('ERROR: Report has no sources property');
+      console.log(`No job to do in ${jobDir} (${nowString()})`);
+      // Wait for the specified interval.
+      await wait(1000 * interval);
+      // Check the directory again.
+      await checkDirJob(true, interval);
     }
-  });
-  // Return the server response.
-  if (ack) {
-    return ack.message || ack;
   }
-  else {
-    return '';
+  catch(error) {
+    console.log(`ERROR: Directory watching failed (${error.message})`);
   }
 };
-// Archives a job.
-const archiveJob = async (job, watchee) => {
-  const jobJSON = JSON.stringify(job, null, 2);
-  const watchJobDir = watchee || jobDir;
-  await fs.writeFile(`${watchJobDir}/done/${job.id}.json`, jobJSON);
-  await fs.rm(`${watchJobDir}/todo/${job.id}.json`);
-};
-// Waits.
-const wait = ms => {
-  return new Promise(resolve => {
-    setTimeout(() => {
-      resolve('');
-    }, ms);
-  });
-};
-// Runs a job.
-const runJob = async (job, isDirWatch) => {
-  // If the job has an ID:
-  const {id} = job;
-  if (id) {
-    try {
-      // Run the job, adding to the report.
-      await doJob(job);
-      // If a directory was watched:
-      if (isDirWatch) {
-        // Save the report.
-        await writeDirReport(job);
+// Checks servers for a network job.
+const checkNetJob = async (servers, serverIndex, isForever, interval, noJobCount) => {
+  console.log('');
+  // If all servers are jobless:
+  if (noJobCount === servers.length) {
+    // Wait for the specified interval.
+    await wait(1000 * interval);
+    // Reset the count of jobless servers.
+    noJobCount = 0;
+  }
+  // Otherwise, i.e. if any server may still have a job:
+  else {
+    // Wait 2 seconds.
+    await wait(2000);
+  }
+  // Check the next server.
+  serverIndex = serverIndex % servers.length;
+  const server = servers[serverIndex];
+  const client = server.startsWith('https://') ? httpsClient : httpClient;
+  const fullURL = `${server}?agent=${agent}`;
+  const logStart = `Requested job from server ${server} and got `;
+  client.request(fullURL, response => {
+    const chunks = [];
+    response
+    // If the response to the job request threw an error:
+    .on('error', async error => {
+      // Report it.
+      console.log(`${logStart}error message ${error.message}`);
+      // Check the next server.
+      await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+    })
+    .on('data', chunk => {
+      chunks.push(chunk);
+    })
+    // When the response arrives:
+    .on('end', async () => {
+      const content = chunks.join('');
+      // If there was no job to do:
+      try {
+        const contentObj = JSON.parse(content);
+        if (! Object.keys(contentObj).length) {
+          // Report this.
+          console.log(`No job to do at ${server}`);
+          // Check the next server.
+          await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+        }
+        // Otherwise, i.e. if there was a job or a message:
+        else {
+          const {message, id, sources} = contentObj;
+          // If the server sent a message, not a job:
+          if (message) {
+            // Report it.
+            console.log(`${logStart}${message}`);
+            // Check the next server.
+            await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+          }
+          // Otherwise, if the server sent a valid job:
+          else if (id && sources) {
+            // Add the agent to it.
+            sources.agent = agent;
+            // If the job specifies a report destination:
+            const {sendReportTo} = sources;
+            if (sendReportTo) {
+              // Perform the job, adding result data to it.
+              console.log(`${logStart}job ${id} for ${sendReportTo} (${nowString()})`);
+              await doJob(contentObj);
+              const reportJSON = JSON.stringify(contentObj, null, 2);
+              console.log(`Job ${id} finished (${nowString()})`);
+              // Send the report to the specified server.
+              console.log(`Sending report ${id} to ${sendReportTo}`);
+              const reportClient = sendReportTo.startsWith('https://') ? httpsClient : httpClient;
+              const reportLogStart = `Sent report ${id} to ${sendReportTo} and got `;
+              reportClient.request(sendReportTo, {method: 'POST'}, repResponse => {
+                const chunks = [];
+                repResponse
+                // If the response to the report threw an error:
+                .on('error', async error => {
+                  // Report this.
+                  console.log(`${reportLogStart}error message ${error.message}`);
+                  // Check the next server.
+                  await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+                })
+                .on('data', chunk => {
+                  chunks.push(chunk);
+                })
+                // When the response arrives:
+                .on('end', async () => {
+                  const content = chunks.join('');
+                  try {
+                    // If the server sent a message, as expected:
+                    const ackObj = JSON.parse(content);
+                    const {message} = ackObj;
+                    if (message) {
+                      // Report it.
+                      console.log(`${reportLogStart}${message}`);
+                      // Archive the job.
+                      await archiveJob(contentObj);
+                      console.log(`Job ${id} archived (${nowString()})`);
+                      // Check the next server.
+                      await checkNetJob(servers, serverIndex + 1, isForever, interval, 0);
+                    }
+                    // Otherwise, i.e. if the server sent anything else:
+                    else {
+                      // Report it.
+                      console.log(
+                        `ERROR: ${reportLogStart}status ${repResponse.statusCode} and error message ${JSON.stringify(ackObj, null, 2)}`
+                      );
+                      // Check the next server, disregarding the failed job.
+                      await checkNetJob(
+                        servers, serverIndex + 1, isForever, interval, noJobCount + 1
+                      );
+                    }
+                  }
+                  // If processing the report threw an error:
+                  catch(error) {
+                    // Report it.
+                    console.log(
+                      `ERROR: ${reportLogStart}status ${repResponse.statusCode} and response ${content.slice(0, 1000)}`
+                    );
+                    // Check the next server, disregarding the failed job.
+                    await checkNetJob(
+                      servers, serverIndex + 1, isForever, interval, noJobCount + 1
+                    );
+                  }
+                });
+              })
+              // If the report submission throws an error:
+              .on('error', async error => {
+                // Report this.
+                console.log(`ERROR: ${reportLogStart}error message ${error.message}`);
+                // Check the next server, disregarding the failed job.
+                await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+              })
+              // Finish submitting the report.
+              .end(reportJSON);
+            }
+            // Otherwise, i.e. if the job specifies no report destination:
+            else {
+              // Report this.
+              const message = `ERROR: ${logStart}job with no report destination`;
+              serveObject({message}, response);
+              console.log(message);
+              // Check the next server, disregarding the defective job.
+              await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+            }
+          }
+          // Otherwise, if the server sent an invalid job:
+          else {
+            // Report this.
+            const message
+            = `ERROR: ${logStart}invalid job:\n${JSON.stringify(contentObj, null, 2)}`;
+            console.log(message);
+            serveObject({message}, response);
+            // Check the next server, disregarding the defective job.
+            await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+          }
+        }
       }
-      // Otherwise, i.e. if the network was watched:
-      else {
-        // Send the report to the server and report its response.
-        const ack = await writeNetReport(job);
-        console.log(ack);
+      // If the response to the job request threw an error:
+      catch(error) {
+        // Report this.
+        console.log(
+          `ERROR: ${logStart}status ${response.statusCode}, error message ${error.message}, and response ${content.slice(0, 1000)}`
+        );
+        // Check the next server.
+        await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
       }
+    });
+  })
+  // If the job request throws an error:
+  .on('error', async error => {
+    // If it was a refusal to connect:
+    const {message} = error;
+    if (message.includes('ECONNREFUSED')) {
+      // Report this.
+      console.log(`${logStart}no connection`);
+      // Check the next server.
+      await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
     }
-    // If the job failed:
-    catch(error) {
+    // Otherwise, if it was a DNS failure:
+    else if (message.includes('ENOTFOUND')) {
+      // Report this.
+      console.log(`${logStart}no domain name resolution`);
+      // Check the next server.
+      await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
+    }
+    // Otherwise, i.e. if it was any other error:
+    else {
       // Report this.
-      console.log(`ERROR: ${error.message}\n${error.stack}`);
+      console.log(`ERROR: ${logStart}no response, but got error message ${error.message}`);
+      // Check the next server.
+      await checkNetJob(servers, serverIndex + 1, isForever, interval, noJobCount + 1);
     }
+  })
+  // Finish sending the request.
+  .end();
+};
+// Composes an interval description.
+const intervalSpec = interval => {
+  if (interval > -1) {
+    return `repeatedly, with ${interval}-second intervals `;
   }
-  // Otherwise, i.e. if the job has no ID:
   else {
-    // Report this.
-    console.log('ERROR: Job has no id');
+    return '';
   }
 };
-// Checks for a job, performs it, and submits a report, once or repeatedly.
-exports.cycle = async (isDirWatch, isForever, interval = 300, watchee = null) => {
-  let statusOK = true;
-  // Prevent a wait before the first iteration.
-  let empty = false;
-  const intervalMS = 1000 * Number.parseInt(interval);
-  const intervalSpec = isForever ? `with intervals of ${interval} seconds when idle ` : '';
-  console.log(`Watching started ${intervalSpec}(${nowString()})\n`);
-  while (statusOK) {
-    if (empty) {
-      await wait(intervalMS);
-    }
-    // Check for a job.
-    let job;
-    if (isDirWatch) {
-      job = await checkDirJob(watchee);
-    }
-    else {
-      job = await checkNetJob(watchee);
-    }
-    // If there was one:
-    if (job.id) {
-      // Run it, save a report, and if applicable send the report to the job source.
-      console.log(`Running job ${job.id} (${nowString()})`);
-      await runJob(JSON.parse(JSON.stringify(job)), isDirWatch);
-      console.log(`Job ${job.id} finished (${nowString()})`);
-      // If a directory was watched:
-      if (isDirWatch) {
-        // Archive the job.
-        await archiveJob(job, watchee);
-        console.log(`Job ${job.id} archived (${nowString()})`);
-      }
-      // If watching was specified for only 1 job, quit.
-      statusOK = isForever;
-      // Prevent a wait before the next iteration.
-      empty = false;
-    }
-    // Otherwise, i.e. if no job was found:
-    else {
-      // Cause a wait before the next check.
-      empty = true;
-    }
+// Checks for a directory job, performs it, and submits a report, once or repeatedly.
+exports.dirWatch = async (isForever, interval = 300) => {
+  console.log(`Directory watching started ${intervalSpec(interval)}(${nowString()})\n`);
+  // Start the checking.
+  await checkDirJob(isForever, interval);
+};
+// Checks for a network job, performs it, and submits a report, once or repeatedly.
+exports.netWatch = async (isForever, interval = 300) => {
+  // If the servers to be checked are valid:
+  const servers = jobURLs
+  .split('+')
+  .map(url => [Math.random(), url])
+  .sort((a, b) => a[0] - b[0])
+  .map(pair => pair[1]);
+  if (
+    servers
+    && servers.length
+    && servers
+    .every(server => ['http://', 'https://'].some(prefix => server.startsWith(prefix)))
+  ) {
+    console.log(`Network watching started ${intervalSpec(interval)}(${nowString()})\n`);
+    // Start checking.
+    await checkNetJob(servers, 0, isForever, interval, 0);
+  }
+  else {
+    console.log('ERROR: List of job URLs invalid');
   }
-  console.log(`Watching ended (${nowString()})`);
 };