testilo 3.13.0 → 4.0.0

package/README.md

@@ -1,72 +1,147 @@
 # testilo
-Scorer and digester of Testaro reports
+Utilities for Testaro
 
 ## Introduction
 
-This application enriches [Testaro](https://www.npmjs.com/package/testaro) reports. Testaro performs digital accessibility tests on web artifacts and creates reports in JSON format. To make those reports more useful, this application, Testilo, computes scores, converts the scored reports to human-readable web pages (_digests_), and compiles human-readable web pages comparing the scores of multiple artifacts.
+The Testilo package contains utilities that facilitate the use of the [Testaro package](https://www.npmjs.com/package/testaro).
+
+Testaro performs digital accessibility tests on web artifacts and creates reports in JSON format. The utilities in Testilo prepare jobs for Testaro to run and create additional value from the reports that Testaro produces.
 
 ## Dependencies
 
-The `dotenv` dependency lets you set environment variables in an untracked `.env` file.
+The `dotenv` dependency lets you set environment variables in an untracked `.env` file. This prevents secret data, such as passwords, from being shared as part of this package.
 
 ## Architecture
 
-The routines that perform scoring, digesting, and comparing are _procs_ and are located in the `procs` directory.
+Testilo is written in Node.js. Commands are given to Testilo in a command-line (terminal) interface or programmatically.
+
+Shared routines that perform scoring, digesting, and comparing are _procs_ and are located in the `procs` directory.
+
+Testilo can be installed wherever Node.js (version 14 or later) is installed. This can be a server or the same workstation on which Testaro is installed.
+
+The reason for Testilo being an independent package, rather than part of Testaro, is that Testilo can be installed on any host, while Testaro can run successfully only on a Windows or Macintosh workstation (and perhaps on some workstations with Ubuntu operating systems). Testaro runs tests similar to those that a human accessibility tester would run, using whatever browsers, input devices, system settings, simulated and attached devices, and assistive technologies tests may require. Thus, Testaro is limited to functionalities that require workstation attributes. For maximum flexibility in the management of Testaro jobs, all other functionalities are located outside of Testaro. You could have software such as Testilo running on a server, communicating with multiple workstations running Testaro, receiving job orders from the server and returning job results to the server for further processing.
+
+## Utilities
+
+### `merge`
+
+Testaro runs _jobs_. A job gives Testaro instructions.
+
+Sometimes you may want Testaro to perform a set of tests on multiple targets. The `merge` utility in Testilo facilitates this. It creates Testaro jobs out of a _script_ and a _batch_.
+
+A script tells Testaro where to go and what to do. If the where-to-go part identifies a specific URL, the script can be a job, ready for Testaro to run. But the where-to-go part can also be generic, such as `http://*.*`. Then the script needs to be converted to a job by having its generic destination replaced with a specific URL. A batch is a list of _hosts_ (URLs and metadata). Merging a batch and a script means generating from the script as many jobs as there are hosts in the batch. In each job, one host from the batch becomes the specific target.
+
+A script is a JSON file representing a `script` object, which contains a sequence of _commands_. Scripts are documented in detail in the Testaro `README.md` file.
+
+A batch is a JSON file representing a `batch` object, which contains an array of _hosts_. Each host is an object with three properties:
+- `id`: a string unique in the batch
+- `which`: a URL
+- `what`: a string naming the entity associated with the URL
+
+The batch as a whole also has three properties:
+- `id`: a string composed of ASCII letters and digits
+- `what`: a string describing the batch
+- `hosts`: an array of host objects
+
+Here is an example of a batch:
+
+```json
+{
+  "id": "usFedExec1",
+  "what": "United States federal executive agencies",
+  "hosts": [
+    {
+      "id": "achp",
+      "which": "https://www.achp.gov/",
+      "what": "Advisory Council on Historic Preservation (ACHP)"
+    },
+    {
+      "id": "agm",
+      "which": "https://www.usagm.gov/",
+      "what": "Agency for Global Media"
+    }
+  ]
+}
+```
+
+The `merge` utility finds all the `url` commands in a script (telling the browser what to visit), replaces them with one of the hosts in the batch, and outputs the modified script as a job. It then does the same with each of the other hosts in the batch.
+
+To execute `merge`, you need to have three environment variables defined to tell `merge` where files are located. You can define the environment variables in the `.env` file. Here is an example:
+
+```
+SCRIPTDIR=../testing/scripts
+BATCHDIR=../testing/batches
+JOBDIR=../testing/jobs
+```
+
+In this example, the script, batch, and job files are located in directories named `scripts`, `batches`, and `jobs` within `testing`, which is a sibling directory of the Testilo project directory.
+
+The syntax of the `merge` statement is documented in `merge.js`.
+
+Suppose that:
+- The environment variables are defined as in the above example.
+- There is a script file named `testall.json`.
+- The above batch example is in a file named `usFedExec1.json`.
+
+The statement `node merge testall usFedExec1 allFedExec` would tell Testilo to merge the batch `../testing/batches/usFedExec1.json` and the script `../testing/scripts/testall.json` and write the two job files in `../testing/jobs/allFedExec`.
+
+### `score`
 
-To score reports, Testilo needs to be told which reports to score and which scoring procedure (or _score proc_) to use.
+Testaro performs tests and produces reports of test results. Testilo can add scores to those reports. In this way, each report can not only detail successes and failures of individual tests but also assign scores to those results and combine the partial scores into total scores.
 
-Similarly, once reports have been scored, Testilo can digest them. For this purpose, Testilo needs to be told which scored reports to digest and which _digest proc_ to use.
+The `score` utility performs scoring. It depends on score procs to define the scoring rules. Some score procs are in the Testilo package (in the `procs/score` directory), and you can create more score procs to implement different rules.
 
-Likewise, Testilo can compare scored reports. It needs to be told which reports to compare and which _comparison proc_ to use.
+You can score multiple reports with a single invocation of `score`.
 
-Testilo includes some score procs, digest procs, and comparison procs. You can add others.
+To execute `score`, you need to have two environment variables defined to tell `score` where files are located. You can define the environment variables in the `.env` file. Here is an example:
 
-## Execution
+```
+REPORTDIR_RAW=../testing/reports/raw
+REPORTDIR_SCORED=../testing/reports/scored
+```
 
-### Scoring
+The named directories must already exist.
 
-#### Process
+The syntax of the `score` statement is documented in `score.js`.
 
-To score Testaro reports, execute the statement `node score procID reportNameStart`. Replace (here and below) `procID` with the base of the name of the score proc. An environment variable named `REPORTDIR_RAW` defines a directory (i.e. a filesystem path, relative to the project directory of Testilo) where Testilo will look for the raw (i.e. not-yet-scored) reports. If you want Testilo to score all of the reports in that directory, you can omit the `reportNameStart` argument. If, instead, you want Testilo to score only the reports in that directory whose names begin with a certain string, replace `reportNameStart` with that string. So, for example, suppose the raw reports are in the `reports/raw` directory of a project, `testing`, that sits alongside of Testilo in your filesystem. Then you would assign the value `'../testing/reports/raw'` to the environment variable `REPORTDIR_RAW`. Now, to score all of the reports in that directory with score proc `sp11a`, execute `node score sp11a`. Or, to score only the reports in that directory whose filenames begin with `4rper`, execute  node score sp11a `4rper`.
+A scored report file is identical to the file it was created from, except that the scored report has new properties named `score` and `scoreProcID`. The `score` property contains the scores.
 
-This procedure has two preconditions:
-- The score proc is compatible with the script (or _test proc_) that produced the report(s).
-- You have defined environment variables `REPORTDIR_RAW` and (for the scored reports that Testilo will write) `REPORTDIR_SCORED`.
+### `digest`
 
-The scored report file has the same name as the original. The scored report has the same content as the original, plus new properties named `score` and `scoreProcID`.
+Testaro reports, both originally and after they are scored, are JSON files. That format is basically designed for machine tractability.
 
-#### Procedures
+The `digest` utility converts scored reports into HTML documents with explanatory content. Thus, it converts machine-oriented reports into human-oriented reports, called _digests_. It depends on digest procs to define the digesting rules. Some digest procs are in the Testilo package (in the `procs/digest` directory), and you can create more digest procs to implement different rules.
 
-The score procs included with Testilo represent milestones in the refinement of a scoring methodology.
+You can digest multiple scored reports with a single invocation of `digest`.
 
-One development has been an expansion of the set of packages. The progression from score proc `sp09a` to `sp10a` included the addition of the Tenon package.
+To execute  `digest`, you need to have two environment variables defined to tell `digest` where files are located. You can define the environment variables in the `.env` file. Here is an example:
 
-The main development has been a change from package-based to issue-based scoring. With package-based scoring, each package yielded a score, and the scores were summed into a total score. With issue-based scoring, the individual tests in each package are classified into groups, such that the tests from various packages in any particular group all seek to discover approximately the same type of accessibility issue. An artifact gets a score on each group, and the group scores are summed into a total score. The change from package-based to issue-based scoring took place in the progression from score proc `sp10a` to `sp10b`.
+```
+REPORTDIR_SCORED=../testing/reports/scored
+REPORTDIR_DIGESTED=../testing/reports/digested
+```
 
-The problem of combining partly similar tests from different packages and producing an appropriate accessibility score has no perfect solution, and each successive score proc embodies efforts to make the result more appropriate.
+The named directories must already exist.
 
-#### Prevention
+The syntax of the `digest` statement is documented in `digest.js`.
 
-Testaro reports an error when it is unable to perform a test on a host. A score proc can take such errors into account. The score procs included with Testilo do that by estimating scores when a package cannot be run.
+The digests created by `digest` are HTML files, and they expect a `style.css` file to exist in their directory. The `reports/digested/style.css` file in Testilo is an appropriate stylesheet to be copied into `process.env.REPORTDIR_DIGESTED`.
 
-### Digesting
+### `compare`
 
-To make scored Testaro reports more useful for humans, Testilo can create digests of scored reports. A digest is an HTML document (a web page) summarizing and explaining the findings, with the scored report appended to it.
+You can summarize the results of a multi-host job by producing a document comparing the scores received by the hosts. The `compare` utility does this.
 
-To make Testilo digest reports, execute the statement `node digest procID reportNameStart`. The rules for this statement are the same as for the `score` statement, except that the directory where Testilo finds the reports in the one referenced by the `REPORTDIR_SCORED` environment variable, and the directory where Testilo will write the digests is the one referenced by `REPORTDIR_DIGESTED`. In order to make the digests appear correct in a browser, you must copy the `reports/digested/style.css` file into the `REPORTDIR_DIGESTED` directory.
+The `compare` utility gathers scores from a set of scored reports and produces an HTML document comparing the scores. It depends on compare procs to define the rules for making and showing the comparative scores. Some compare procs are in the Testilo package (in the `procs/compare` directory), and you can create more compare procs to implement different rules.
 
-### Comparing
+To execute  `compare`, you need to have two environment variables defined to tell `digest` where files are located. You can define the environment variables in the `.env` file. Here is an example:
 
-You can use Testilo to publish comparisons of accessibility scores. To do this, execute the statement `node compare abc xyz`, replacing `abc` with a filename base for the comparison and `xyz` with the name of a subdirectory of the `procs/compare` directory.
+```
+REPORTDIR_SCORED=../testing/reports/scored
+COMPARISONDIR=../testing/reports/compared
+```
 
-Testilo will examine all of the scored reports in the `REPORTDIR_SCORED` directory. The comparison proc in the `xyz` directory will construct a web page. Testilo will save the page in the `COMPARISONDIR` directory. The name of the file will be `abc.html`.
+The named directories must already exist.
 
-Comparison procs can design various pages on the basis of a set of scored reports. As an example, the `cp0` comparison proc creates a page that contains a table of scores, shown both numerically and with a bar graph. In the table, the first column contains descriptions of the pages (the `what` property of the hosts in the batch), such as “Wikipedia English”. Each such description is a link to the page on the web. The second column contains the scores of the pages. Each score is a link to the digest for its page. The link points to a digest located in a `digests` directory adjacent to the page itself. Thus, to use the `cp0` comparison proc, you would copy its output file to a web server, create a `digests` directory on the server as a sibling of that file, and copy the digest files into the `digests` directory.
+The syntax of the `compare` statement is documented in `compare.js`.
 
-This procedure has some preconditions:
-- The comparison proc is compatible with the score proc that scored the report.
-- Testilo can find the scored report files in the directory whose relative path (relative to the project directory of Testilo) is the value of the `REPORTDIR_SCORED` environment variable.
-- Testilo can read in the `REPORTDIR_SCORED` directory.
-- There is a `COMPARISONDIR` environment variable, whose value is the relative path of a directory that Testilo can write to.
-- The `procs/compare` directory contains a subdirectory named `xyz`, which in turn contains files named `index.html` and `index.js`.
-- You have copied the `reports/comparative/style.css` file into the `COMPARISONDIR` directory.
+The comparisons created by `compare` are HTML files, and they expect a `style.css` file to exist in their directory. The `reports/comparative/style.css` file in Testilo is an appropriate stylesheet to be copied into `process.env.COMPARISONDIR`.

package/compare.js

@@ -1,7 +1,12 @@
 /*
   compare.js
-  Testilo comparison script.
-  Usage example: node compare cp12a candidates
+  Creates comparisons from scored reports.
+  Reads reports in process.env.REPORTDIR_SCORED and outputs into process.env.COMPARISONDIR.
+  Arguments:
+    0. Base of name of compare proc located in procs/compare.
+    1. Base of name of comparative report to be written.
+  Usage example: node compare cp18a usFedExec
+  Note: Compares all reports in process.env.REPORTDIR_SCORED (but not in its subdirectories).
 */
 
 // ########## IMPORTS

package/digest.js

@@ -1,7 +1,13 @@
 /*
   digest.js
-  Testilo digesting script.
-  Usage example: node digest 35k1r-railpass dp10a
+  Creates digests from scored reports.
+  Reads reports in process.env.REPORTDIR_SCORED and outputs into process.env.REPORTDIR_DIGESTED.
+  Arguments:
+    0. Base of name of digest proc located in procs/digest.
+    1?. starting substring of names of reports in process.env.REPORTDIR_SCORED.
+  Usage examples:
+    node digest dp18a 35k1r (to digest all scored reports with names starting with 35k1r)
+    node digest dp18a (to digest all scored reports)
 */
 
 // ########## IMPORTS
@@ -23,7 +29,7 @@ const reportIDStart = process.argv[3];
 // Replaces the placeholders in content with eponymous query parameters.
 const replaceHolders = (content, query) => content
 .replace(/__([a-zA-Z]+)__/g, (ph, qp) => query[qp]);
-// Creates a digest.
+// Creates digests.
 const digest = async () => {
   const reportDirScoredAbs = `${__dirname}/${reportDirScored}`;
   let reportFileNames = await fs.readdir(reportDirScoredAbs);

package/merge.js

@@ -0,0 +1,72 @@
+/*
+  merge.js
+  Merges a batch and a script to produce jobs.
+  Arguments:
+    0. base of name of script located in process.env.SCRIPTDIR.
+    1. base of name of batch located in process.env.BATCHDIR.
+    2. name of subdirectory of process.env.JOBDIR into which to write host scripts.
+  Usage example:
+    node merge tp18 weborgs tp18-weborgs-1
+  Note: The subdirectory for host scripts will be created if it does not exist.
+*/
+
+// ########## IMPORTS
+
+// Module to keep secrets.
+require('dotenv').config();
+// Module to read and write files.
+const fs = require('fs/promises');
+
+// ########## CONSTANTS
+
+const scriptDir = process.env.SCRIPTDIR || 'scripts';
+const batchDir = process.env.BATCHDIR || 'batches';
+const jobDir = process.env.JOBDIR || 'jobs';
+const scriptName = process.argv[2];
+const batchName = process.argv[3];
+const jobSubdir = process.argv[4];
+
+// ########## FUNCTIONS
+
+// Merges a batch into a script and writes host scripts.
+const merge = async (script, batch, timeStamp, outDir) => {
+  await fs.mkdir(outDir, {recursive: true});
+  const {hosts} = batch;
+  // For each host in the batch:
+  const jobs = hosts.map(host => {
+    // Copy the script.
+    const newScript = JSON.parse(JSON.stringify(script));
+    // In the copy, make all url commands visit the host.
+    newScript.commands.forEach(command => {
+      if (command.type === 'url') {
+        command.which = host.which;
+        command.what = host.what;
+      }
+    });
+    // Create a host script.
+    return {
+      id: `${timeStamp}-${host.id}`,
+      host,
+      script: newScript
+    };
+  });
+  // Write the host scripts.
+  for (const job of jobs) {
+    await fs.writeFile(`${outDir}/${job.id}.json`, JSON.stringify(job, null, 2));
+  };
+  console.log(`Merger completed. Count of jobs created: ${hosts.length}`);
+};
+
+// ########## OPERATION
+
+fs.readFile(`${scriptDir}/${scriptName}.json`, 'utf8')
+.then(scriptFile => {
+  fs.readFile(`${batchDir}/${batchName}.json`, 'utf8')
+  .then(async batchFile => {
+    const script = JSON.parse(scriptFile);
+    const batch = JSON.parse(batchFile);
+    const timeStamp = Math.floor((Date.now() - Date.UTC(2022, 1)) / 2000).toString(36);
+    const outDir = `${jobDir}/${jobSubdir}`;
+    await merge(script, batch, timeStamp, outDir);
+  });
+});

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testilo",
-  "version": "3.13.0",
+  "version": "4.0.0",
   "description": "Client that scores and digests Testaro reports",
   "main": "index.js",
   "scripts": {

package/score.js

@@ -1,9 +1,13 @@
 /*
   score.js
-  Testilo scoring script.
+  Scores Testaro reports.
+  Reads reports in process.env.REPORTDIR_RAW and outputs into process.env.REPORTDIR_SCORED.
+  Arguments:
+    0. Base of name of score proc located in procs/score.
+    1?. Starting substring of names of reports in process.env.REPORTDIR_RAW.
   Usage examples:
-    node score sp12b 35k1r-railpass
-    node score sp12b
+    node score sp18a 35k1r (to score all reports with names starting with 35k1r)
+    node score sp18a (to score all reports)
 */
 
 // ########## IMPORTS