testaro 32.4.0 → 33.0.0

package/README.md

@@ -47,6 +47,7 @@ Testaro performs tests of these tools:
 - [alfa](https://alfa.siteimprove.com/) (Siteimprove)
 - [aslint](https://www.npmjs.com/package/@essentialaccessibility/aslint) (eSSENTIAL Accessibility)
 - [axe-playwright](https://www.npmjs.com/package/axe-playwright) (Deque)
+- [Editoria11y](https://github.com/itmaybejj/editoria11y) (Princeton University)
 - [HTML CodeSniffer](https://www.npmjs.com/package/html_codesniffer) (Squiz Labs)
 - [Nu Html Checker](https://github.com/validator/validator) (World Wide Web Consortium)
 - [QualWeb core](https://www.npmjs.com/package/@qualweb/core) (University of Lisbon)
@@ -57,37 +58,24 @@ Some of the tests of Testaro are designed to act as approximate alternatives to
 
 ## Rules
 
-Each tool accessed with Testaro defines _rules_ and tests _targets_ for compliance with its rules. In total, the nine tools define about 960 rules. The latest tabulation of tool rules is:
+Each tool accessed with Testaro defines _rules_ and tests _targets_ for compliance with its rules. In total, the nine tools define about a thousand rules. The latest tabulation of tool rules is:
 
 ```
 alfa: 59
 aslint: 136
 axe: 80
+ed11y: 24
 htmlcs: 115
 ibm: 132
 nuVal: 215
 qualWeb: 131
-testaro: 36
+testaro: 40
 wave: 58
-total: 962
+total: 990
 ```
 
 Some of the tools are under active development, and their rule counts change over time.
 
-When you ask Testaro to run tests of a tool, you may specify a subset of the rules of that tool, and the report will give you the results of only the tests for those rules. These tools will perform only those tests:
-- `alfa`
-- `axe`
-- `htmlcs`
-- `qualWeb`
-- `testaro`
-
-These tools always perform a fixed set of tests, and Testaro disregards irrelevant results when you specify a set of rules:
-- `ibm`
-- `nuVal`
-- `wave`
-
-The `aslint` tool does not yet allow rule specification.
-
 ## Job data
 
 A report produced by Testaro discloses:
@@ -154,6 +142,9 @@ Here is an example of a job:
   what: 'Test W3C with 2 alfa rules',
   strict: true,
   timeLimit: 65,
+  standard: 'only',
+  observe: false,
+  timeStamp: '231208T1200',
   acts: [
     {
       type: 'launch',
@@ -178,8 +169,7 @@ Here is an example of a job:
     },
     requester: 'user@domain.org'
   },
-  creationTime: '2024-12-10T14:28Z',
-  timeStamp: '241213T1200'
+  creationTime: '2024-12-10T14:28Z'
 }
 ```
 
@@ -192,14 +182,16 @@ Job properties:
 - `what`: a description of the job.
 - `strict`: `true` or `false`, indicating whether _substantive redirections_ should be treated as failures. These are redirections that do more than add or subtract a final slash.
 - `timeLimit`: the number of seconds allowed for the execution of the job.
+- `standard`: `'also'`, `'only'`, or `'no'`, indicating whether rule-violation instances are to be reported in tool-native formats and also in the Testaro standard format, only in the standard format, or only in the tool-native formats.
+- `observe`: `true` or `false`, indicating whether tool and Testaro-rule invocations are to be reported as they occur to the server.
+- `timeStamp`: a string in `yymmddThhMM` format, specifying a date and time before which the job is not to be performed.
 - `acts`: an array of the acts to be performed (documented below).
 - `sources`: an object describing where the job came from:
    - `script` (optional): the ID of the script from which the job was made.
    - `batch` (optional): a set of targets (URLs) from which the target of this job was drawn.
    - `target` (optional): an object describing the target being tested by this job.
    - `requester` (optional): the email address that should receive a notice of completion of the job.
-- `creationTime`: the time in ISO 8601 format when the job was created.
-- `timeStamp`: a string representing the date and time before which the job is not to be performed.
+- `creationTime`: the date and time in ISO 8601 format when the job was created.
 
 ### Reports
 
@@ -426,7 +418,7 @@ When you include a `rules` property, you limit the tests of the tool that are pe
 
 The `nuVal`, `qualWeb`, and `testaro` tools require specific formats for the `rules` property. Those formats are described below in the sections about those tools.
 
-The `aslint` tool does not yet allow rule specification.
+The `aslint` and `ed11y` tools do not yet allow rule specification.
 
 ###### Examples
 
@@ -504,11 +496,32 @@ A typical use for an `expect` property is checking the correctness of a Testaro
 
 When a `test` act has an `expect` property, the result for that act has an `expectations` property reporting whether the expectations were satisfied. The value of `expectations` is an array of objects, one object per expectation. Each object includes a `property` property identifying the expectation, and a `passed` property with `true` or `false` value reporting whether the expectation was satisfied. If applicable, it also has other properties identifying what was expected and what was actually reported.
 
-###### ASLint
+##### Branching
+
+An example of a **branching** act is:
+
+```json
+{
+  "type": "next",
+  "if": ["totals.invalid", ">", 0],
+  "jump": -4,
+  "what": "redo search if any invalid elements"
+}
+```
+
+This act checks the result of the previous act to determine whether its `result.totals.invalid` property has a positive value. If so, it changes the next act to be performed, specifying the act 4 acts before this one.
+
+A `next` act can use a `next` property instead of a `jump` property. The value of the `next` property is an act name. It tells Testaro to continue performing acts starting with the act having that value as the value of its `name` property.
+
+#### Tools
+
+The tools whose tests Testaro performs have particularities described below.
+
+##### ASLint
 
 The `aslint` tool makes use of the [`aslint-testaro` fork](https://www.npmjs.com/package/aslint-testaro) of the [`aslint` repository](https://github.com/essentialaccessibility/aslint), which, unlike the published `aslint` package, contains the `aslint.bundle.js` file.
 
-###### HTML CodeSniffer
+##### HTML CodeSniffer
 
 The `htmlcs` tool makes use of the `htmlcs/HTMLCS.js` file. That file was created, and can be recreated if necessary, as follows:
 
@@ -540,7 +553,7 @@ The changes in `htmlcs/HTMLCS.js` are:
 >       );
 ```
 
-###### IBM Equal Access
+##### IBM Equal Access
 
 The `ibm` tests require the `aceconfig.js` file.
 
@@ -567,13 +580,13 @@ The `ibm` tool is one of two tools (`testaro` is the other) with a `withItems` p
 
 Experimentation indicates that the `ibm` tools emits untrappable errors for some targets when the content argument given to it is the page content rather than the page URL. Therefore, it is safer to use `true` as the value of `withNewContent` for the `ibm` tool.
 
-###### Nu Html Checker
+##### Nu Html Checker
 
 The `nuVal` tool performs the tests of the Nu Html Checker.
 
 Its `rules` argument is **not** an array of rule IDs, but instead is an array of rule _specifications_. A rule specification for `nuVal` is a string with the format `=ruleID` or `~ruleID`. The `=` prefix indicates that the rule ID is invariable. The `~` prefix indicates that the rule ID is variable, in which case the `ruleID` part of the specification is a matching regular expression, rather than the exact text of a message. This `rules` format arises from the fact that `nuVal` generates customized messages and does not accompany them with rule identifiers.
 
-###### QualWeb
+##### QualWeb
 
 The `qualWeb` tool performs the ACT rules, WCAG Techniques, and best-practices tests of QualWeb. Only failures and warnings are included in the report. The EARL report of QualWeb is not generated, because it is equivalent to the report of the ACT rules tests.
 
@@ -597,7 +610,7 @@ Thus, when the `rules` argument is omitted, QualWeb will test for all of the rul
 
 The target can be provided to QualWeb either as an existing page or as a URL. Experience indicates that the results can differ between these methods, with each method reporting some rule violations or some instances that the other method does not report.
 
-###### Testaro
+##### Testaro
 
 If you do not specify rules when using the `testaro` tool, Testaro will test for the rules listed in the `evalRules` object of the `tests/testaro.js` file.
 
@@ -613,7 +626,7 @@ Several Testaro tests make use of the `init()` function in the `procs/testaro` m
 
 You can add custom rules to the rules of any tool. Testaro provides a template, `data/template.js`, for the definition of a rule to be added. Once you have created a copy of the template with revisions, you can move the copy into the `testaro` directory and add an entry for your custom rule to the `evalRules` object in the `tests/testaro.js` file. Then your custom rule will act as a Testaro rule. Some `testaro` rules are simple enough to be fully specified in JSON files. You can use any of those as a template if you want to create a sufficiently simple custom rule, namely a rule whose prohibited elements are all and only the elements matching a CSS selector. More details about rule creation are in the `CONTRIBUTING.md` file.
 
-###### WAVE
+##### WAVE
 
 If a `wave` test act is included in the job, an environment variable named `WAVE_KEY` must exist, with your WAVE API key as its value. You can get it from [WebAIM](https://wave.webaim.org/api/).
 
@@ -621,23 +634,6 @@ The `wave` API does not accept a transmitted document for testing. WAVE must be
 
 This limitation of WAVE may be overcome in a future version of Testaro by means of the invocation of the WAVE Chrome extension with Playwright.
 
-##### Branching
-
-An example of a **branching** act is:
-
-```json
-{
-  "type": "next",
-  "if": ["totals.invalid", ">", 0],
-  "jump": -4,
-  "what": "redo search if any invalid elements"
-}
-```
-
-This act checks the result of the previous act to determine whether its `result.totals.invalid` property has a positive value. If so, it changes the next act to be performed, specifying the act 4 acts before this one.
-
-A `next` act can use a `next` property instead of a `jump` property. The value of the `next` property is an act name. It tells Testaro to continue performing acts starting with the act having that value as the value of its `name` property.
-
 #### Browser types
 
 After any act in a job, you can change the browser type by inserting a `launch` act. One reason for specifying a particular browser type is that particular tests have different results with different browser types. Another is that you may wish to perform tests with more than a single browser type.
@@ -884,6 +880,12 @@ Testing to determine what happens when a control or link is activated is straigh
 
 The Playwright “Receives Events” actionability check does **not** check whether an event is dispatched on an element. It checks only whether a click on the location of the element makes the element the target of that click, rather than some other element occupying the same location.
 
+### Test prevention
+
+Test targets employ mechanisms to prevent scraping, automated form submission, and other automated actions. These mechanisms may interfere with testing. When a test act is prevented by a target, Testaro reports this prevention.
+
+Some targets prohibit the execution of alien scripts unless the client can demonstrate that it is the requester of the page. Failure to provide that evidence results in the script being blocked and an error message being logged, saying “Refused to execute a script because its hash, its nonce, or unsafe-inline does not appear in the script-src directive of the Content Security Policy”. This mechanism affects tools that insert scripts into a target in order to test it. Those tools include `axe`, `asLint`, `ed11y`, and `htmlcs`. To comply with this requirement, Testaro obtains a _nonce_ from the response that serves the target. Then the file that runs the tool adds that nonce to the script as the value of a `nonce` attribute when it inserts its script into the target.
+
 ### Tool duplicativity
 
 Tools sometimes do redundant testing, in that two or more tools test for the same defects, although such duplications are not necessarily perfect. This fact creates three problems:

package/call.js

@@ -59,9 +59,11 @@ const rawDir = `${reportDir}/raw`;
 const callRun = async jobIDStart => {
   // Find the job.
   const jobDirFileNames = await fs.readdir(todoDir);
-  const jobFileName = jobIDStart
-  ? jobDirFileNames.find(fileName => fileName.startsWith(jobIDStart))
-  : jobDirFileNames[0];
+  const jobFileNames = jobDirFileNames.filter(fileName => fileName.endsWith('.json'));
+  const specifiedJobFileNames = jobIDStart
+  ? jobFileNames.filter(fileName => fileName.startsWith(jobIDStart))
+  : jobFileNames;
+  const jobFileName = specifiedJobFileNames[0];
   // If it exists:
   if (jobFileName) {
     // Get it.
@@ -78,7 +80,7 @@ const callRun = async jobIDStart => {
   // Otherwise, i.e. if the job does not exist.
   else {
     // Report the error.
-    console.log(`ERROR: No to-do job ID starts with ${jobIDStart}`);
+    console.log('ERROR: No specified job exists');
   }
 };
 // Starts a directory watch, converting the interval argument to a number.