testaro 68.0.0 → 69.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +232 -683
- package/UPGRADES.md +4 -0
- package/VALIDATION.md +62 -1
- package/actSpecs-doc.md +53 -0
- package/package.json +1 -1
- package/procs/catalog.js +2 -2
- package/tests/htmlcs.js +1 -1
- package/tests/testaro.js +1 -4
package/README.md
CHANGED
|
@@ -6,48 +6,35 @@ Ensemble testing for web accessibility
|
|
|
6
6
|
|
|
7
7
|
Version 68.0.0 introduces major breaking changes.
|
|
8
8
|
|
|
9
|
-
Any application that has successfully relied on version 67.1.0 is likely to fail if it updates
|
|
9
|
+
Any application that has successfully relied on version 67.1.0 is likely to fail if it updates the `testaro` dependency to version 68.0.0 or later. To prevent such failures, pin `testaro` to version 67.1.0 in your `package.json` file.
|
|
10
10
|
|
|
11
11
|
Revision of this `README` document to reflect version 68.0.0 is in progress but is incomplete.
|
|
12
12
|
|
|
13
|
-
##
|
|
13
|
+
## Purposes
|
|
14
14
|
|
|
15
|
-
Testaro is an application
|
|
15
|
+
Testaro is an application that performs ensemble testing of web pages, primarily for accessibility.
|
|
16
16
|
|
|
17
17
|
The purposes of Testaro are to:
|
|
18
18
|
|
|
19
|
-
- provide programmatic access to
|
|
20
|
-
-
|
|
19
|
+
- provide programmatic access to tests defined by multiple tools
|
|
20
|
+
- standardize and integrate the reports of the tools
|
|
21
|
+
|
|
22
|
+
The need for ensemble testing of web accessibility, and the obstacles to it, are discussed in [Accessibility Metatesting: Comparing Nine Testing Tools](https://arxiv.org/abs/2304.07591).
|
|
21
23
|
|
|
22
24
|
Testaro is described in two papers:
|
|
23
25
|
|
|
24
26
|
- [How to run a thousand accessibility tests](https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3)
|
|
25
27
|
- [Testaro: Efficient Ensemble Testing for Web Accessibility](https://arxiv.org/abs/2309.10167)
|
|
26
28
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
Testaro launches and controls web browsers, performing operations, conducting tests, and recording results.
|
|
29
|
+
## Functionality
|
|
30
30
|
|
|
31
|
-
Testaro
|
|
31
|
+
Testaro performs tasks defined by a _job_. Typically, a job identifies the URL of a web page and asks Testaro to call an ensemble of 11 tools to test the page. Testaro adds the results of the testing to the job, thereby converting the job into a _report_.
|
|
32
32
|
|
|
33
|
-
Testaro
|
|
33
|
+
Testaro can be given a job to perform, in which case it performs the job, delivers the report, and quits.
|
|
34
34
|
|
|
35
|
-
|
|
35
|
+
Alternatively, testaro can run as a daemon, listening for jobs and performing them when they appear.
|
|
36
36
|
|
|
37
|
-
-
|
|
38
|
-
- Converting user specifications into jobs
|
|
39
|
-
- Job scheduling
|
|
40
|
-
- Monitoring of the health of Testaro
|
|
41
|
-
- Management of clusters of workstations sharing workloads
|
|
42
|
-
- Allocation of responsibilities among workstations
|
|
43
|
-
- Receiving and fulfilling user requests for jobs
|
|
44
|
-
- Allocating testing responsibilities to human testers
|
|
45
|
-
- Combining reports from workstations and human testers
|
|
46
|
-
- Analyzing and summarizing (e.g., computing scores on the basis of) test results
|
|
47
|
-
- Sending notifications
|
|
48
|
-
- Revising, combining, and publishing reports
|
|
49
|
-
|
|
50
|
-
One software product that performs some such functions is [Testilo](https://www.npmjs.com/package/testilo).
|
|
37
|
+
A practical application that leverages Testaro will use other software to prepare jobs, schedule them, post-process the reports as needed, and manage the report files. Some utilities for such purposes can be found in the [Testilo project](https://www.npmjs.com/package/testilo). One application that leverages Testaro is [Kilotest](https://github.com/jrpool/kilotest).
|
|
51
38
|
|
|
52
39
|
## Dependencies
|
|
53
40
|
|
|
@@ -57,8 +44,9 @@ Testaro uses:
|
|
|
57
44
|
- [playwright-extra](https://www.npmjs.com/package/playwright-extra) and [puppeteer-extra-plugin-stealth](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to make a Playwright-controlled browser more indistinguishable from a human-operated browser and thus make their requests more likely to succeed
|
|
58
45
|
- [playwright-dompath](https://www.npmjs.com/package/playwright-dompath) to retrieve XPaths of elements
|
|
59
46
|
- [BlazeDiff](https://blazediff.dev/) to measure motion
|
|
47
|
+
- [dotenv](https://www.npmjs.com/package/dotenv) to load environment variables
|
|
60
48
|
|
|
61
|
-
Testaro
|
|
49
|
+
Testaro can perform tests of these _tools_:
|
|
62
50
|
|
|
63
51
|
- [Accessibility Checker](https://www.npmjs.com/package/accessibility-checker) (IBM)
|
|
64
52
|
- [Alfa](https://alfa.siteimprove.com/) (Siteimprove)
|
|
@@ -72,99 +60,107 @@ Testaro performs tests of these _tools_:
|
|
|
72
60
|
- [WallyAX](https://www.npmjs.com/package/@wally-ax/wax-dev) (Wally Solutions)
|
|
73
61
|
- [WAVE](https://wave.webaim.org/api/) (WebAIM)
|
|
74
62
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
## Rules
|
|
78
|
-
|
|
79
|
-
Each tool accessed with Testaro defines _rules_ and tests _targets_ for compliance with its rules. Testilo has classified the rules into _issues_ and deprecated some rules as poorly implemented. If the deprecated rules are excluded, the counts are:
|
|
80
|
-
|
|
81
|
-
```text
|
|
82
|
-
Accessibility Checker: 93
|
|
83
|
-
Alfa: 64
|
|
84
|
-
ASLint: 129
|
|
85
|
-
Axe: 79
|
|
86
|
-
Editoria11y: 23
|
|
87
|
-
HTML CodeSniffer: 110
|
|
88
|
-
Nu Html Checker: 260
|
|
89
|
-
QualWeb: 115
|
|
90
|
-
Testaro: 57
|
|
91
|
-
WallyAX: 27
|
|
92
|
-
WAVE: 60
|
|
93
|
-
total: 1017
|
|
94
|
-
```
|
|
63
|
+
For the tools that are open-source, the identified organizations are their principal or original sponsors.
|
|
95
64
|
|
|
96
|
-
|
|
65
|
+
As shown, Testaro is not only an integrator but also one of the 11 integrated tools. That is because it provides about 50 tests of its own, mostly to complement tests provided by the other 10 tools. Some of those Testaro tests are designed to act as approximate alternatives to tests of vulnerable, restricted, or no longer available tools. In all such cases the Testaro tests are independently designed and implemented, without reference to the code of the tests that inspired them.
|
|
97
66
|
|
|
98
|
-
##
|
|
67
|
+
## Concepts and terms
|
|
99
68
|
|
|
100
|
-
The main
|
|
69
|
+
The main concepts of Testaro are:
|
|
101
70
|
|
|
102
|
-
-
|
|
103
|
-
- `
|
|
104
|
-
- `
|
|
105
|
-
- `
|
|
71
|
+
- `job`: a document that tells Testaro what to do.
|
|
72
|
+
- `act`: one step in a job
|
|
73
|
+
- `report`: a job that Testaro has added results to.
|
|
74
|
+
- `tool`: one of the (currently 11) testing applications in the ensemble that Testaro has created.
|
|
75
|
+
- `rule`: a success or failure criterion defined by a tool (currently about a thousand across all tools).
|
|
76
|
+
- `test`: the software that a tool uses to apply a rule.
|
|
77
|
+
- `target`: a web page that a job tells Testaro to test.
|
|
78
|
+
- `result`: the information that Testaro adds to a job to describe the test outcomes.
|
|
79
|
+
- `native result`: the test outcomes of a tool in the native form of that tool.
|
|
80
|
+
- `standard result`: the test outcomes of a tool in a uniform Testaro-defined form.
|
|
81
|
+
- `catalog`: a collection of data on the HTML elements that fail one or more tests.
|
|
106
82
|
|
|
107
83
|
## System requirements
|
|
108
84
|
|
|
109
|
-
|
|
85
|
+
Testaro can be installed under a MacOS, Windows, Debian, or Ubuntu operating system.
|
|
110
86
|
|
|
111
|
-
|
|
87
|
+
Testaro is tested with the latest long-term-support version of [Node.js](https://nodejs.org/en/).
|
|
112
88
|
|
|
113
|
-
|
|
89
|
+
Testaro is configured so that, when Playwright or Puppeteer (a dependency of Playwright and of some tools) launches a `chromium` browser, the browser is [sandboxed](https://www.geeksforgeeks.org/ethical-hacking/what-is-browser-sandboxing/) for improved security. That is the default for Playwright and Puppeteer, and Testaro does not override that default. The host must therefore permit sandboxed browsers. Documentation on how to configure an Ubuntu Linux host for this purpose is available in the [`SERVICE.md` file of the Kilotest repository](https://github.com/jrpool/kilotest/blob/main/SERVICE.md#browser-privileges). If you try to run Testaro on a host that prohibits sandboxed browsers, each attempted launch of a `chromium` browser will throw an error with a message complaining about the unavailability of a sandbox.
|
|
114
90
|
|
|
115
|
-
|
|
91
|
+
## Installation
|
|
116
92
|
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
cd testaro
|
|
121
|
-
npm install
|
|
122
|
-
npx playwright install
|
|
123
|
-
```
|
|
93
|
+
### As an independent application
|
|
94
|
+
|
|
95
|
+
To install Testaro as an independent application, clone the [Testaro repository](https://github.com/jrpool/testaro). To ensure that the binary browsers of its Playwright dependency get installed, execute `(p)npx playwright install` after executing `(p)npm install`.
|
|
124
96
|
|
|
125
|
-
|
|
97
|
+
To update Testaro when it is an independent application, execute:
|
|
126
98
|
|
|
127
99
|
```bash
|
|
128
|
-
cd testaro
|
|
129
100
|
git checkout package-lock.json
|
|
130
101
|
git pull
|
|
131
|
-
npm run deps
|
|
102
|
+
(p)npm run deps
|
|
132
103
|
```
|
|
133
104
|
|
|
134
|
-
Once it is installed as an application, you can use its features with a terminal interface by executing the “By a user” statements below.
|
|
135
|
-
|
|
136
105
|
### As a dependency
|
|
137
106
|
|
|
138
|
-
You can make
|
|
139
|
-
|
|
140
|
-
Once it is installed as a dependency, your application can use Testaro features by executing the “By a module” statements below.
|
|
141
|
-
|
|
142
|
-
### Prerequisites
|
|
107
|
+
You can make `testaro` a dependency in another application. As noted at the beginning of this file, the entry in `package.json` should be `"testaro": "67.1.0"` if your application has not been designed to work with version 68.0.0 or later.
|
|
143
108
|
|
|
144
|
-
|
|
109
|
+
## Environment configuration
|
|
145
110
|
|
|
146
|
-
|
|
111
|
+
The `.env` file stores your decisions about the environment in which Testaro runs. The variables that can be defined there are:
|
|
147
112
|
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
113
|
+
```conf
|
|
114
|
+
# Whether the browsers launched by Testaro should have visible windows.
|
|
115
|
+
HEADED_BROWSER=false
|
|
116
|
+
# Whether console logging in launched browsers should be mirrored to the Testaro console.
|
|
117
|
+
DEBUG=false
|
|
118
|
+
# Whether to disable Puppeteer log warnings of a future headless-mode deprecation.
|
|
119
|
+
PUPPETEER_DISABLE_HEADLESS_WARNING=true
|
|
120
|
+
# How much time, in milliseconds, to insert between Playwright operations for debugging.
|
|
121
|
+
WAITS=0
|
|
122
|
+
# API key to enable the WAVE tool.
|
|
123
|
+
WAVE_KEY=yourwavekey (get it from [WebAim](https://wave.webaim.org/api/)).
|
|
124
|
+
# `proTestKit` API key to enable the `npm Package` of the WallyAX tool.
|
|
125
|
+
WAX_KEY=yourwaxkey (get it from [WallyAX](https://account.wallyax.com/?ref_app=Developer&app_type=npm)).
|
|
126
|
+
#----------------------------
|
|
127
|
+
# When Testaro listens for new jobs in a directory:
|
|
128
|
+
# Directory where it listens for them.
|
|
129
|
+
JOBDIR=../testing/jobs
|
|
130
|
+
# Directory into which Testaro saves the reports of those jobs.
|
|
131
|
+
REPORTDIR=../testing/reports
|
|
132
|
+
# Name of this Testaro instance when it listens for jobs and sends reports to requesting hosts.
|
|
133
|
+
AGENT=agentabc
|
|
134
|
+
#----------------------------
|
|
135
|
+
# When Testaro polls network hosts to ask for new jobs, data on those hosts.
|
|
136
|
+
# URL of host 0 to poll.
|
|
137
|
+
NETWATCH_URL_0_JOB=http://localhost:3000/api/assignJob/agentabc
|
|
138
|
+
# URL of host 0 to which to send progress reports during jobs.
|
|
139
|
+
NETWATCH_URL_0_OBSERVE=http://localhost:3000/api/granular/agentabc
|
|
140
|
+
# URL of host 0 to which to send completed job reports.
|
|
141
|
+
NETWATCH_URL_0_REPORT=http://localhost:3000/api/takeReport/agentabc
|
|
142
|
+
# Password to give to host 0 to authenticate this instance.
|
|
143
|
+
NETWATCH_URL_0_AUTH=abcxyz
|
|
144
|
+
# Which network hosts to poll for jobs (comma-separated list of indices).
|
|
145
|
+
NETWATCH_URLS=0
|
|
146
|
+
```
|
|
151
147
|
|
|
152
148
|
## Jobs
|
|
153
149
|
|
|
154
|
-
|
|
150
|
+
Jobs tell Testaro what and how to test.
|
|
155
151
|
|
|
156
|
-
###
|
|
152
|
+
### Job example
|
|
157
153
|
|
|
158
|
-
Here is
|
|
154
|
+
Here is a sample job, showing properties that you can set:
|
|
159
155
|
|
|
160
156
|
```javaScript
|
|
161
157
|
{
|
|
162
|
-
id: '
|
|
163
|
-
what: 'monthly health check',
|
|
164
|
-
strict: true,
|
|
165
|
-
standard: 'also',
|
|
166
|
-
observe: false,
|
|
167
|
-
device: {
|
|
158
|
+
id: 'healthcheck2611', // Job identifier
|
|
159
|
+
what: 'monthly health check', // Job description
|
|
160
|
+
strict: true, // Whether to reject redirections from the target URL
|
|
161
|
+
standard: 'also', // or 'only' or 'no' (whether to report a standard result)
|
|
162
|
+
observe: false, // Whether to send progress notices to requesting hosts
|
|
163
|
+
device: { // Device to emulate
|
|
168
164
|
id: 'iPhone 8',
|
|
169
165
|
windowOptions: {
|
|
170
166
|
reducedMotion: 'no-preference',
|
|
@@ -179,35 +175,35 @@ Here is an example of a job:
|
|
|
179
175
|
defaultBrowserType: 'webkit'
|
|
180
176
|
}
|
|
181
177
|
},
|
|
182
|
-
browserID: 'webkit'
|
|
183
|
-
creationTimeStamp: '241229T0537',
|
|
184
|
-
executionTimeStamp: '250110T1200',
|
|
178
|
+
browserID: 'chromium', // or 'webkit' or 'firefox'
|
|
179
|
+
creationTimeStamp: '241229T0537', // When job was created
|
|
180
|
+
executionTimeStamp: '250110T1200', // When job will be ready to be performed
|
|
185
181
|
target: {
|
|
186
182
|
what: 'Real Estate Management',
|
|
187
|
-
url: 'https://abccorp.com/mgmt/realproperty
|
|
183
|
+
url: 'https://abccorp.com/mgmt/realproperty'
|
|
188
184
|
},
|
|
189
|
-
sources: {
|
|
185
|
+
sources: { // Any data the requester chooses to add
|
|
190
186
|
script: 'ts99',
|
|
191
187
|
batch: 'departments',
|
|
192
188
|
mergeID: '7f',
|
|
193
189
|
requester: 'malavu@abccorp.com'
|
|
194
190
|
},
|
|
195
|
-
acts: [
|
|
191
|
+
acts: [ // Steps in this job
|
|
196
192
|
{
|
|
197
|
-
type: 'test',
|
|
198
|
-
launch: {},
|
|
199
|
-
which: 'axe',
|
|
200
|
-
detailLevel: 2,
|
|
201
|
-
rules: ['landmark-complementary-is-top-level'],
|
|
193
|
+
type: 'test', // Act type (the 'test' type performs tests of a tool)
|
|
194
|
+
launch: {}, // Act-specific overrides for the browserID and/or target
|
|
195
|
+
which: 'axe', // ID of the tool
|
|
196
|
+
detailLevel: 2, // An argument required by this tool
|
|
197
|
+
rules: ['landmark-complementary-is-top-level'], // Which rules of the tool to test for
|
|
202
198
|
what: 'Axe'
|
|
203
199
|
},
|
|
204
200
|
{
|
|
205
201
|
type: 'test',
|
|
206
202
|
launch: {
|
|
207
|
-
browserID: '
|
|
208
|
-
target: {
|
|
209
|
-
what: 'Real Estate Management',
|
|
210
|
-
url: 'https://abccorp.com/mgmt/realproperty
|
|
203
|
+
browserID: 'webkit', // For this act, use Webkit instead of Chromium
|
|
204
|
+
target: { // For this act, test the contact page instead of the home page
|
|
205
|
+
what: 'Real Estate Management contact',
|
|
206
|
+
url: 'https://abccorp.com/mgmt/realproperty/contactus'
|
|
211
207
|
}
|
|
212
208
|
},
|
|
213
209
|
which: 'qualWeb',
|
|
@@ -219,229 +215,186 @@ Here is an example of a job:
|
|
|
219
215
|
}
|
|
220
216
|
```
|
|
221
217
|
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
Each act includes a `launch` object property. In the first act it is an empty object, so the browser ID and target URL specified by the job are used. In the second act it overrides the job values with per-act values.
|
|
218
|
+
The `device` property lets you choose among [about 125 devices recognized by Playwright](https://github.com/microsoft/playwright/blob/main/packages/playwright-core/src/server/deviceDescriptorsSource.json).
|
|
225
219
|
|
|
226
|
-
|
|
220
|
+
There are 18 act types. They and their options are documented in the `etc` property of the [actSpecs.js](actSpecs.js) object. Documentation for the `actSpecs.js` file is located in the `actSpecs-doc.md` file.
|
|
227
221
|
|
|
228
|
-
|
|
229
|
-
- `what`: a description of the job.
|
|
230
|
-
- `strict`: `true` or `false`, indicating whether _substantive redirections_ should be treated as failures. These are redirections that do more than add or subtract a final slash.
|
|
231
|
-
- `standard`: whether standardized versions of tool reports are to accompany the original versions (`'also'`), replace the original versions (`'only'`), or not be produced (`'no'`).
|
|
232
|
-
- `observe`: `true` or `false`, indicating whether tool and Testaro-rule invocations are to be reported to the server as they occur, so that the server can update a waiting client.
|
|
233
|
-
- `device`: the ID of a device and the properties of each new browser context (window) that will be set for conformity to that device, unless overridden by an act. It must be `'default'` or the ID of one of [about 125 devices recognized by Playwright](https://github.com/microsoft/playwright/blob/main/packages/playwright-core/src/server/deviceDescriptorsSource.json).
|
|
234
|
-
- `browserID`: the ID of the browser to be used, unless overridden by an act. It must be `'chromium'`, `'firefox'`, or `'webkit`'.
|
|
235
|
-
- `creationTimeStamp`: a string in `yymmddThhMM` format, describing when the job was created.
|
|
236
|
-
- `executionTimeStamp`: a string in `yymmddThhMM` format, specifying a date and time before which the job is not to be performed.
|
|
237
|
-
- `target`: data about the target of the job, or `{}` if the job involves multiple targets.
|
|
238
|
-
- `sources`: data optionally inserted into the job by the job creator for use by the job creator.
|
|
239
|
-
- `acts`: an array of the acts to be performed (documented below).
|
|
222
|
+
### Running jobs
|
|
240
223
|
|
|
241
|
-
|
|
224
|
+
#### Immediate job execution
|
|
242
225
|
|
|
243
|
-
|
|
226
|
+
An application can execute a job with::
|
|
244
227
|
|
|
245
|
-
|
|
228
|
+
```javascript
|
|
229
|
+
const {doJob} = require('testaro/run');
|
|
230
|
+
doJob(job)
|
|
231
|
+
.then(report => {
|
|
232
|
+
// Process the report.
|
|
233
|
+
});
|
|
234
|
+
```
|
|
246
235
|
|
|
247
|
-
|
|
236
|
+
A user can make Testaro execute a job with a command like either of:
|
|
248
237
|
|
|
249
|
-
|
|
238
|
+
```bash
|
|
239
|
+
node call run
|
|
240
|
+
node call run 250725T
|
|
241
|
+
```
|
|
250
242
|
|
|
251
|
-
|
|
243
|
+
#### Directory-watched job execution
|
|
252
244
|
|
|
253
|
-
|
|
254
|
-
- _moves_ (clicks, text inputs, hovers, etc.)
|
|
255
|
-
- _alterations_ (changes to the page)
|
|
256
|
-
- _tests_ (one or more of the tests defined by a tool)
|
|
257
|
-
- _branching_ (continuing from an act other than the next one)
|
|
245
|
+
An application can watch a directory for jobs with::
|
|
258
246
|
|
|
259
|
-
|
|
247
|
+
```javaScript
|
|
248
|
+
const {dirWatch} = require('testaro/dirWatch');
|
|
249
|
+
dirWatch(true, 300);
|
|
250
|
+
```
|
|
260
251
|
|
|
261
|
-
|
|
252
|
+
A user can make Testaro watch a directory for jobs with::
|
|
262
253
|
|
|
263
|
-
```
|
|
264
|
-
|
|
265
|
-
"type": "wait",
|
|
266
|
-
"which": "travel",
|
|
267
|
-
"what": "title"
|
|
268
|
-
}
|
|
254
|
+
```javaScript
|
|
255
|
+
node call dirWatch true 300
|
|
269
256
|
```
|
|
270
257
|
|
|
271
|
-
In
|
|
258
|
+
In both cases, the first argument of `dirWatch` tells Testaro whether to continue watching after performing one job, and the second argument tells Testaro how many secods to wait after not finding a job to perform.
|
|
272
259
|
|
|
273
|
-
|
|
260
|
+
Except for the first (`doJob`) case, Testaro finds a file containing a job in the `todo` subdirectory of the `process.env.JOBDIR` directory and saves the report of that job in the `done/raw` subdirectory. In the `node call run` case, the job selected will be the first one whose file name begins with the argument of `run`, or the first one if that argument is absent.
|
|
274
261
|
|
|
275
|
-
####
|
|
262
|
+
#### Network-watched job execution
|
|
276
263
|
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
```json
|
|
280
|
-
{
|
|
281
|
-
"type": "radio",
|
|
282
|
-
"which": "No",
|
|
283
|
-
"index": 2,
|
|
284
|
-
"what": "No, I am not a smoker"
|
|
285
|
-
}
|
|
286
|
-
```
|
|
264
|
+
Testaro can poll servers for jobs to be performed. Such a server can act as the “controller” described in [How to run a thousand accessibility tests](https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3). The server is responsible for preparing Testaro jobs, assigning them to Testaro agents, receiving reports back from those agents, and performing any further processing of the reports, including enhancement, storage, and disclosure to audiences. It can be any server reachable with a URL. That includes a server running on the same host as Testaro, with a URL such as `localhost:3000`.
|
|
287
265
|
|
|
288
|
-
|
|
266
|
+
To make Testaro poll a server for jobs, define the following environment variables, replacing `0` wih any integer:
|
|
289
267
|
|
|
290
|
-
|
|
268
|
+
- `NETWATCH_URL_0_JOB`: which URL to poll (the URL must contain the value of the `AGENT` environment variable)
|
|
269
|
+
- `NETWATCH_URL_0_OBSERVE`: which URL to send progress messages to, if requested, during job execution
|
|
270
|
+
- `NETWATCH_URL_0_REPORT`: which URL to send job reports to
|
|
271
|
+
- `NETWATCH_URL_0_AUTH`: the password to supply to the server when polling and when delivering a report
|
|
272
|
+
- `NETWATCH_URLS`: the integer IDs of the servers to be polled, comma-delimited (e.g., `0,3,4`)
|
|
291
273
|
|
|
292
|
-
|
|
274
|
+
The job request sent to the server can be a `POST` request, in which the `agentPW` property of the payload will be the password. Or it can be a `GET` request with the URL containing the password.
|
|
293
275
|
|
|
294
|
-
|
|
276
|
+
Testaro will send the report as a `POST` request whose payload is a JSON object with two properties: `agentPW` (the password) and `report` (the report). However, if the environment does not contain a password, the payload is a JSON object containing only the report.
|
|
295
277
|
|
|
296
|
-
An
|
|
278
|
+
An application can watch the network for jobs with:
|
|
297
279
|
|
|
298
|
-
```
|
|
299
|
-
{
|
|
300
|
-
|
|
301
|
-
"what": "make everything visible"
|
|
302
|
-
}
|
|
280
|
+
```javaScript
|
|
281
|
+
const {netWatch} = require('testaro/netWatch');
|
|
282
|
+
netWatch(true, 300, true);
|
|
303
283
|
```
|
|
304
284
|
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
#### Branching
|
|
285
|
+
A user can make Testaro watch the network for jobs with:
|
|
308
286
|
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
```json
|
|
312
|
-
{
|
|
313
|
-
"type": "next",
|
|
314
|
-
"if": ["totals.invalid", ">", 0],
|
|
315
|
-
"jump": -4,
|
|
316
|
-
"what": "redo search if any invalid elements"
|
|
317
|
-
}
|
|
287
|
+
```bash
|
|
288
|
+
node call netWatch true 300 true
|
|
318
289
|
```
|
|
319
290
|
|
|
320
|
-
|
|
291
|
+
The first argument of `netWatch` tells Testaro whether to continue watching after performing the first job. The second argument tells Testaro how many seconds to wait after receiving a no-jobs response. The third argument tells Testaro whether to be certificate-tolerant, i.e. to accept SSL certificates that fail verification against a list of certificate authorities (the default is `true`).
|
|
321
292
|
|
|
322
|
-
|
|
293
|
+
## Reports
|
|
323
294
|
|
|
324
|
-
|
|
295
|
+
A report is a job with information about the results of the performance of the job inserted by Testaro into the job.
|
|
325
296
|
|
|
326
|
-
|
|
297
|
+
### Whole-job insertions
|
|
327
298
|
|
|
328
|
-
|
|
299
|
+
As Testaro performs a job, information about the job as a whole is inserted into the job. That information is organized into one or two properties:
|
|
329
300
|
|
|
330
|
-
|
|
301
|
+
- `jobData`: Miscellaneous facts about the completed job
|
|
302
|
+
- `catalog`: A collection of data about the elements on the target that failed any test(s)
|
|
331
303
|
|
|
332
|
-
|
|
304
|
+
Testaro inserts the `jobData` property into every job, but inserts the `catalog` property only into jobs that instruct Testaro to produce standard results.
|
|
333
305
|
|
|
334
|
-
|
|
306
|
+
#### Catalog
|
|
335
307
|
|
|
336
|
-
|
|
337
|
-
- a `.js` file, defining the operation of the tool, in the `tests` directory, whose name base is the name of the tool
|
|
308
|
+
Whenever a job requires any testing and requires the production of standard results, Testaro inserts a _catalog_ into the report before calling any of the testing tools. The catalog is an inventory of HTML elements in the DOM of the target.
|
|
338
309
|
|
|
339
|
-
The `
|
|
310
|
+
The `catalog` property has an object value. Here is an example:
|
|
340
311
|
|
|
341
312
|
```javascript
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
313
|
+
'123': {
|
|
314
|
+
tagName: 'SUMMARY',
|
|
315
|
+
id: 'functionsummary',
|
|
316
|
+
startTag: '<summary>',
|
|
317
|
+
text: 'Functional\nAlways active',
|
|
318
|
+
textLinkable: true,
|
|
319
|
+
boxID: '46:230:860:263',
|
|
320
|
+
pathID: '/html/body/div[1]/div[1]/div[3]/div[2]/details[1]/summary[1]'
|
|
321
|
+
},
|
|
351
322
|
```
|
|
352
323
|
|
|
353
|
-
|
|
324
|
+
If the inner text of the element consists of only one line, that is the value of `text`. If the inner text consists of two or more lines, `text` is the first and last of these, delimited with a newline.
|
|
354
325
|
|
|
355
|
-
|
|
326
|
+
The `textLinkable` property has a true value whenever `text` is non-empty and can generate a text-fragment URL that uniquely identifies an element.
|
|
356
327
|
|
|
357
|
-
|
|
328
|
+
The segments of `boxID` are `x`, `y`, `width`, and `height`.
|
|
358
329
|
|
|
359
|
-
The
|
|
330
|
+
The catalog is a mechanism for the integration of the tools. Most rule violations that tools report are blamed on particular HTML elements. A tool typically reports that an element violated a rule by having some defect in its configuration or behavior. But tools describe elements differently. Testaro makes the tools identify the XPaths of the elements they report as violators. Testaro then finds, for each XPath, the correct catalog entry.
|
|
360
331
|
|
|
361
|
-
|
|
332
|
+
Testaro uses the following techniques to make the tools calculate XPaths:
|
|
362
333
|
|
|
363
|
-
|
|
334
|
+
- `alfa` and `aslint`: They report XPaths, so Testaro needs only to normalize them.
|
|
335
|
+
- `ed11y`: Testaro adds it and a `window.getXPath` method to the page; when the tool reports an element, Testaro computes its XPath.
|
|
336
|
+
- `wave`: It reports a selector for each element; Testaro finds each element in the page via its selector and executes `window.getXPath` on the element.
|
|
337
|
+
- `axe`, `htmlcs`, `ibm`, `nuVal`, `nuVnu`, `qualWeb`, `wax`: Testaro adds `data-xpath` attributes to all elements; the tools include code excerpts, with the `data-expath` attributes, in the reported violations.
|
|
338
|
+
- `testaro`: Testaro designs each of its own tests to report element XPaths.
|
|
364
339
|
|
|
365
|
-
|
|
366
|
-
{
|
|
367
|
-
"type": "test",
|
|
368
|
-
"which": "wave",
|
|
369
|
-
"reportType": 1,
|
|
370
|
-
"what": "WAVE summary"
|
|
371
|
-
}
|
|
372
|
-
```
|
|
340
|
+
By attaching a catalog entry to each reported element, Testaro allows an application that uses Testaro to tell users, for any particular HTML element, which tools ascribed violations of which rules to that element. An application could, for example, use a screenshot or a text-fragment link or could ask the user to paste the XPath into a browser developer tool.
|
|
373
341
|
|
|
374
|
-
|
|
342
|
+
In some cases no catalog entry can be found. The reasons may include:
|
|
375
343
|
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
which: 'alfa',
|
|
380
|
-
what: 'Siteimprove alfa tool',
|
|
381
|
-
rules: ['y', 'r25', 'r71']
|
|
382
|
-
}
|
|
383
|
-
```
|
|
344
|
+
- The element was dynamically created after the catalog was created.
|
|
345
|
+
- The element is inside a `noscript` element and therefore not considered an element in the DOM.
|
|
346
|
+
- The violation is not ascribed to a single element.
|
|
384
347
|
|
|
385
|
-
|
|
348
|
+
### Act insertions
|
|
386
349
|
|
|
387
|
-
|
|
350
|
+
As Testaro performs the acts of a job, information about the results of each act is inserted into that act. For acts of type `test`, the added properties are:
|
|
388
351
|
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
what: 'Testaro tool',
|
|
394
|
-
rules: ['y', 'hover', 'focInd'],
|
|
395
|
-
args: {
|
|
396
|
-
hover: [20],
|
|
397
|
-
focInd: [false, 300]
|
|
398
|
-
}
|
|
399
|
-
}
|
|
400
|
-
```
|
|
352
|
+
- `startTime`: When Testaro began to perform the act
|
|
353
|
+
- `actualURL`: The tested URL (different from the target URL if the request was redirected)
|
|
354
|
+
- `data`: Data generated by the tool
|
|
355
|
+
- `result`: Results of the testing by the tool
|
|
401
356
|
|
|
402
|
-
|
|
357
|
+
The `result` property is an object with one or two (depending on the value of `standard`, as described above) subproperties:
|
|
403
358
|
|
|
404
|
-
|
|
359
|
+
- `nativeResult`: The result (or a compact version of the result) natively produced by the tool
|
|
360
|
+
- `standardResult`: A Testaro-standardized version of the result
|
|
405
361
|
|
|
406
|
-
|
|
362
|
+
If an act of type `test` contains an `expect` property (specifying expectations about the result), then Testaro also inserts these properties into the act:
|
|
407
363
|
|
|
408
|
-
|
|
364
|
+
- `expectations`: Data on what was expected versus the actual result
|
|
365
|
+
- `expectationFailures`: The count of failed expectations
|
|
409
366
|
|
|
410
|
-
|
|
411
|
-
'expect': [
|
|
412
|
-
['standardResult.totals.0', '=', 0],
|
|
413
|
-
['standardResult.instances.length', '=', 0]
|
|
414
|
-
]
|
|
415
|
-
```
|
|
367
|
+
Details about these expectation properties are documened in the `VALIDATION.md` file.
|
|
416
368
|
|
|
417
|
-
|
|
369
|
+
#### Standard results
|
|
418
370
|
|
|
419
|
-
|
|
371
|
+
If the job instructs Testaro to include standard results, then the `result.standardResult` property of each act of type `test` will have three properties:
|
|
420
372
|
|
|
421
|
-
|
|
373
|
+
- `prevented`: Whether the tool was prevented from performing the act
|
|
374
|
+
- `totals`: An array of 4 integers, counting the rule violations at 4 severity levels
|
|
375
|
+
- `instances`: An array of data about the violations reported by the tool
|
|
422
376
|
|
|
423
|
-
|
|
377
|
+
More specifically:
|
|
424
378
|
|
|
425
|
-
-
|
|
426
|
-
-
|
|
427
|
-
-
|
|
428
|
-
-
|
|
429
|
-
- `
|
|
430
|
-
- `
|
|
379
|
+
- The `totals` value is an array like this: `[3, 0, 87, 4]`. This example would mean that the tool reported 3 failures at severity 0 (the least severe level), none at severity 1, 87 at severity 2, and 4 at severity 3. These four severities are conceptually ordinal, not metric.
|
|
380
|
+
- The `instances` value is an array of objects, each having these properties:
|
|
381
|
+
- `ruleId`: The ID of the rule that was violated
|
|
382
|
+
- `what`: A description of the rule or of the violation
|
|
383
|
+
- `ordinalSeverity`: The severity of the violation
|
|
384
|
+
- `count`: How many violations of the rule this instance reports
|
|
385
|
+
- `catalogIndex`: Key of the HTML element in the catalog
|
|
431
386
|
|
|
432
|
-
|
|
387
|
+
If no catalog entry was found for the instance, then instead of a `catalogIndex` property Testaro tries to insert a `pathID` property, whose value is a normalized XPath of the offending HTML element.
|
|
433
388
|
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
### Tool details
|
|
389
|
+
## Tool details
|
|
437
390
|
|
|
438
391
|
The tools whose tests Testaro performs have particularities described below.
|
|
439
392
|
|
|
440
|
-
|
|
393
|
+
### ASLint
|
|
441
394
|
|
|
442
395
|
The `aslint` tool makes use of the [`aslint-testaro` fork](https://www.npmjs.com/package/aslint-testaro) of the [`aslint` repository](https://github.com/essentialaccessibility/aslint), which, unlike the published `aslint` package, contains the `aslint.bundle.js` file.
|
|
443
396
|
|
|
444
|
-
|
|
397
|
+
### HTML CodeSniffer
|
|
445
398
|
|
|
446
399
|
The `htmlcs` tool makes use of the `htmlcs/HTMLCS.js` file. That file was created, and can be recreated if necessary, as follows:
|
|
447
400
|
|
|
@@ -473,7 +426,7 @@ The changes in `htmlcs/HTMLCS.js` are:
|
|
|
473
426
|
> );
|
|
474
427
|
```
|
|
475
428
|
|
|
476
|
-
|
|
429
|
+
### Accessibility Checker
|
|
477
430
|
|
|
478
431
|
The `ibm` tests require the `aceconfig.js` file.
|
|
479
432
|
|
|
@@ -500,13 +453,13 @@ The `ibm` tool is one of two tools (`testaro` is the other) with a `withItems` p
|
|
|
500
453
|
|
|
501
454
|
In a previous version of the package, the tool operated on the page content when the `withNewContent` property was `false`. In some cases the tool threw untrappable errors for some targets under that condition. The tool launched a Puppeteer browser to create pages to perform its tests on. On any host that did not permit sandboxed browsers to be launched, the `aceconfig.js` file needed to specify nonsandboxed browsers. Starting in December 2025, the tool operates on the page rather than the page content.
|
|
502
455
|
|
|
503
|
-
|
|
456
|
+
### Nu Html Checker
|
|
504
457
|
|
|
505
458
|
The `nuVal` and `nuVnu` tools perform the tests of the Nu Html Checker.
|
|
506
459
|
|
|
507
460
|
Its `rules` argument is **not** an array of rule IDs, but instead is an array of rule _specifications_. A rule specification for `nuVal` or `nuVnu` is a string with the format `=ruleID` or `~ruleID`. The `=` prefix indicates that the rule ID is invariable. The `~` prefix indicates that the rule ID is variable, in which case the `ruleID` part of the specification is a matching regular expression, rather than the exact text of a message. This `rules` format arises from the fact that `nuVal` and `nuVnu` generate customized messages and do not accompany them with rule identifiers.
|
|
508
461
|
|
|
509
|
-
|
|
462
|
+
### QualWeb
|
|
510
463
|
|
|
511
464
|
The `qualWeb` tool performs the ACT rules, WCAG Techniques, and best-practices tests of QualWeb. Only failures and warnings are included in the report. The EARL report of QualWeb is not generated, because it is equivalent to the report of the ACT rules tests.
|
|
512
465
|
|
|
@@ -533,7 +486,7 @@ The target can be provided to QualWeb either as HTML or as a URL. Experience ind
|
|
|
533
486
|
|
|
534
487
|
QualWeb creates sandboxed Puppeteer pages to perform its tests on. Therefore, the host must permit sandboxed browsers to be launched. See the pertinent [Kilotest documentation](https://github.com/jrpool/kilotest/blob/main/SERVICE.md#browser-privileges) for information about the configuration of an Ubuntu Linux host for this purpose.
|
|
535
488
|
|
|
536
|
-
|
|
489
|
+
### Testaro
|
|
537
490
|
|
|
538
491
|
The rules that Testaro can test for are implemented in files within the `testaro` directory.
|
|
539
492
|
|
|
@@ -541,14 +494,12 @@ The Testaro rules are classified by an `allRules` array defined in the `tests/te
|
|
|
541
494
|
|
|
542
495
|
- `id`: the rule ID.
|
|
543
496
|
- `what`: a description of the rule.
|
|
544
|
-
- `
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
- `defaultOn`: whether the rule is to be tested for by default.
|
|
549
|
-
- `timeOut`: the maximum time in seconds allowed for a test for the rule.
|
|
497
|
+
- `contaminates`: whether the test for the rule modifies the page, requiring the next test to launch a new browser for test isolation
|
|
498
|
+
- `needsAccessibleName`: whether the rule requires an added script adding an accessible-name computation method to `window`
|
|
499
|
+
- `timeOut`: the maximum time in seconds allowed for a test of the rule
|
|
500
|
+
- `defaultOn`: whether the rule is to be tested for by default
|
|
550
501
|
|
|
551
|
-
If you do not specify rules when using the `testaro` tool, Testaro will test for its default rules
|
|
502
|
+
If you do not specify rules when using the `testaro` tool, Testaro will test for its default rules, in the order in which they appear in the array.
|
|
552
503
|
|
|
553
504
|
The optional `rules` argument for a `testaro` test act is an array whose first item is either `'y'` or `'n'` and whose remaining items are rule IDs. If `'y'`, then only the specified rules’ tests are performed. If `'n'`, then all the default rules are tested for, **except** for the specified rules.
|
|
554
505
|
|
|
@@ -556,17 +507,15 @@ The `testaro` tool (like the `ibm` tool) has a `withItems` property. If you set
|
|
|
556
507
|
|
|
557
508
|
Unlike any other tool, the `testaro` tool requires a `stopOnFail` property, which specifies whether a failure to conform to any rule (i.e. any value of `totals` other than `[0, 0, 0, 0]`) should terminate the execution of tests for the remaining rules.
|
|
558
509
|
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
A new pattern for rule definition was introduced in version 60.7.0 and is implemented for only some of the applicable Testaro rules. In this pattern, the `launch` function in the `run` module adds a script to the page that runs whenever a new page is added to a browser context. That script adds `window` methods to the page. When the browser is launched for a Testaro test, the added `window` methods include a `getXPath` method and a `getInstance` method. These methods are used in rule definitions. For examples of this pattern, see the `adbID` and `lineHeight` rules.
|
|
510
|
+
Tests of the `testaro` tests (i.e. _validation_) can be performed as documented in the `VALIDATION.md` file.
|
|
562
511
|
|
|
563
|
-
|
|
512
|
+
### WallyAX
|
|
564
513
|
|
|
565
514
|
If a `wax` test act is included in the job, an environment variable named `WAX_KEY` must exist, with your WallyAX API key as its value. You can obtain it from [WallyAX](https://account.wallyax.com/?ref_app=Developer&app_type=npm).
|
|
566
515
|
|
|
567
516
|
The `wax` tool imposes a limit on the size of a page to be tested. If the page exceeds the limit, Testaro treats the page as preventing `wax` from performing its tests. The limit is less than 500,000 characters.
|
|
568
517
|
|
|
569
|
-
|
|
518
|
+
### WAVE
|
|
570
519
|
|
|
571
520
|
If a `wave` test act is included in the job, the WAVE tests will be performed either by the subscription API or by the stand-alone API.
|
|
572
521
|
|
|
@@ -574,401 +523,15 @@ If you want the subscription API to perform the tests, you must get a WAVE API k
|
|
|
574
523
|
|
|
575
524
|
If you want the stand-alone API to perform the tests, you need to have that API installed and running, and the `wave` test act needs to define the URL of your stand-alone API. The test act can also define a `prescript` script and/or a `postscript` script.
|
|
576
525
|
|
|
577
|
-
### Browser types
|
|
578
|
-
|
|
579
|
-
When you want to run some tests of a tool with one browser type and other tests of the same tool with another browser type, you can do so by splitting the rules into two test acts. For example, one test act can specify the rules as
|
|
580
|
-
|
|
581
|
-
```javascript
|
|
582
|
-
['y', 'r15', 'r54']
|
|
583
|
-
```
|
|
584
|
-
|
|
585
|
-
and the other test act can specify the rules as
|
|
586
|
-
|
|
587
|
-
```javascript
|
|
588
|
-
['n', 'r15', 'r54']
|
|
589
|
-
```
|
|
590
|
-
|
|
591
|
-
Together, they get all tests of the tool performed. Before each test act, you can ensure that the latest `launch` act has specified the browser type to be used in that test act.
|
|
592
|
-
|
|
593
|
-
### `actSpecs` file
|
|
594
|
-
|
|
595
|
-
#### Introduction to the `actSpecs` file
|
|
596
|
-
|
|
597
|
-
The `actSpecs.js` file contains rules governing acts. The rules determine whether an act is valid.
|
|
598
|
-
|
|
599
|
-
#### Rule format
|
|
600
|
-
|
|
601
|
-
The rules in `actSpecs.js` are organized into two objects, `etc` and `tests`. The `etc` object contains rules for acts of all types. The `tools` object contains additional rules that apply to some acts of type `test`, depending on the values of their `which` properties, namely which tools they perform tests of.
|
|
602
|
-
|
|
603
|
-
Here is an example of an act:
|
|
604
|
-
|
|
605
|
-
```json
|
|
606
|
-
{
|
|
607
|
-
"type": "link",
|
|
608
|
-
"which": "warming",
|
|
609
|
-
"what": "article on climate change"
|
|
610
|
-
}
|
|
611
|
-
```
|
|
612
|
-
|
|
613
|
-
And here is the applicable property of the `etc` object in `actSpecs.js`:
|
|
614
|
-
|
|
615
|
-
```js
|
|
616
|
-
link: [
|
|
617
|
-
'Click a link',
|
|
618
|
-
{
|
|
619
|
-
which: [true, 'string', 'hasLength', 'substring of the link text'],
|
|
620
|
-
what: [false, 'string', 'hasLength', 'comment']
|
|
621
|
-
}
|
|
622
|
-
]
|
|
623
|
-
```
|
|
624
|
-
|
|
625
|
-
The rule is an array with two elements: a string ('Click a link') describing the act and an object containing requirements for any act of type `link`.
|
|
626
|
-
|
|
627
|
-
The requirement `which: [true, 'string', 'hasLength', 'substring of the link text']` specifies what is required for the `which` property of a `link`-type act. The requirement is an array.
|
|
628
|
-
|
|
629
|
-
In most cases, the array has length 4:
|
|
630
|
-
|
|
631
|
-
- Item 0. Is the property (here `which`) required (`true` or `false`)? The value `true` here means that every `link`-type act **must** contain a `which` property.
|
|
632
|
-
- Item 1. What format must the property value have (`'string'`, `'array'`, `'boolean'`, `'number'`, or `'object'`)?
|
|
633
|
-
- Item 2. What other validity criterion applies (if any)? (Empty string if none.) The `hasLength` criterion means that the string must be at least 1 character long.
|
|
634
|
-
- Item 3. Description of the property. In this example, the description says that the value of `which` must be a substring of the text content of the link that is to be clicked. Thus, a `link` act tells Testaro to find the first link whose text content has this substring and click it.
|
|
635
|
-
|
|
636
|
-
The validity criterion named in item 2 may be any of these:
|
|
637
|
-
|
|
638
|
-
- `'hasLength'`: is not a blank string
|
|
639
|
-
- `'isURL`': is a string starting with `http`, `https`, or `file`, then `://`, then ending with 1 or more non-whitespace characters
|
|
640
|
-
- `'isBrowserType'`: is `'chromium'`, `'firefox'`, or `'webkit'`
|
|
641
|
-
- `'isFocusable'`: is `'a'`, `'button'`, `'input'`, `'select'`, or `'option'`
|
|
642
|
-
- `'isState'`: is `'loaded'` or `'idle'`
|
|
643
|
-
- `'isTest'`: is the name of a tool
|
|
644
|
-
- `'isWaitable'`: is `'url'`, `'title'`, or `'body'`
|
|
645
|
-
- `'areStrings'`: is an array of strings
|
|
646
|
-
|
|
647
|
-
## Reports
|
|
648
|
-
|
|
649
|
-
Any Testaro job produces a report, which is a copy of the job with additional data produced by Testaro as it performed the job. Like a job, a report is an object that can be serialized to JSON for file storage and network transmission.
|
|
650
|
-
|
|
651
|
-
### Job-level data
|
|
652
|
-
|
|
653
|
-
The data that Testaro adds to a job to create a report include job-level data: data describing the how the job as a whole was performed. Examples: when it was completed and how long it took to run.
|
|
654
|
-
|
|
655
|
-
Properties that were in a job when it was given to Testaro remain unchanged in the report. New data produced by Testaro during its performance of a job are inserted into a new `jobData` property in the report.
|
|
656
|
-
|
|
657
|
-
### Act-level data
|
|
658
|
-
|
|
659
|
-
Testaro also adds act-level data to each job. The new act-level data are properties added to each `act` object.
|
|
660
|
-
|
|
661
|
-
#### Act-level data from `test` acts
|
|
662
|
-
|
|
663
|
-
In a `test` act, one of the 11 tools performs tests and reports the results. Testaro manages this performance with the `reporter` function of a file located in the `tests` directory. Each tool has a corresponding file, such as `alfa.js` for the `alfa` tool.
|
|
664
|
-
|
|
665
|
-
The `reporter` function returns an object with this structure:
|
|
666
|
-
|
|
667
|
-
```js
|
|
668
|
-
{
|
|
669
|
-
data: {
|
|
670
|
-
prevented: boolean (whether the tool failed to perform its tests on the page),
|
|
671
|
-
error: string (if `prevented` is `true`, a description of the error)
|
|
672
|
-
… (other tool-specific data)
|
|
673
|
-
},
|
|
674
|
-
result: object (the results of the tests performed by the tool)
|
|
675
|
-
}
|
|
676
|
-
```
|
|
677
|
-
|
|
678
|
-
On the completion of a job, Testaro has added these properties to each `test` act to produce a report:
|
|
679
|
-
|
|
680
|
-
- `what` (string): the name of the tool
|
|
681
|
-
- `actualURL` (string): the URL of the visited page, after any redirections
|
|
682
|
-
- `startTime` (string): when the tool was started
|
|
683
|
-
- `endTime` (string): when the tool reported its results
|
|
684
|
-
- `data` (object): other tool-specific data:
|
|
685
|
-
- `prevented` (boolean): whether the tool failed to perform its tests
|
|
686
|
-
- `error` (string): a description of the failure, if any
|
|
687
|
-
- other tool-specific data, if any
|
|
688
|
-
|
|
689
|
-
Testaro may also add these properties to any `test` act:
|
|
690
|
-
|
|
691
|
-
- `expectations` (object): the results of validations specified by the act in `expect` properties
|
|
692
|
-
- `expectationFailures` (number): the count of failed validations
|
|
693
|
-
|
|
694
|
-
The value of `expectations` is an array of objects, one object per expectation. Each object includes a `property` property identifying the expectation, and a `passed` property with `true` or `false` value reporting whether the expectation was satisfied. If applicable, it also has other properties identifying what was expected and what was actually reported.
|
|
695
|
-
|
|
696
|
-
Testaro also adds one or both of these properties to each `test` act:
|
|
697
|
-
|
|
698
|
-
- `result` (object): the results of the tests performed by the tool, in the native format of the tool
|
|
699
|
-
- `standardResult` (object): the `result` property converted to a Testaro-standard structure
|
|
700
|
-
|
|
701
|
-
A job specifies whether the report should include, for each `test` act, the `result` property, the `standardResult` property, or both.
|
|
702
|
-
|
|
703
|
-
#### Act-level data from `testaro` test acts
|
|
704
|
-
|
|
705
|
-
Each Testaro rule module exports a `reporter` function, which returns an object with `data`, `totals`, and `standardInstances` properties. Testaro combines the values of those properties with the corresponding values of the same properties from the other `testaro` tests to create the `data` and `result` properties added to `testaro` test acts.
|
|
706
|
-
|
|
707
|
-
#### Standard result
|
|
708
|
-
|
|
709
|
-
##### Properties
|
|
710
|
-
|
|
711
|
-
The `standardResult` property, when added to a `test` act, includes three properties:
|
|
712
|
-
|
|
713
|
-
- `prevented` (boolean): whether the tool failed to perform its tests on the page.
|
|
714
|
-
- `totals` (array of numbers): counts of rule violations at 4 ordinal severity levels. For example, the array `[3, 0, 14, 10]` reports that there were 3 violations at level 0, 0 at level 1, 14 at level 2, and 10 at level 3.
|
|
715
|
-
- `instances` (array of objects): descriptions of rule violations reported. An instance can describe a single violation, usually by one element in the page, or can summarize multiple violations of the same rule.
|
|
716
|
-
|
|
717
|
-
If the value of `prevented` is `true`, the standard result also includes an `error` property describing the reason for the failure.
|
|
718
|
-
|
|
719
|
-
##### Instances
|
|
720
|
-
|
|
721
|
-
Here is an example of an instance in a standard result:
|
|
722
|
-
|
|
723
|
-
```javascript
|
|
724
|
-
{
|
|
725
|
-
ruleID: 'rule01',
|
|
726
|
-
what: 'Button type invalid',
|
|
727
|
-
ordinalSeverity: 2,
|
|
728
|
-
count: 1,
|
|
729
|
-
tagName: 'BUTTON'
|
|
730
|
-
id: '',
|
|
731
|
-
location: {
|
|
732
|
-
doc: 'dom',
|
|
733
|
-
type: 'xpath',
|
|
734
|
-
spec: '/html[1]/body[1]/section[3]/div[2]/div[1]/ul[1]/li[1]/button[1]'
|
|
735
|
-
},
|
|
736
|
-
excerpt: '<button type="link">Create an account</button>',
|
|
737
|
-
text: 'Create an account'
|
|
738
|
-
boxID: '12:340:46:50',
|
|
739
|
-
pathID: '/html/body/section[3]/div[2]/div/ul/li[1]/button[1]',
|
|
740
|
-
}
|
|
741
|
-
```
|
|
742
|
-
|
|
743
|
-
This instance says that a `button` element violates a rule named `rule01`.
|
|
744
|
-
|
|
745
|
-
The element has no `id` attribute to distinguish it from other `button` elements, but the tool describes its location. This tool uses an XPath to do that. Tools use various methods for location description, namely:
|
|
746
|
-
|
|
747
|
-
- `code` (line, starting column, and ending column): Nu Html Checker (API and installed)
|
|
748
|
-
- `selector` (CSS selector): Axe, QualWeb, WAVE
|
|
749
|
-
- `xpath`: Alfa, ASLint, Equal Access
|
|
750
|
-
- `box` (coordinates, width, and height of the element box): Editoria11y, Testaro
|
|
751
|
-
- none: HTML CodeSniffer
|
|
752
|
-
|
|
753
|
-
The tool or Testaro also reproduces an excerpt of the element code.
|
|
754
|
-
|
|
755
|
-
##### Element identification
|
|
756
|
-
|
|
757
|
-
While the above properties can help you find the offending element, Testaro makes this easier by adding, where practical, three standard element identifiers to each standard instance:
|
|
758
|
-
|
|
759
|
-
- `boxID`: a compact representation of the x, y, width, and height of the element bounding box, if the element can be identified and is visible.
|
|
760
|
-
- `pathID`: the XPath of the element, if the element can be identified.
|
|
761
|
-
- `text`: the text content of the element, if the element can be identified.
|
|
762
|
-
|
|
763
|
-
These standard identifiers can help you determine whether violations reported by different tools belong to the same element or different elements. The `boxID` property can also support the making of images of the violating elements.
|
|
764
|
-
|
|
765
|
-
Some tools limit the efficacy of the current algorithm for standard identifiers:
|
|
766
|
-
|
|
767
|
-
- HTML CodeSniffer does not report element locations, and the reported code excerpts exclude all text content.
|
|
768
|
-
- Nu Html Checker reports line and column boundaries of element start tags and truncates element text content in reported code excerpts.
|
|
769
|
-
|
|
770
|
-
Testaro aims to overcome these limitations by inserting uniquely identifying attributes into all elements of the pages being tested by these tools. Those attribute values permit Testaro to identify the elements in the tested page. Except for elements excluded from the DOM, such as descendants of `noscript` elements, this mechanism allows Testaro to provide a `pathID` property in almost all standard instances. The `boxID` property is less universal, since some elements, such as `script` elements and hidden elements, have no bounding boxes.
|
|
771
|
-
|
|
772
|
-
Testing can change the pages being tested, and such changes can cause a particular element to change its physical or logical location. In such cases, an element may appear multiple times in a tool report with different `boxID` or `pathID` values, even though it is, for practical purposes, the same element.
|
|
773
|
-
|
|
774
|
-
##### Standardization configuration
|
|
775
|
-
|
|
776
|
-
Each job specifies how Testaro is to handle report standardization. A job contains a `standard` property, with one of the following values to determine which results the report will include:
|
|
777
|
-
|
|
778
|
-
- `'also'`: original and standard.
|
|
779
|
-
- `'only'`: standard only.
|
|
780
|
-
- `'no'`: original only.
|
|
781
|
-
|
|
782
|
-
If a tool has the option to be used without itemization and is being so used, the `instances` array may be empty, or may contain one or more summary instances. Summary instances disclose the numbers of instances that they summarize with the `count` property. They typically summarize violations by multiple elements, in which case their `id`, `location`, `excerpt`, `boxID`, and `pathID` properties will have empty values.
|
|
783
|
-
|
|
784
|
-
##### Standardization opinionation
|
|
785
|
-
|
|
786
|
-
This standard format reflects some judgments. For example:
|
|
787
|
-
|
|
788
|
-
- The `ordinalSeverity` property of an instance involves interpretation. Tools may report severity, certainty, priority, or some combination of those. They may use ordinal or metric quantifications. If they quantify ordinally, their scales may have more or fewer than 4 ranks. Testaro coerces each tool’s severity, certainty, and/or priority classification into a 4-rank ordinal classification. This classification is deemed to express the most common pattern among the tools.
|
|
789
|
-
- The `tagName` property of an instance may not always be obvious, because in some cases the rule being tested for requires a relationship among more than one element (e.g., “An X element may not have a Y element as its parent”).
|
|
790
|
-
- The `ruleID` property of an instance is a matching rule if the tool issues a message but no rule identifier for each instance. The `nuVal` and `nuVnu` tools do this. In this case, Testaro is classifying the messages into rules.
|
|
791
|
-
- The `ruleID` property of an instance may reclassify tool rules. For example, if a tool rule covers multiple situations that are dissimilar, that rule may be split into multiple rules with distinct `ruleID` properties.
|
|
792
|
-
|
|
793
|
-
You are not dependent on the judgments incorporated into the standard format, because Testaro can give you the original reports from the tools as the `result` property of a `test` act.
|
|
794
|
-
|
|
795
|
-
The standard format does not express opinions on issue classification. A rule ID identifies something deemed to be an issue by a tool. Useful reporting from ensemble testing still requires the classification of tool **rules** into **issues**. If tool `A` has `alt-incomplete` as a rule ID and tool `B` has `image_alt_stub` as a rule ID, Testaro does not decide whether those are really the same issue or different issues. That decision belongs to you. The standardization of tool reports by Testaro eliminates some of the drudgery in issue classification, but not any of the judgment required for issue classification.
|
|
796
|
-
|
|
797
|
-
## Invocation
|
|
798
|
-
|
|
799
|
-
Testaro features can be invoked by modules of your application when Testaro is a dependency, or directly by users who have installed Testaro as an application.
|
|
800
|
-
|
|
801
|
-
Before a module can execute a Testaro function, it must import that function from the Testaro module that exports it. A module can import function `f` from module `m` with the statement
|
|
802
|
-
|
|
803
|
-
```javascript
|
|
804
|
-
const {f} = require('testaro/m');`
|
|
805
|
-
```
|
|
806
|
-
|
|
807
|
-
## Immediate job execution
|
|
808
|
-
|
|
809
|
-
A job can be immediately executed as follows:
|
|
810
|
-
|
|
811
|
-
### By a module
|
|
812
|
-
|
|
813
|
-
```javascript
|
|
814
|
-
const {doJob} = require('testaro/run');
|
|
815
|
-
doJob(job)
|
|
816
|
-
.then(report => …);
|
|
817
|
-
```
|
|
818
|
-
|
|
819
|
-
Testaro will run the job and return a `report` object, a copy of the job with the `acts` and `jobData` properties containing the results. The final statement can further process the `report` object as desired in the `then` callback.
|
|
820
|
-
|
|
821
|
-
The Testilo package contains functions that can create jobs from scripts, add scores and explanations to reports, and create HTML documents summarizing reports.
|
|
822
|
-
|
|
823
|
-
### By a user
|
|
824
|
-
|
|
825
|
-
```bash
|
|
826
|
-
node call run
|
|
827
|
-
node call run 250525T
|
|
828
|
-
```
|
|
829
|
-
|
|
830
|
-
In the second example, `250525T` is the initial characters of the ID of a job saved as a JSON file in the `todo` subdirectory of the `JOBDIR` directory (`JOBDIR` refers to the value of the environment variable `JOBDIR`, obtained via `process.env.JOBDIR`).
|
|
831
|
-
|
|
832
|
-
The `call` module will find the first job file with a matching name if an argument is given, or the first job file if not. Then the module will execute the `doJob` function of the `run` module on the job, save the report in the `raw` subdirectory of the `REPORTDIR` directory, and archive the job file in the `done` subdirectory of the `JOBDIR` directory. (The report destination is named `raw` because the report has not yet been further processed by your application, perhaps using Testilo, to convert the report data into user-friendly reports.)
|
|
833
|
-
|
|
834
|
-
## Job watching
|
|
835
|
-
|
|
836
|
-
In watch mode, Testaro periodically checks for a job to run and, when a job is obtained, performs it.
|
|
837
|
-
|
|
838
|
-
### Directory watching
|
|
839
|
-
|
|
840
|
-
Testaro can watch for a job in a directory of the filesystem where Testaro or your application is located, with the `dirWatch` function.
|
|
841
|
-
|
|
842
|
-
#### Directory watching by a module
|
|
843
|
-
|
|
844
|
-
```javaScript
|
|
845
|
-
const {dirWatch} = require('testaro/dirWatch');
|
|
846
|
-
dirWatch(true, 300);
|
|
847
|
-
```
|
|
848
|
-
|
|
849
|
-
In this example, a moduleof your application asks Testaro to check a directory for a job every 300 seconds, to perform the jobs in the directory if any are found, and then to continue checking. If the first argument is `false`, Testaro will stop checking after performing 1 job. If it is `true`, Testaro continues checking until the `dirWatch` process is stopped.
|
|
850
|
-
|
|
851
|
-
Testaro checks for jobs in the `todo` subdirectory of `JOBDIR`. When it has performed a job, Testaro moves it into the `done` subdirectory.
|
|
852
|
-
|
|
853
|
-
Testaro creates a report for each job and saves the report in the `raw` subdirectory of `REPORTDIR`.
|
|
854
|
-
|
|
855
|
-
#### Directory watching by a user
|
|
856
|
-
|
|
857
|
-
```javaScript
|
|
858
|
-
node call dirWatch true 300
|
|
859
|
-
```
|
|
860
|
-
|
|
861
|
-
The arguments and behaviors described above for execution by a module apply here, too. If the first argument is `true`, you can terminate the process by entering `CTRL-c`.
|
|
862
|
-
|
|
863
|
-
### Network watching
|
|
864
|
-
|
|
865
|
-
Testaro can poll servers for jobs to be performed. Such a server can act as the “controller” described in [How to run a thousand accessibility tests](https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3). The server is responsible for preparing Testaro jobs, assigning them to Testaro agents, receiving reports back from those agents, and performing any further processing of the reports, including enhancement, storage, and disclosure to audiences. It can be any server reachable with a URL. That includes a server running on the same host as Testaro, with a URL such as `localhost:3000`.
|
|
866
|
-
|
|
867
|
-
Network watching is governed by environment variables of the form `NETWATCH_URL_0_JOB`, `NETWATCH_URL_0_OBSERVE`, `NETWATCH_URL_0_REPORT`, and `NETWATCH_URL_0_AUTH`, and by an environment variable `NETWATCH_URLS`.
|
|
868
|
-
|
|
869
|
-
You can create as many quadruples of `…JOB`, `OBSERVE`, `…REPORT`, and `AUTH` variables as you want, one quadruple for each server that the agent may get jobs from. Each quadruple has a different number inside the variable name. The `…JOB` variable is the URL that the agent needs to send a job request to (a typical URL could be `https://testcontroller.xyz.com/api/getJob/agent3`). The `…OBSERVE` variable is the URL that the agent needs to send granular job progress messages to if the job requests that. The `…REPORT` variable is the URL that the agent needs to send a completed report to (such as `localhost:3000/api/submitReport/agent3`). The `…AUTH` variable is the password of the agent that will be recognized by the server. Each URL can contain segments and/or query parameters that identify the purpose of the request, the identity and authorization of the agent, etc.
|
|
870
|
-
|
|
871
|
-
In each quadruple, the `…AUTH` variable is optional. If it is truthy (i.e. it exists and has a non-empty value), then the job request sent to the server will be a `POST` request and the payload will be an object with an `agentPW` property, whose value is the password. Otherwise, i.e. if the variable has an empty string as its value or does not exist, the request will be a `GET` request, and an agent password, if required by the server, will need to be provided in the URL.
|
|
872
|
-
|
|
873
|
-
The `NETWATCH_URLS` variable has a value of the form `0,3,4`. This is a comma-delimited list of the numbers of the servers to be polled.
|
|
874
|
-
|
|
875
|
-
Once Testaro obtains a network job from one of the servers, Testaro performs it and adds the result data to the job, which then becomes a report. Testaro also makes its `AGENT` value the value of the `sources.agent` property of the report. Testaro then sends the report in a `POST` request to the `…REPORT` URL with the same server number. If there is a truthy `…AUTH` variable for the server, the request payload has this format:
|
|
876
|
-
|
|
877
|
-
```json
|
|
878
|
-
{
|
|
879
|
-
"agentPW": "abcdef",
|
|
880
|
-
"report": {
|
|
881
|
-
…
|
|
882
|
-
}
|
|
883
|
-
}
|
|
884
|
-
```
|
|
885
|
-
|
|
886
|
-
If there is no truthy `…AUTH` variable for the server, the request payload is simply the report in JSON format.
|
|
887
|
-
|
|
888
|
-
Thus, the `…AUTH` variables allow Testaro to comply with servers that object to agent passwords being visible in job request URLs and report-submission URLs and in any log messages that reproduce such URLs.
|
|
889
|
-
|
|
890
|
-
If granular reporting is desired, Testaro sends progress messages to the observation URL.
|
|
891
|
-
|
|
892
|
-
Network watching can be repeated or 1-job. 1-job watching stops after 1 job has been performed.
|
|
893
|
-
|
|
894
|
-
After checking all the URLs in succession without getting a job from any of them, Testaro waits for the prescribed time before continuing to check.
|
|
895
|
-
|
|
896
|
-
#### Network watching by a module
|
|
897
|
-
|
|
898
|
-
```javaScript
|
|
899
|
-
const {netWatch} = require('testaro/netWatch');
|
|
900
|
-
netWatch(true, 300, true);
|
|
901
|
-
```
|
|
902
|
-
|
|
903
|
-
In this example, a module of your application asks Testaro to check the servers for a job every 300 seconds, to perform any jobs obtained from any of the servers, and then to continue checking until the process is stopped. If the first argument is `false`, Testaro will stop checking after performing 1 job.
|
|
904
|
-
|
|
905
|
-
The third argument specifies whether Testaro should be certificate-tolerant. A `true` value makes Testaro accept SSL certificates that fail verification against a list of certificate authorities. This allows testing of `https` targets that, for example, use self-signed certificates. If the third argument is omitted, the default for that argument is implemented. The default is `true`.
|
|
906
|
-
|
|
907
|
-
#### Network watching by a user
|
|
908
|
-
|
|
909
|
-
```javaScript
|
|
910
|
-
node call netWatch true 300 true
|
|
911
|
-
```
|
|
912
|
-
|
|
913
|
-
The arguments and behaviors described above for execution by a module apply here, too. If the first argument is `true`, you can terminate the process by entering `CTRL-c`.
|
|
914
|
-
|
|
915
|
-
## Environment variables
|
|
916
|
-
|
|
917
|
-
In addition to their uses described above, environment variables can be used by acts of type `test`, as documented in the `actSpecs.js` file.
|
|
918
|
-
|
|
919
|
-
Before making Testaro run a job, you can optionally also set `HEADED_BROWSER`, `DEBUG`, and/or `WAITS`. The effects of these variables are:
|
|
920
|
-
|
|
921
|
-
- `HEADED_BROWSER`: whether to run the browser in headed mode instead of the default headless mode
|
|
922
|
-
- `DEBUG`: whether to make logging verbose
|
|
923
|
-
- `WAITS`: the number of milliseconds to wait between actions
|
|
924
|
-
|
|
925
|
-
You may store environment variables in an untracked `.env` file if you wish, and Testaro will recognize them. Here is a template for a `.env` file:
|
|
926
|
-
|
|
927
|
-
```conf
|
|
928
|
-
AGENT=agentabc
|
|
929
|
-
HEADED_BROWSER=false
|
|
930
|
-
DEBUG=false
|
|
931
|
-
JOBDIR=../testing/jobs
|
|
932
|
-
NETWATCH_URL_0_JOB=http://localhost:3000/api/assignJob/agentabc
|
|
933
|
-
NETWATCH_URL_0_OBSERVE=http://localhost:3000/api/granular/agentabc
|
|
934
|
-
NETWATCH_URL_0_REPORT=http://localhost:3000/api/takeReport/agentabc
|
|
935
|
-
NETWATCH_URL_0_AUTH=abcxyz
|
|
936
|
-
NETWATCH_URLS=0
|
|
937
|
-
PUPPETEER_DISABLE_HEADLESS_WARNING=true
|
|
938
|
-
REPORTDIR=../testing/reports
|
|
939
|
-
WAITS=0
|
|
940
|
-
WAVE_KEY=yourwavekey
|
|
941
|
-
WAX_KEY=yourwaxkey
|
|
942
|
-
```
|
|
943
|
-
|
|
944
|
-
## Validation
|
|
945
|
-
|
|
946
|
-
### Validators
|
|
947
|
-
|
|
948
|
-
Testaro and the tests of the Testaro tool can be validated with the _executors_ located in the `validation/executors` directory.
|
|
949
|
-
|
|
950
|
-
The executor for a single test is `test`. To execute it for any test `xyz`, call it with the statement `npm test xyz`.
|
|
951
|
-
|
|
952
|
-
The other executors are:
|
|
953
|
-
|
|
954
|
-
- `run`: validates immediate test execution
|
|
955
|
-
- `watchDir`: validates directory watching
|
|
956
|
-
- `watchNet`: validates network watching
|
|
957
|
-
- `tests`: validates all the Testaro tests
|
|
958
|
-
|
|
959
|
-
To execute any executor `xyz` among these, call it with the statement `npm run xyz`.
|
|
960
|
-
|
|
961
|
-
The `tests` executor makes use of the jobs in the `validation/tests/jobs` directory, and they, in turn, run tests on HTML files in the `validation/tests/targets` directory.
|
|
962
|
-
|
|
963
526
|
## Contribution
|
|
964
527
|
|
|
965
|
-
You can define additional Testaro
|
|
528
|
+
You can define additional Testaro rules and functionality. Contributions are welcome.
|
|
966
529
|
|
|
967
|
-
Please report any issues, including feature requests, at the [repository](https://github.com/
|
|
530
|
+
Please report any issues, including feature requests, at the [repository](https://github.com/jrpool/testaro/issues).
|
|
968
531
|
|
|
969
532
|
## Accessibility principles
|
|
970
533
|
|
|
971
|
-
The rationales motivating the Testaro-defined tests can be found in comments within the files of those tests, in the `
|
|
534
|
+
The rationales motivating the Testaro-defined tests can be found in comments within the files of those tests, in the `testaro` directory. Unavoidably, each test is opinionated. Testaro itself, however, can accommodate other tests representing different opinions. Testaro is intended to be neutral with respect to questions such as the criteria for accessibility, the severities of accessibility defects, whether accessibility is binary or graded, and the distinction between usability and accessibility.
|
|
972
535
|
|
|
973
536
|
## Challenges
|
|
974
537
|
|
|
@@ -978,7 +541,7 @@ On some occasions a test throws an error that cannot be handled with a `try`-`ca
|
|
|
978
541
|
|
|
979
542
|
Some tools take apparently infinite time to perform their tests on some pages. One website whose pages prevent 5 of the tools from ever completing their tests is the site of BrowserStack.
|
|
980
543
|
|
|
981
|
-
To handle such fatal errors
|
|
544
|
+
To handle such fatal errors and stalls, Testaro runs the tests of each tool in a separate forked child process that executes the `procs/doTestAct.js` module. The parent process subjects each tool to a time limit and kills the child if the time limit expires.
|
|
982
545
|
|
|
983
546
|
### Activation
|
|
984
547
|
|
|
@@ -990,7 +553,7 @@ The Playwright “Receives Events” actionability check does **not** check whet
|
|
|
990
553
|
|
|
991
554
|
Test targets employ mechanisms to prevent scraping, multiple requests within a short time, automated form submission, and other automated actions. These mechanisms may interfere with testing. When a test act is prevented by a target, Testaro reports this prevention.
|
|
992
555
|
|
|
993
|
-
Some targets prohibit the execution of alien scripts unless the client can demonstrate that it is the requester of the page. Failure to provide that evidence results in the script being blocked and an error message being logged, saying “Refused to execute a script because its hash, its nonce, or unsafe-inline does not appear in the script-src directive of the Content Security Policy”. This mechanism affects tools that insert scripts into a target in order to test it.
|
|
556
|
+
Some targets prohibit the execution of alien scripts unless the client can demonstrate that it is the requester of the page. Failure to provide that evidence results in the script being blocked and an error message being logged, saying “Refused to execute a script because its hash, its nonce, or unsafe-inline does not appear in the script-src directive of the Content Security Policy”. This mechanism affects tools that insert scripts into a target in order to test it. To comply with this requirement, Testaro obtains a _nonce_ from the response that serves the target. Then the file that runs the tool adds that nonce to the script as the value of a `nonce` attribute when it inserts its script into the target.
|
|
994
557
|
|
|
995
558
|
### Tool duplicativity
|
|
996
559
|
|
|
@@ -999,33 +562,23 @@ Tools sometimes do redundant testing, in that two or more tools test for the sam
|
|
|
999
562
|
- One cannot be confident in excluding some tests of some tools on the assumption that they perfectly duplicate tests of other tools.
|
|
1000
563
|
- The Testaro report from a job documents each tool’s results separately, so a single defect may be documented in multiple locations within the report, making the direct consumption of the report inefficient.
|
|
1001
564
|
- An effort to aggregate the results into a single score may distort the scores by inflating the weights of defects that happen to be discovered by multiple tools.
|
|
1002
|
-
-
|
|
1003
|
-
|
|
1004
|
-
To deal with the above problems, you can:
|
|
1005
|
-
|
|
1006
|
-
- configure `test` acts for tools to exclude tests that you consider duplicative
|
|
1007
|
-
- create derivative reports that organize results by defect types rather than by tool
|
|
1008
|
-
- take duplication into account when defining scoring rules
|
|
1009
|
-
|
|
1010
|
-
Some measures of these kinds are included in the scoring and reporting features of the Testilo package.
|
|
565
|
+
- Tools use different methods for identifying the locations of elements that violate tool rules.
|
|
1011
566
|
|
|
1012
567
|
### Tool malfunctions
|
|
1013
568
|
|
|
1014
569
|
Tools can become faulty. For example, Alfa stopped reporting any rule violations in mid-April 2024 and resumed doing so at the end of April. In some cases, such as this, the tool maker corrects the fault. In others, the tool changes and forces Testaro to change its handling of the tool.
|
|
1015
570
|
|
|
1016
|
-
Testaro would become more reliable if the behavior of its tools were monitored for suspect changes.
|
|
1017
|
-
|
|
1018
571
|
### Dependency deployment
|
|
1019
572
|
|
|
1020
|
-
The behavior of Testaro as a dependency of an application deployed on a virtual private server has been observed to be vulnerable to slower performance and more frequent test
|
|
573
|
+
The behavior of Testaro as a dependency of an application deployed on a virtual private server has been observed to be vulnerable to slower performance and more frequent test preventions than when Testaro is deployed as a stand-alone application on a workstation. The configuration of Testaro has been tuned for mitigation of such behaviors.
|
|
1021
574
|
|
|
1022
575
|
### Containerized deployment
|
|
1023
576
|
|
|
1024
|
-
|
|
577
|
+
Experimental deployments of Testaro as a dependency in a containerized application has sometimes resulted in thrown errors that are not thrown when the same application is deployed without containerization.
|
|
1025
578
|
|
|
1026
579
|
### Headless browser fidelity
|
|
1027
580
|
|
|
1028
|
-
Testaro normally performs tests with headless browsers. Some experiments appear to have shown that some test results are inaccurate with headless browsers, but this has not been replicated. The `launch` function in the `run` module accepts a `headEmulation` argument with `'high'` and `'low'` values. Its purpose is to permit optimizations of headless browsers to be turned off
|
|
581
|
+
Testaro normally performs tests with headless browsers. Some experiments appear to have shown that some test results are inaccurate with headless browsers, but this has not been replicated. The `launch` function in the `run` module accepts a `headEmulation` argument with `'high'` and `'low'` values. Its purpose is to permit optimizations of headless browsers to be turned off, so browsers behave and appear more similar to headed browsers. Observation has failed to show any performance cost, so `'high'` is the default value.
|
|
1029
582
|
|
|
1030
583
|
## Repository exclusions
|
|
1031
584
|
|
|
@@ -1046,21 +599,17 @@ Testilo contains procedures that reorganize report data by issue and by element,
|
|
|
1046
599
|
|
|
1047
600
|
Report standardization could be performed by other software rather than by Testaro. That would require sending the original reports to the server. They are typically larger than standardized reports. Whenever users want only standardized reports, the fact that Testaro standardizes them eliminates the need to send the original reports anywhere.
|
|
1048
601
|
|
|
1049
|
-
### Automated accessibility testing at Slack
|
|
1050
|
-
|
|
1051
|
-
[Automated accessibility testing at Slack](https://slack.engineering/automated-accessibility-testing-at-slack/) is based on Playwright, with Axe as a single tool.
|
|
1052
|
-
|
|
1053
602
|
## Code style
|
|
1054
603
|
|
|
1055
604
|
The JavaScript code in this project generally conforms to the ESLint configuration file `.eslintrc.json`. However, the `htmlcs/HTMLCS.js` file implements an older version of JavaScript. Its style is regulated by the `htmlcs/.eslintrc.json` file.
|
|
1056
605
|
|
|
1057
606
|
## History
|
|
1058
607
|
|
|
1059
|
-
Work on the
|
|
608
|
+
Work on the `testaro` tests in this package began in 2017, and work on the multi-package ensemble that Testaro implements began in early 2018. These two aspects were combined into an “Autotest” package in early 2021 and into the more single-purpose packages, Testaro and Testilo, in January 2022.
|
|
1060
609
|
|
|
1061
610
|
On 12 February 2024 ownership of the Testaro repository was transfered from the personal account of contributor Jonathan Pool to the organization account `cvs-health` of CVS Health. The MIT license of the [repository](https://github.com/cvs-health/testaro) did not change, but the copyright holder changed to CVS Health.
|
|
1062
611
|
|
|
1063
|
-
Maintenance of the repository owned by CVS Health came to an end on 30 September 2025. The current repository was forked from the `cvs-health` repository in October 2025 and then unlinked from the fork network.
|
|
612
|
+
Maintenance of the repository owned by CVS Health came to an end on 30 September 2025. The current repository was forked from the `cvs-health` repository in October 2025 and then unlinked from the fork network, by agreement with CVS Health.
|
|
1064
613
|
|
|
1065
614
|
## Contributing
|
|
1066
615
|
|