testaro 11.0.0 → 12.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.eslintrc.json +41 -41
- package/LICENSE +21 -21
- package/README.md +57 -61
- package/actSpecs.js +364 -358
- package/call.js +85 -85
- package/continuum/AccessEngine.community.js +791 -791
- package/continuum/Continuum.community.js +2697 -2697
- package/continuum/continuum.conf.js +19 -19
- package/data/roles.txt +62 -62
- package/htmlcs/.eslintrc.json +67 -67
- package/htmlcs/HTMLCS.js +8204 -8204
- package/htmlcs/licence.txt +24 -24
- package/package.json +2 -1
- package/procs/allText.js +76 -76
- package/procs/allVis.js +17 -17
- package/procs/linksByType.js +54 -54
- package/procs/textOf.txt +73 -73
- package/procs/wavedoc.json +1617 -1617
- package/run.js +1549 -1548
- package/samples/00000-ts18-example.json +231 -231
- package/tests/alfa.js +138 -138
- package/tests/allHidden.js +49 -49
- package/tests/attVal.js +48 -48
- package/tests/axe.js +118 -118
- package/tests/bulk.js +22 -22
- package/tests/continuum.js +71 -71
- package/tests/docType.js +14 -14
- package/tests/elements.js +159 -159
- package/tests/embAc.js +36 -36
- package/tests/filter.js +47 -47
- package/tests/focAll.js +57 -57
- package/tests/focInd.js +167 -167
- package/tests/focOp.js +138 -138
- package/tests/focVis.js +29 -29
- package/tests/hover.js +381 -381
- package/tests/htmlcs.js +90 -90
- package/tests/labClash.js +159 -159
- package/tests/linkTo.js +25 -25
- package/tests/linkUl.js +68 -68
- package/tests/menuNav.js +264 -264
- package/tests/miniText.js +48 -48
- package/tests/motion.js +125 -125
- package/tests/nonTable.js +67 -67
- package/tests/nuVal.js +92 -92
- package/tests/qualWeb.js +85 -0
- package/tests/radioSet.js +97 -97
- package/tests/role.js +483 -483
- package/tests/styleDiff.js +153 -153
- package/tests/tabNav.js +328 -328
- package/tests/tenon.js +130 -130
- package/tests/textNodes.js +140 -140
- package/tests/title.js +11 -11
- package/tests/titledEl.js +31 -31
- package/tests/wave.js +59 -59
- package/tests/zIndex.js +49 -49
- package/validation/done/README.md +3 -3
- package/validation/executors/run.js +40 -40
- package/validation/executors/test.js +7 -7
- package/validation/executors/tests.js +22 -22
- package/validation/executors/watchDir.js +28 -28
- package/validation/executors/watchNet.js +117 -117
- package/validation/jobs/done/README.md +3 -3
- package/validation/jobs/todo/00000-simple-example.json +35 -35
- package/validation/jobs/todo/README.md +3 -3
- package/validation/tests/jobs/allHidden.json +314 -314
- package/validation/tests/jobs/bulk.json +48 -48
- package/validation/tests/jobs/docType.json +46 -46
- package/validation/tests/jobs/elements.json +140 -140
- package/validation/tests/jobs/embAc.json +54 -54
- package/validation/tests/jobs/filter.json +55 -55
- package/validation/tests/jobs/focAll.json +68 -68
- package/validation/tests/jobs/focInd.json +69 -69
- package/validation/tests/jobs/focOp.json +62 -62
- package/validation/tests/jobs/focVis.json +35 -35
- package/validation/tests/jobs/hover.json +118 -118
- package/validation/tests/jobs/labClash.json +52 -52
- package/validation/tests/jobs/linkTo.json +35 -35
- package/validation/tests/jobs/linkUl.json +71 -71
- package/validation/tests/jobs/menuNav.json +106 -106
- package/validation/tests/jobs/miniText.json +36 -36
- package/validation/tests/jobs/motion.json +62 -62
- package/validation/tests/jobs/nonTable.json +37 -37
- package/validation/tests/jobs/radioSet.json +52 -52
- package/validation/tests/jobs/role.json +60 -60
- package/validation/tests/jobs/styleDiff.json +71 -71
- package/validation/tests/jobs/tabNav.json +106 -106
- package/validation/tests/jobs/textNodes.json +98 -98
- package/validation/tests/jobs/title.json +46 -46
- package/validation/tests/jobs/titledEl.json +37 -37
- package/validation/tests/jobs/zIndex.json +49 -49
- package/validation/tests/targets/allHidden/ariaHiddenMain.html +15 -15
- package/validation/tests/targets/allHidden/good.html +15 -15
- package/validation/tests/targets/allHidden/hiddenMain.html +16 -16
- package/validation/tests/targets/allHidden/mixedHidden0.html +16 -16
- package/validation/tests/targets/allHidden/mixedHidden1.html +16 -16
- package/validation/tests/targets/allHidden/noBody.html +11 -11
- package/validation/tests/targets/allHidden/noMain.html +13 -13
- package/validation/tests/targets/allHidden/noneDoc.html +15 -15
- package/validation/tests/targets/allHidden/noneMain.html +15 -15
- package/validation/tests/targets/allHidden/semiHidden.html +15 -15
- package/validation/tests/targets/allHidden/visHiddenMain.html +15 -15
- package/validation/tests/targets/bulk/bad.html +48 -48
- package/validation/tests/targets/bulk/good.html +15 -15
- package/validation/tests/targets/docType/bad.html +14 -14
- package/validation/tests/targets/docType/good.html +15 -15
- package/validation/tests/targets/elements/index.html +40 -40
- package/validation/tests/targets/embAc/bad.html +21 -21
- package/validation/tests/targets/embAc/good.html +15 -15
- package/validation/tests/targets/filter/bad.html +27 -27
- package/validation/tests/targets/filter/good.html +19 -19
- package/validation/tests/targets/focAll/good.html +15 -15
- package/validation/tests/targets/focAll/less.html +15 -15
- package/validation/tests/targets/focAll/more.html +16 -16
- package/validation/tests/targets/focInd/bad.html +66 -66
- package/validation/tests/targets/focInd/good.html +22 -22
- package/validation/tests/targets/focOp/bad.html +18 -18
- package/validation/tests/targets/focOp/good.html +15 -15
- package/validation/tests/targets/focVis/index.html +23 -23
- package/validation/tests/targets/hover/bad.html +25 -25
- package/validation/tests/targets/hover/good.html +20 -20
- package/validation/tests/targets/hover/large.html +27 -27
- package/validation/tests/targets/hover/styleBad.html +35 -35
- package/validation/tests/targets/labClash/bad.html +24 -24
- package/validation/tests/targets/labClash/good.html +22 -22
- package/validation/tests/targets/linkTo/index.html +15 -15
- package/validation/tests/targets/linkUl/bad.html +35 -35
- package/validation/tests/targets/linkUl/good.html +30 -30
- package/validation/tests/targets/linkUl/na.html +20 -20
- package/validation/tests/targets/menuNav/bad.html +106 -106
- package/validation/tests/targets/menuNav/bad.js +348 -348
- package/validation/tests/targets/menuNav/good.html +106 -106
- package/validation/tests/targets/menuNav/good.js +365 -365
- package/validation/tests/targets/menuNav/style.css +22 -22
- package/validation/tests/targets/miniText/index.html +33 -33
- package/validation/tests/targets/motion/bad.css +15 -15
- package/validation/tests/targets/motion/bad.html +16 -16
- package/validation/tests/targets/motion/good.html +15 -15
- package/validation/tests/targets/nonTable/index.html +65 -65
- package/validation/tests/targets/radioSet/bad.html +34 -34
- package/validation/tests/targets/radioSet/good.html +27 -27
- package/validation/tests/targets/role/bad.html +48 -48
- package/validation/tests/targets/role/good.html +22 -22
- package/validation/tests/targets/styleDiff/bad.html +35 -35
- package/validation/tests/targets/styleDiff/good.html +36 -36
- package/validation/tests/targets/tabNav/bad.html +51 -51
- package/validation/tests/targets/tabNav/bad.js +35 -35
- package/validation/tests/targets/tabNav/good.html +53 -53
- package/validation/tests/targets/tabNav/good.js +83 -83
- package/validation/tests/targets/tabNav/goodMoz.js +205 -205
- package/validation/tests/targets/tabNav/style.css +34 -34
- package/validation/tests/targets/textNodes/index.html +40 -40
- package/validation/tests/targets/title/bad.html +14 -14
- package/validation/tests/targets/title/good.html +15 -15
- package/validation/tests/targets/titledEl/index.html +17 -17
- package/validation/tests/targets/zIndex/bad.html +17 -17
- package/validation/tests/targets/zIndex/good.html +15 -15
- package/validation/validateTest.js +44 -44
- package/validation/watch/done/README.md +3 -3
- package/validation/watch/todo/README.md +3 -3
- package/watch.js +4 -1
package/.eslintrc.json
CHANGED
|
@@ -1,41 +1,41 @@
|
|
|
1
|
-
{
|
|
2
|
-
"env": {
|
|
3
|
-
"browser": true,
|
|
4
|
-
"commonjs": true,
|
|
5
|
-
"es2021": true,
|
|
6
|
-
"node": true
|
|
7
|
-
},
|
|
8
|
-
"extends": "eslint:recommended",
|
|
9
|
-
"parserOptions": {
|
|
10
|
-
"ecmaVersion": 2021
|
|
11
|
-
},
|
|
12
|
-
"rules": {
|
|
13
|
-
"indent": [
|
|
14
|
-
"error",
|
|
15
|
-
2,
|
|
16
|
-
{
|
|
17
|
-
"MemberExpression": 0,
|
|
18
|
-
"ObjectExpression": "first"
|
|
19
|
-
}
|
|
20
|
-
],
|
|
21
|
-
"linebreak-style": [
|
|
22
|
-
"error",
|
|
23
|
-
"unix"
|
|
24
|
-
],
|
|
25
|
-
"quotes": [
|
|
26
|
-
"error",
|
|
27
|
-
"single"
|
|
28
|
-
],
|
|
29
|
-
"semi": [
|
|
30
|
-
"error",
|
|
31
|
-
"always"
|
|
32
|
-
],
|
|
33
|
-
"no-use-before-define": [
|
|
34
|
-
"error"
|
|
35
|
-
],
|
|
36
|
-
"brace-style": [
|
|
37
|
-
"error",
|
|
38
|
-
"stroustrup"
|
|
39
|
-
]
|
|
40
|
-
}
|
|
41
|
-
}
|
|
1
|
+
{
|
|
2
|
+
"env": {
|
|
3
|
+
"browser": true,
|
|
4
|
+
"commonjs": true,
|
|
5
|
+
"es2021": true,
|
|
6
|
+
"node": true
|
|
7
|
+
},
|
|
8
|
+
"extends": "eslint:recommended",
|
|
9
|
+
"parserOptions": {
|
|
10
|
+
"ecmaVersion": 2021
|
|
11
|
+
},
|
|
12
|
+
"rules": {
|
|
13
|
+
"indent": [
|
|
14
|
+
"error",
|
|
15
|
+
2,
|
|
16
|
+
{
|
|
17
|
+
"MemberExpression": 0,
|
|
18
|
+
"ObjectExpression": "first"
|
|
19
|
+
}
|
|
20
|
+
],
|
|
21
|
+
"linebreak-style": [
|
|
22
|
+
"error",
|
|
23
|
+
"unix"
|
|
24
|
+
],
|
|
25
|
+
"quotes": [
|
|
26
|
+
"error",
|
|
27
|
+
"single"
|
|
28
|
+
],
|
|
29
|
+
"semi": [
|
|
30
|
+
"error",
|
|
31
|
+
"always"
|
|
32
|
+
],
|
|
33
|
+
"no-use-before-define": [
|
|
34
|
+
"error"
|
|
35
|
+
],
|
|
36
|
+
"brace-style": [
|
|
37
|
+
"error",
|
|
38
|
+
"stroustrup"
|
|
39
|
+
]
|
|
40
|
+
}
|
|
41
|
+
}
|
package/LICENSE
CHANGED
|
@@ -1,21 +1,21 @@
|
|
|
1
|
-
MIT License
|
|
2
|
-
|
|
3
|
-
Copyright (c) 2021 Jonathan Pool
|
|
4
|
-
|
|
5
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
-
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
-
in the Software without restriction, including without limitation the rights
|
|
8
|
-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
-
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
-
furnished to do so, subject to the following conditions:
|
|
11
|
-
|
|
12
|
-
The above copyright notice and this permission notice shall be included in all
|
|
13
|
-
copies or substantial portions of the Software.
|
|
14
|
-
|
|
15
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
-
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
-
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
-
SOFTWARE.
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2021 Jonathan Pool
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -4,9 +4,9 @@ Federated accessibility test automation
|
|
|
4
4
|
|
|
5
5
|
## Summary
|
|
6
6
|
|
|
7
|
-
Testaro is a collection of
|
|
7
|
+
Testaro is a collection of web accessibility testing tools.
|
|
8
8
|
|
|
9
|
-
The purpose of Testaro is to provide programmatic access to accessibility tests defined
|
|
9
|
+
The purpose of Testaro is to provide programmatic access to accessibility tests defined by several tools, including Testaro itself.
|
|
10
10
|
|
|
11
11
|
Testaro launches and controls web browsers, performing operations, conducting tests, and recording results.
|
|
12
12
|
|
|
@@ -31,30 +31,32 @@ Testaro uses:
|
|
|
31
31
|
- [Playwright](https://playwright.dev/) to launch browsers, perform user actions in them, and perform tests
|
|
32
32
|
- [pixelmatch](https://www.npmjs.com/package/pixelmatch) to measure motion
|
|
33
33
|
|
|
34
|
-
Testaro includes some of its own accessibility tests. In addition, it performs the tests
|
|
34
|
+
Testaro includes some of its own accessibility tests. In addition, it performs the tests of these tools:
|
|
35
35
|
- [accessibility-checker](https://www.npmjs.com/package/accessibility-checker) (the IBM Equal Access Accessibility Checker)
|
|
36
36
|
- [alfa](https://alfa.siteimprove.com/) (Siteimprove alfa)
|
|
37
|
+
- [axe-playwright](https://www.npmjs.com/package/axe-playwright) (Deque Axe-core)
|
|
37
38
|
- [Continuum Community Edition](https://www.webaccessibility.com/tools/)
|
|
38
39
|
- [HTML CodeSniffer](https://www.npmjs.com/package/html_codesniffer) (Squiz HTML CodeSniffer)
|
|
39
|
-
- [
|
|
40
|
+
- [Nu Html Checker](https://github.com/validator/validator)
|
|
41
|
+
- [QualWeb core](https://www.npmjs.com/package/@qualweb/core)
|
|
40
42
|
- [Tenon](https://tenon.io/documentation/what-tenon-tests.php) (Level Access)
|
|
41
43
|
- [WAVE API](https://wave.webaim.org/api/) (WebAIM WAVE)
|
|
42
|
-
- [Nu Html Checker](https://github.com/validator/validator)
|
|
43
44
|
|
|
44
45
|
Some of the Testaro tests are derived from tests performed by the [BBC Accessibility Standards Checker](https://github.com/bbc/bbc-a11y).
|
|
45
46
|
|
|
46
|
-
As of this version, the counts of tests
|
|
47
|
+
As of this version, the counts of tests of the tools referenced above were:
|
|
47
48
|
- Alfa: 103
|
|
48
49
|
- Axe-core: 138
|
|
49
50
|
- Continuum Community Edition: 267
|
|
50
51
|
- Equal Access: 163
|
|
51
52
|
- HTML CodeSniffer: 98
|
|
53
|
+
- Nu Html Checker: 147
|
|
54
|
+
- QualWeb core: 121
|
|
52
55
|
- Tenon: 180
|
|
53
56
|
- WAVE: 110
|
|
54
|
-
-
|
|
55
|
-
- subtotal: 1206
|
|
57
|
+
- subtotal: 1327
|
|
56
58
|
- Testaro tests: 24
|
|
57
|
-
- grand total:
|
|
59
|
+
- grand total: 1351
|
|
58
60
|
|
|
59
61
|
## Quasi-tests
|
|
60
62
|
|
|
@@ -103,7 +105,7 @@ However, if the Playwright dependency is ever updated to a newer version, you mu
|
|
|
103
105
|
|
|
104
106
|
## Payment
|
|
105
107
|
|
|
106
|
-
All of the tests that Testaro can perform are free of cost, except those
|
|
108
|
+
All of the tests that Testaro can perform are free of cost, except those performed by the Tenon and WAVE tools. The owner of each of those tools gives new registrants a free allowance of credits before it becomes necessary to pay for use of the API of the tool. The required environment variables for authentication and payment are described below under “Environment variables”.
|
|
107
109
|
|
|
108
110
|
## Process objects
|
|
109
111
|
|
|
@@ -111,7 +113,7 @@ All of the tests that Testaro can perform are free of cost, except those in the
|
|
|
111
113
|
|
|
112
114
|
A _job_ is an object containing instructions for Testaro.
|
|
113
115
|
|
|
114
|
-
A _report_ is a job with
|
|
116
|
+
A _report_ is a job with properties added by Testaro, describing the results.
|
|
115
117
|
|
|
116
118
|
### Jobs
|
|
117
119
|
|
|
@@ -138,7 +140,7 @@ Here is an example of a job:
|
|
|
138
140
|
{
|
|
139
141
|
type: 'test',
|
|
140
142
|
which: 'alfa',
|
|
141
|
-
what: 'Siteimprove alfa
|
|
143
|
+
what: 'Siteimprove alfa tool'
|
|
142
144
|
}
|
|
143
145
|
],
|
|
144
146
|
sources: {
|
|
@@ -157,21 +159,21 @@ Here is an example of a job:
|
|
|
157
159
|
|
|
158
160
|
This job contains three `acts`, telling Testaro to:
|
|
159
161
|
1. open a page in the Chromium browser
|
|
160
|
-
1. navigate to
|
|
161
|
-
1. perform the tests
|
|
162
|
+
1. navigate to a specified URL
|
|
163
|
+
1. perform the tests of the `alfa` tool on that URL
|
|
162
164
|
|
|
163
165
|
Job properties:
|
|
164
|
-
- `id`: This is a string consisting of alphanumeric ASCII characters and hyphen-minus (-), intended to be unique. When
|
|
166
|
+
- `id`: This is a string consisting of alphanumeric ASCII characters and hyphen-minus (-), intended to be unique. When the above example job is saved as a JSON file, the file name is `be76p-ts25-w3c.json`. Typically, a job is created from a _script_, and the job ID adds a timestamp prefix and a target suffix to the script ID. Here the script ID would have been `ts25`.
|
|
165
167
|
- `what`: This is a description of the job.
|
|
166
168
|
- `strict`: This is `true` or `false`, indicating whether _substantive redirections_ should be treated as failures. These are redirections that do more than add or subtract a final slash. For example, if `strict` is true, a redirection from `xyz.com` to `www.xyz.com` or to `xyz.com/en` will abort the job.
|
|
167
169
|
- `timeLimit`: This property is the number of seconds allowed for the execution of the job.
|
|
168
170
|
- `acts`: This is an array of the acts to be performed. Acts are documented below.
|
|
169
171
|
- `sources`: This object has properties describing where the job came from:
|
|
170
|
-
- `script`: This is the ID of the script from which the job was made. Other applications, such as Testilo, can make jobs from scripts. When Testilo creates a job, the job inherits its `id`, `what`, `strict`, `timeLimit`, and `acts` properties from the script. However, Testilo can create multiple jobs from a single script, replacing acts of type `placeholder` with one or more target-specific acts. Examples of scripts can be found in the Testilo package.
|
|
172
|
+
- `script`: This is the ID of the script from which the job was made, if it was made from a script, or is otherwise an empty string. Other applications, such as Testilo, can make jobs from scripts. When Testilo creates a job, the job inherits its `id`, `what`, `strict`, `timeLimit`, and `acts` properties from the script. However, Testilo can create multiple jobs from a single script, replacing acts of type `placeholder` with one or more target-specific acts. Examples of scripts can be found in the Testilo package.
|
|
171
173
|
- `batch`: If the job was one of a set of jobs created by a merger of a script and a batch of targets, this property’s value is the ID of the batch, or otherwise is an empty string.
|
|
172
|
-
- `target`: If the job was made from a script with placeholder acts, this property describes the target whose target-specific acts have replaced the placeholder acts. Otherwise `target` is an empty object. Testilo also
|
|
174
|
+
- `target`: If the job was made from a script with placeholder acts, this property describes the target whose target-specific acts have replaced the placeholder acts. Otherwise `target` is an empty object. Testilo also uses the `id` property of the target as the third segment of the job ID.
|
|
173
175
|
- `requester`: This string is the email address to receive a notice of completion of the running of the job.
|
|
174
|
-
- `creationTime`: This is the time when the job was created
|
|
176
|
+
- `creationTime`: This is the time when the job was created.
|
|
175
177
|
- `timeStamp`: This string is a compact representation of the job creation time, suitable for inclusion in the ID of the job.
|
|
176
178
|
|
|
177
179
|
### Reports
|
|
@@ -198,7 +200,7 @@ The subsequent acts can tell Testaro to perform any of:
|
|
|
198
200
|
- _moves_ (clicks, text inputs, hovers, etc.)
|
|
199
201
|
- _navigations_ (browser launches, visits to URLs, waits for page conditions, etc.)
|
|
200
202
|
- _alterations_ (changes to the page)
|
|
201
|
-
- _tests_ (whether in dependency
|
|
203
|
+
- _tests_ (whether in dependency tools or defined within Testaro)
|
|
202
204
|
- _branching_ (continuing from an act other than the next one)
|
|
203
205
|
|
|
204
206
|
##### Moves
|
|
@@ -222,9 +224,9 @@ When the texts of multiple elements of the same type will contain the same `whic
|
|
|
222
224
|
|
|
223
225
|
##### Navigations
|
|
224
226
|
|
|
225
|
-
An example of a **navigation** is the
|
|
227
|
+
An example of a **navigation** is the act of type `url` above.
|
|
226
228
|
|
|
227
|
-
Once you have included a `url`
|
|
229
|
+
Once you have included a `url` act in a job, you do not need to add more `url` acts unless you want the browser to visit a different URL or revisit the same URL.
|
|
228
230
|
|
|
229
231
|
If any act alters the page, you can restore the page to its original state for the next act by inserting new `launch` and `url` acts (and, if necessary, additional page-specific acts) between them.
|
|
230
232
|
|
|
@@ -240,7 +242,7 @@ Another navigation example is:
|
|
|
240
242
|
|
|
241
243
|
In this case, Testaro waits until the page title contains the string “travel” (case-insensitively).
|
|
242
244
|
|
|
243
|
-
The `launch` navigation
|
|
245
|
+
The `launch` navigation act allows you to specify a “lowMotion” property as `true`. If you do, then the browser creates tabs with the `reduce-motion` option set to `reduce` instead of `no-preference`. This makes the browser act as if the user has chosen a [motion-reduction option in the settings of the operating system or browser](https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-reduced-motion#user_preferences). However, there are often motions on web pages that this option fails to suppress, such as those on the [Inditex](https://www.inditex.com/itxcomweb/en/home) and [Rescuing Leftover Cuisine](https://www.rescuingleftovercuisine.org) home pages. Carousel motion is also not suppressed.
|
|
244
246
|
|
|
245
247
|
##### Alterations
|
|
246
248
|
|
|
@@ -253,19 +255,15 @@ An example of an **alteration** is:
|
|
|
253
255
|
}
|
|
254
256
|
```
|
|
255
257
|
|
|
256
|
-
This
|
|
258
|
+
This act causes Testaro to alter the `display` and `visibility` style properties of all elements, where necessary, so those properties do not make any element invisible.
|
|
257
259
|
|
|
258
260
|
##### Tests
|
|
259
261
|
|
|
260
262
|
###### Introduction
|
|
261
263
|
|
|
262
|
-
|
|
264
|
+
An act of type `test` performs operations and reports a result. The result may indicate that a page passes or fails requirements. Typically, accessibility tests report successes and failures. But a test in Testaro is defined less restrictively, so it can report any result. As one example, the Testaro `elements` test reports facts about certain elements on a page, without asserting that those facts are successes or failures.
|
|
263
265
|
|
|
264
|
-
The
|
|
265
|
-
- An act is a test (_test command_) if its `type` property has the value `test`.
|
|
266
|
-
- An act with type `test` whose `which` value is the name of a package, such as Continuum, performs multiple tests defined by that _test package_.
|
|
267
|
-
|
|
268
|
-
Thus, if a command of type `test` runs Continuum, Continuum performs multiple tests and reports their results.
|
|
266
|
+
The `which` property of a `test` act identifies the operations to perform. If the value of `which` is the name of one of the tools, such as `alfa`, then the operations are some or all of the tests of that tool. If the value is the name of a Testaro test, then the operations are those of that single Testaro test. Thus, a single `test` act may specify performing anything from a single test to hundreds of tests.
|
|
269
267
|
|
|
270
268
|
###### Configuration
|
|
271
269
|
|
|
@@ -285,13 +283,13 @@ test: [
|
|
|
285
283
|
],
|
|
286
284
|
```
|
|
287
285
|
|
|
288
|
-
That means that a test (i.e.
|
|
286
|
+
That means that a test (i.e. an act with a `type` property having the value `'test'`) must have a string-valued `which` property naming a test and may optionally have a string-valued `what` property describing the test.
|
|
289
287
|
|
|
290
288
|
If a particular test either must have or may have any other properties, those properties must be specified in the `tests` property in `actSpecs.js`.
|
|
291
289
|
|
|
292
290
|
###### Examples
|
|
293
291
|
|
|
294
|
-
An example of a
|
|
292
|
+
An example of a `test` act invoking a **tool** is:
|
|
295
293
|
|
|
296
294
|
```json
|
|
297
295
|
{
|
|
@@ -304,7 +302,7 @@ An example of a **packaged test** is:
|
|
|
304
302
|
|
|
305
303
|
In this case, Testaro runs the WAVE test with report type 1.
|
|
306
304
|
|
|
307
|
-
An example of a **Testaro
|
|
305
|
+
An example of a `test` act invoking a **Testaro** test is:
|
|
308
306
|
|
|
309
307
|
```json
|
|
310
308
|
{
|
|
@@ -321,15 +319,15 @@ In this case, Testaro runs the `motion` test with the specified parameters.
|
|
|
321
319
|
|
|
322
320
|
###### Continuum
|
|
323
321
|
|
|
324
|
-
The `continuum`
|
|
322
|
+
The `continuum` tests makes use of the files in the `continuum` directory. The test inserts the contents of all three files into the page as scripts and then uses them to perform the tests of the Continuum tool.
|
|
325
323
|
|
|
326
324
|
Level Access on 22 August 2022 granted authorization for the copying of the `AccessEngine.community.js` file insofar as necessary for allowing Continuum community edition tests to be included in Testaro.
|
|
327
325
|
|
|
328
326
|
###### IBM Equal Access
|
|
329
327
|
|
|
330
|
-
The `ibm`
|
|
328
|
+
The `ibm` tests require the `aceconfig.js` file.
|
|
331
329
|
|
|
332
|
-
As of 2 March 2023 (version 3.1.45 of `accessibility-checker`), the `ibm`
|
|
330
|
+
As of 2 March 2023 (version 3.1.45 of `accessibility-checker`), the `ibm` tool threw errors when hosted under the Windows operating system. To prevent these errors, it was possible to edit two files in the `accessibility-checker` package as follows:
|
|
333
331
|
|
|
334
332
|
In `node_modules/accessibility-checker/lib/ACEngineManager.js`, remove or comment out these lines starting on line 169:
|
|
335
333
|
|
|
@@ -348,11 +346,11 @@ results.label = results.label.replace(/:/g, '-');
|
|
|
348
346
|
|
|
349
347
|
These changes were proposed as pull requests 1333 and 1334 (https://github.com/IBMa/equal-access/pulls).
|
|
350
348
|
|
|
351
|
-
If you choose to invoke
|
|
349
|
+
If you choose to invoke the `ibm` tests with the `withNewContent` property specified, you will choose whether the tested content is the content of the existing page or is retrieved anew with the document URL. Typically, both methods succeed and deliver similar results. However, sometimes one method succeeds and the other fails, or one method reports more violations than the other does. In those cases, most often the success, or the larger violation count, arises from the existing page (`withNewContent: false`).
|
|
352
350
|
|
|
353
351
|
###### HTML CodeSniffer
|
|
354
352
|
|
|
355
|
-
The `htmlcs`
|
|
353
|
+
The `htmlcs` tests make use of the `htmlcs/HTMLCS.js` file. That file was created, and can be recreated if necessary, as follows:
|
|
356
354
|
|
|
357
355
|
1. Clone the [HTML CodeSniffer package](https://github.com/squizlabs/HTML_CodeSniffer).
|
|
358
356
|
1. Make that package’s directory the active directory.
|
|
@@ -384,7 +382,7 @@ The changes in `htmlcs/HTMLCS.js` are:
|
|
|
384
382
|
|
|
385
383
|
###### Tenon
|
|
386
384
|
|
|
387
|
-
Most
|
|
385
|
+
Most tools require only one act, but the `tenon` tool requires two acts:
|
|
388
386
|
- An act of type `tenonRequest`.
|
|
389
387
|
- An act of type `test` with `tenon` as the value of `which`.
|
|
390
388
|
|
|
@@ -410,23 +408,23 @@ Example:
|
|
|
410
408
|
}
|
|
411
409
|
```
|
|
412
410
|
|
|
413
|
-
The reason for this is that the Tenon API operates asynchronously. You ask it to perform a test, and it puts your request into a queue. To learn whether Tenon has completed your test, you make a status request. You can continue making status requests until Tenon replies that your test has been completed. Then you submit a request for the test result, and Tenon replies with the result. (As of May 2022, however, status requests were observed to misreport still-running tests as completed. The `tenon` test works around that by requesting only the result and using the response to determine whether the tests have been completed.)
|
|
411
|
+
The reason for this is that the Tenon API operates asynchronously. You ask it to perform a test, and it puts your request into a queue. To learn whether Tenon has completed your test, you make a status request. You can continue making status requests until Tenon replies that your test has been completed. Then you submit a request for the test result, and Tenon replies with the result. (As of May 2022, however, status requests were observed to misreport still-running tests as completed. The `tenon` test act works around that by requesting only the result and using the response to determine whether the tests have been completed.)
|
|
414
412
|
|
|
415
413
|
Tenon says that tests are typically completed in 3 to 6 seconds but that the latency can be longer, depending on demand.
|
|
416
414
|
|
|
417
415
|
Therefore, you can include a `tenonRequest` act early in your job, and a `tenon` test act late in your job. Tenon will move your request through its queue while Testaro is processing your job. When Testaro reaches your `tenon` test act, Tenon will most likely have completed your test. If not, the `tenon` test will wait and then make a second request before giving up.
|
|
418
416
|
|
|
419
|
-
Thus, a `tenon` test actually does not perform any test; it merely collects the result. The page that was active when the `tenonRequest` act was performed is the one that Tenon tests.
|
|
417
|
+
Thus, a `tenon` test act actually does not perform any test; it merely collects the result. The page that was active when the `tenonRequest` act was performed is the one that Tenon tests.
|
|
420
418
|
|
|
421
|
-
In case you want to perform more than
|
|
419
|
+
In case you want to perform the Tenon tests more than once in the same job, you can do so. Just give each pair of acts a distinct `id` property, so each `tenon` test act will request the correct result.
|
|
422
420
|
|
|
423
421
|
Tenon recommends giving it a public URL rather than giving it the content of a page, if possible. So, it is best to give the `withNewContent` property of the `tenonRequest` act the value `true`, unless the page is not public.
|
|
424
422
|
|
|
425
|
-
If a `tenon` test is included in a job, environment variables named `TENON_USER` and `TENON_PASSWORD` must exist, with your Tenon username and password, respectively, as their values.
|
|
423
|
+
If a `tenon` test act is included in a job, environment variables named `TENON_USER` and `TENON_PASSWORD` must exist, with your Tenon username and password, respectively, as their values. These could be obtained from [Tenon](https://tenon.io/documentation/overview) until Tenon was closed to new subscribers in or about October 2022.
|
|
426
424
|
|
|
427
425
|
###### WAVE
|
|
428
426
|
|
|
429
|
-
If a `wave` test is included in the job, an environment variable named `WAVE_KEY` must exist, with your WAVE API key as its value. You can get it from [WebAIM](https://wave.webaim.org/api/).
|
|
427
|
+
If a `wave` test act is included in the job, an environment variable named `WAVE_KEY` must exist, with your WAVE API key as its value. You can get it from [WebAIM](https://wave.webaim.org/api/).
|
|
430
428
|
|
|
431
429
|
The `wave` API does not accept a transmitted document for testing. WAVE must be given only a URL, which it then visits to perform its tests. Therefore, you cannot manipulate a page and then have WAVE test it, or ask WAVE to test a page that cannot be reached directly with a URL.
|
|
432
430
|
|
|
@@ -438,7 +436,7 @@ The BBC Accessibility Standards Checker has obsolete dependencies with security
|
|
|
438
436
|
|
|
439
437
|
##### Branching
|
|
440
438
|
|
|
441
|
-
An example of a **branching**
|
|
439
|
+
An example of a **branching** act is:
|
|
442
440
|
|
|
443
441
|
```json
|
|
444
442
|
{
|
|
@@ -449,9 +447,9 @@ An example of a **branching** command is:
|
|
|
449
447
|
}
|
|
450
448
|
```
|
|
451
449
|
|
|
452
|
-
This
|
|
450
|
+
This act checks the result of the previous act to determine whether its `result.totals.invalid` property has a positive value. If so, it changes the next act to be performed, specifying the act 4 acts before this one.
|
|
453
451
|
|
|
454
|
-
A `next
|
|
452
|
+
A `next` act can use a `next` property instead of a `jump` property. The value of the `next` property is an act name. It tells Testaro to continue performing acts starting with the act having that value as the value of its `name` property.
|
|
455
453
|
|
|
456
454
|
#### `actSpecs` file
|
|
457
455
|
|
|
@@ -507,7 +505,7 @@ The validity criterion named in item 2 may be any of these:
|
|
|
507
505
|
|
|
508
506
|
Any `test` act can also (in addition to the requirements in `actSpecs.js`) contain an `expect` property. If it does, the value of that property must be an array of arrays. Each array specifies expectations about the results of the test.
|
|
509
507
|
|
|
510
|
-
For example, a `test`
|
|
508
|
+
For example, a `test` act might have this `expect` property:
|
|
511
509
|
|
|
512
510
|
```javaScript
|
|
513
511
|
'expect': [
|
|
@@ -533,7 +531,7 @@ The second item in each array, if there are 3 items in the array, is an operator
|
|
|
533
531
|
|
|
534
532
|
The third item in each array, if there are 3 items in the array, is the criterion with which the value of the first property is compared.
|
|
535
533
|
|
|
536
|
-
A typical use for an `expect` property is checking the correctness of a Testaro test. Thus, the validation jobs in the `validation/tests/jobs` directory all contain `test`
|
|
534
|
+
A typical use for an `expect` property is checking the correctness of a Testaro test. Thus, the validation jobs in the `validation/tests/jobs` directory all contain `test` acts with `expect` properties. See the “Validation” section below.
|
|
537
535
|
|
|
538
536
|
## Execution
|
|
539
537
|
|
|
@@ -636,7 +634,7 @@ The URL to which Testaro sends reports is given by the `sources.sendReportTo` pr
|
|
|
636
634
|
|
|
637
635
|
### Environment variables
|
|
638
636
|
|
|
639
|
-
In addition to their uses described above, environment variables can be used by
|
|
637
|
+
In addition to their uses described above, environment variables can be used by acts of type `text`, as documented in the `actSpecs.js` file.
|
|
640
638
|
|
|
641
639
|
Before making Testaro run a job, you can optionally also set `process.env.DEBUG` (to `'true'` or anything else) and/or `process.env.WAITS` (to a non-negative integer). The effects of these variables are described in the `run.js` file.
|
|
642
640
|
|
|
@@ -677,13 +675,13 @@ The `tests` executor makes use of the jobs in the `validation/tests/jobs` direct
|
|
|
677
675
|
|
|
678
676
|
## Contribution
|
|
679
677
|
|
|
680
|
-
You can define additional Testaro
|
|
678
|
+
You can define additional Testaro acts and functionality. Contributions are welcome.
|
|
681
679
|
|
|
682
680
|
Please report any issues, including feature requests, at the [repository](https://github.com/jrpool/testaro/issues).
|
|
683
681
|
|
|
684
682
|
## Accessibility principles
|
|
685
683
|
|
|
686
|
-
The rationales motivating the Testaro-defined tests can be found in comments within the files of those tests, in the `tests` directory. Unavoidably, each test is opinionated. Testaro itself, however, can accommodate other tests representing different opinions. Testaro is intended to be neutral with respect to questions such as the criteria for accessibility, the severities of accessibility
|
|
684
|
+
The rationales motivating the Testaro-defined tests can be found in comments within the files of those tests, in the `tests` directory. Unavoidably, each test is opinionated. Testaro itself, however, can accommodate other tests representing different opinions. Testaro is intended to be neutral with respect to questions such as the criteria for accessibility, the severities of accessibility defects, whether accessibility is binary or graded, and the distinction between usability and accessibility.
|
|
687
685
|
|
|
688
686
|
## Testing challenges
|
|
689
687
|
|
|
@@ -697,18 +695,16 @@ Testing to determine what happens when a control or link is activated is straigh
|
|
|
697
695
|
|
|
698
696
|
The Playwright “Receives Events” actionability check does **not** check whether an event is dispatched on an element. It checks only whether a click on the location of the element makes the element the target of that click, rather than some other element occupying the same location.
|
|
699
697
|
|
|
700
|
-
###
|
|
701
|
-
|
|
702
|
-
Test packages sometimes do redundant testing, in that two or more packages test for the same issues, although such duplications are not necessarily perfect. This fact creates three problems:
|
|
703
|
-
- One cannot be confident in excluding some tests of some packages on the assumption that they perfectly duplicate tests of other packages.
|
|
704
|
-
- The Testaro report from a job documents each package’s results separately, so a single defect may be documented in multiple locations within the report, making the direct consumption of the report inefficient.
|
|
705
|
-
- An effort to aggregate the results into a single score may distort the scores by inflating the weights of defects that happen to be discovered by multiple packages.
|
|
698
|
+
### Tool duplicativity
|
|
706
699
|
|
|
707
|
-
|
|
700
|
+
Tools sometimes do redundant testing, in that two or more tools test for the same defects, although such duplications are not necessarily perfect. This fact creates three problems:
|
|
701
|
+
- One cannot be confident in excluding some tests of some tools on the assumption that they perfectly duplicate tests of other tools.
|
|
702
|
+
- The Testaro report from a job documents each tool’s results separately, so a single defect may be documented in multiple locations within the report, making the direct consumption of the report inefficient.
|
|
703
|
+
- An effort to aggregate the results into a single score may distort the scores by inflating the weights of defects that happen to be discovered by multiple tools.
|
|
708
704
|
|
|
709
705
|
To deal with the above problems, you can:
|
|
710
|
-
-
|
|
711
|
-
- create derivative reports that organize results by defect types rather than by
|
|
706
|
+
- configure `test` acts for tools to exclude tests that you consider duplicative
|
|
707
|
+
- create derivative reports that organize results by defect types rather than by tool
|
|
712
708
|
- take duplication into account when defining scoring rules
|
|
713
709
|
|
|
714
710
|
Some measures of these kinds are included in the scoring and reporting features of the Testilo package.
|
|
@@ -725,7 +721,7 @@ The files in the `temp` directory are presumed ephemeral and are not tracked by
|
|
|
725
721
|
- produces human-oriented HTML digests from scored reports
|
|
726
722
|
- produces human-oriented HTML comparisons of the scores of targets
|
|
727
723
|
|
|
728
|
-
Testilo contains procedures that reorganize report data by defect rather than
|
|
724
|
+
Testilo contains procedures that reorganize report data by defect rather than tool, and that compensate for duplicative tests when computing scores.
|
|
729
725
|
|
|
730
726
|
Testaro is derived from [Autotest](https://github.com/jrpool/autotest). Autotest was created as a monolithic accessibility testing package, but that forced functionalities to be hosted on a workstation merely because it was impractical to host Playwright elsewhere. Testaro embodies an architectural decision to isolate workstation-dependent functionalities.
|
|
731
727
|
|