testaro 60.4.1 → 60.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +8 -5
- package/README.md +105 -52
- package/UPGRADES.md +53 -0
- package/package.json +1 -1
- package/procs/doTestAct.js +27 -26
- package/procs/shoot.js +51 -0
- package/procs/visChange.js +2 -4
- package/run.js +32 -28
- package/testaro/motion.js +41 -24
- package/testaro/motionSolo.js +1 -1
- package/testaro/shoot0.js +26 -0
- package/testaro/shoot1.js +26 -0
- package/tests/testaro.js +26 -8
- package/tests/wave.js +3 -2
- package/testaro/shoot.js +0 -57
package/AGENTS.md
CHANGED
|
@@ -1,11 +1,7 @@
|
|
|
1
|
-
/*
|
|
2
|
-
© 2025 Jonathan Robert Pool. All rights reserved.
|
|
3
|
-
Licensed under the MIT License. See LICENSE file for details.
|
|
4
|
-
*/
|
|
5
|
-
|
|
6
1
|
# Testaro Agent Guidelines
|
|
7
2
|
|
|
8
3
|
## Commands
|
|
4
|
+
|
|
9
5
|
- **Run all tests**: `npm run tests`
|
|
10
6
|
- **Run single test**: `npm test <testname>` (e.g., `npm test hover`)
|
|
11
7
|
- **Run job**: `npm run run [jobID]`
|
|
@@ -14,6 +10,7 @@
|
|
|
14
10
|
- **Lint**: `npx eslint <file>` (follows `.eslintrc.json`)
|
|
15
11
|
|
|
16
12
|
## Architecture
|
|
13
|
+
|
|
17
14
|
- **Main dirs**: `/tests` (tool test definitions), `/procs` (shared procedures), `/validation` (test validators), root (core modules)
|
|
18
15
|
- **Key files**: `run.js` (main executor), `actSpecs.js` (act specifications), `call.js` (CLI entry), `tests/testaro.js` (Testaro tool rules)
|
|
19
16
|
- **Tools**: Integrates 11 a11y tools (Axe, Alfa, IBM Checker, QualWeb, ASLint, WAVE, WallyAX, Ed11y, HTML CodeSniffer, Nu Validator, Testaro)
|
|
@@ -21,8 +18,14 @@
|
|
|
21
18
|
- **Env vars**: Required for WallyAX (`WAX_KEY`), WAVE (`WAVE_KEY`); optional `DEBUG`, `WAITS`, `JOBDIR`, `REPORTDIR`, `AGENT`
|
|
22
19
|
|
|
23
20
|
## Code Style
|
|
21
|
+
|
|
24
22
|
- **ESLint**: 2-space indent, single quotes, semicolons required, Stroustrup brace style, no use-before-define
|
|
25
23
|
- **Imports**: CommonJS (`require`/`module.exports`), not ES modules
|
|
26
24
|
- **Naming**: camelCase for vars/functions, descriptive names, rule IDs in lowercase
|
|
27
25
|
- **Error handling**: Try-catch blocks, report failures via `prevented: true` in results
|
|
28
26
|
- **Comments**: Explain complex logic, but keep concise
|
|
27
|
+
|
|
28
|
+
## License
|
|
29
|
+
|
|
30
|
+
© 2025 Jonathan Robert Pool. All rights reserved.
|
|
31
|
+
Licensed under the MIT License. See LICENSE file for details.
|
package/README.md
CHANGED
|
@@ -7,10 +7,12 @@ Ensemble testing for web accessibility
|
|
|
7
7
|
Testaro is an application for automated web accessibility testing.
|
|
8
8
|
|
|
9
9
|
The purposes of Testaro are to:
|
|
10
|
+
|
|
10
11
|
- provide programmatic access to accessibility tests defined by several tools
|
|
11
12
|
- facilitate the integration of the reports of the tools into a unified report
|
|
12
13
|
|
|
13
14
|
Testaro is described in two papers:
|
|
15
|
+
|
|
14
16
|
- [How to run a thousand accessibility tests](https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3)
|
|
15
17
|
- [Testaro: Efficient Ensemble Testing for Web Accessibility](https://arxiv.org/abs/2309.10167)
|
|
16
18
|
|
|
@@ -23,6 +25,7 @@ Testaro can be installed under a MacOS, Windows, Debian, or Ubuntu operating sys
|
|
|
23
25
|
Testaro accepts _jobs_, performs them, and returns _reports_.
|
|
24
26
|
|
|
25
27
|
Other software, located on the same or a different host, can make use of Testaro, performing functions such as:
|
|
28
|
+
|
|
26
29
|
- Job preparation
|
|
27
30
|
- Converting user specifications into jobs
|
|
28
31
|
- Job scheduling
|
|
@@ -41,12 +44,14 @@ One software product that performs some such functions is [Testilo](https://www.
|
|
|
41
44
|
## Dependencies
|
|
42
45
|
|
|
43
46
|
Testaro uses:
|
|
47
|
+
|
|
44
48
|
- [Playwright](https://playwright.dev/) to launch browsers, perform user actions in them, and perform tests
|
|
45
49
|
- [playwright-extra](https://www.npmjs.com/package/playwright-extra) and [puppeteer-extra-plugin-stealth](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to make a Playwright-controlled browser more indistinguishable from a human-operated browser and thus make their requests more likely to succeed
|
|
46
50
|
- [playwright-dompath](https://www.npmjs.com/package/playwright-dompath) to retrieve XPaths of elements
|
|
47
51
|
- [pixelmatch](https://www.npmjs.com/package/pixelmatch) to measure motion
|
|
48
52
|
|
|
49
53
|
Testaro performs tests of these _tools_:
|
|
54
|
+
|
|
50
55
|
- [Accessibility Checker](https://www.npmjs.com/package/accessibility-checker) (IBM)
|
|
51
56
|
- [Alfa](https://alfa.siteimprove.com/) (Siteimprove)
|
|
52
57
|
- [ASLint](https://www.npmjs.com/package/@essentialaccessibility/aslint) (eSSENTIAL Accessibility)
|
|
@@ -65,7 +70,7 @@ Some of the tests of Testaro are designed to act as approximate alternatives to
|
|
|
65
70
|
|
|
66
71
|
Each tool accessed with Testaro defines _rules_ and tests _targets_ for compliance with its rules. Testilo has classified the rules into _issues_ and deprecated some rules as poorly implemented. If the deprecated rules are excluded, the counts are:
|
|
67
72
|
|
|
68
|
-
```
|
|
73
|
+
```text
|
|
69
74
|
Accessibility Checker: 93
|
|
70
75
|
Alfa: 64
|
|
71
76
|
ASLint: 129
|
|
@@ -85,6 +90,7 @@ This tabulation may not be exact, because some of the tools are under active dev
|
|
|
85
90
|
## Code organization
|
|
86
91
|
|
|
87
92
|
The main directories containing code files are:
|
|
93
|
+
|
|
88
94
|
- package root: main code files
|
|
89
95
|
- `tests`: files containing the code defining particular tests
|
|
90
96
|
- `procs`: shared procedures
|
|
@@ -134,9 +140,9 @@ All of the tests that Testaro can perform are free of cost, except those perform
|
|
|
134
140
|
|
|
135
141
|
## Jobs
|
|
136
142
|
|
|
137
|
-
A _job_ is an object that specifies what Testaro is to do. As Testaro performs a job, Testaro reports results by adding data to the job.
|
|
143
|
+
A _job_ is an object that specifies what Testaro is to do. As Testaro performs a job, Testaro reports results by adding data to the job and making that enhanced object available as a _report_.
|
|
138
144
|
|
|
139
|
-
### Example
|
|
145
|
+
### Example of a job
|
|
140
146
|
|
|
141
147
|
Here is an example of a job:
|
|
142
148
|
|
|
@@ -201,6 +207,7 @@ This job tells Testaro to perform two _acts_. One performs one test of the Axe t
|
|
|
201
207
|
Each act includes a `launch` property with a default value. That instructs Testaro, before performing those tests, to launch a new Webkit browser, open a context (window) with some properties of an iPhone 8 and without a reduced-motion setting, create a page (tab), and navigate to a particular page of the `abccorp.com` website.
|
|
202
208
|
|
|
203
209
|
Job properties:
|
|
210
|
+
|
|
204
211
|
- `id`: a string uniquely identifying the job.
|
|
205
212
|
- `what`: a description of the job.
|
|
206
213
|
- `strict`: `true` or `false`, indicating whether _substantive redirections_ should be treated as failures. These are redirections that do more than add or subtract a final slash.
|
|
@@ -216,13 +223,16 @@ Job properties:
|
|
|
216
223
|
|
|
217
224
|
## Acts
|
|
218
225
|
|
|
219
|
-
|
|
226
|
+
As shown above, `acts` is a property of a job and has an array value. Each item in the array is an object that specifies an _act_.
|
|
227
|
+
|
|
228
|
+
### Introduction to acts
|
|
220
229
|
|
|
221
230
|
Each act object has a `type` property and optionally has a `name` property (used in branching, described below). It must or may have other properties, depending on the value of `type`.
|
|
222
231
|
|
|
223
232
|
### Act types
|
|
224
233
|
|
|
225
234
|
The acts can tell Testaro to perform any of:
|
|
235
|
+
|
|
226
236
|
- _navigations_ (browser launches, visits to URLs, waits for page conditions, etc.)
|
|
227
237
|
- _moves_ (clicks, text inputs, hovers, etc.)
|
|
228
238
|
- _alterations_ (changes to the page)
|
|
@@ -294,9 +304,9 @@ This act checks the result of the previous act to determine whether its `result.
|
|
|
294
304
|
|
|
295
305
|
A `next` act can use a `next` property instead of a `jump` property. The value of the `next` property is an act name. It tells Testaro to continue performing acts starting with the act having that value as the value of its `name` property.
|
|
296
306
|
|
|
297
|
-
####
|
|
307
|
+
#### Tools
|
|
298
308
|
|
|
299
|
-
##### Introduction
|
|
309
|
+
##### Introduction to tools
|
|
300
310
|
|
|
301
311
|
An act of type `test` performs the tests of a tool and reports a result. The result may indicate that a page passes or fails requirements. Typically, accessibility tests report successes and failures. But a test in Testaro is defined less restrictively, so it can report any result. As one example, the Testaro `elements` test reports facts about certain elements on a page, without asserting that those facts are successes or failures.
|
|
302
312
|
|
|
@@ -305,6 +315,7 @@ The `which` property of a `test` act identifies a tool, such as `alfa` or `testa
|
|
|
305
315
|
##### Configuration
|
|
306
316
|
|
|
307
317
|
Every tool invoked by Testaro must have:
|
|
318
|
+
|
|
308
319
|
- a property in the `tests` object defined in the `run.js` file, where the property name is the ID representing the tool and the value is the name of the tool
|
|
309
320
|
- a `.js` file, defining the operation of the tool, in the `tests` directory, whose name base is the name of the tool
|
|
310
321
|
|
|
@@ -330,7 +341,7 @@ When you include a `rules` property, you limit the tests of the tool that are pe
|
|
|
330
341
|
|
|
331
342
|
The `nuVal`, `qualWeb`, and `testaro` tools require specific formats for the `rules` property. Those formats are described below in the sections about those tools.
|
|
332
343
|
|
|
333
|
-
##### Examples
|
|
344
|
+
##### Examples of test acts
|
|
334
345
|
|
|
335
346
|
An example of a `test` act is:
|
|
336
347
|
|
|
@@ -393,6 +404,7 @@ The first item in each array is an identifier of a property of the act. The item
|
|
|
393
404
|
If there is only 1 item in an array, it states the expectation that the specified property does not exist. Otherwise, there are 3 items in the array.
|
|
394
405
|
|
|
395
406
|
The second item in each array, if there are 3 items, is an operator, drawn from:
|
|
407
|
+
|
|
396
408
|
- `<`: less than
|
|
397
409
|
- `=`: equal to
|
|
398
410
|
- `>`: greater than
|
|
@@ -404,9 +416,7 @@ The third item in each array, if there are 3 items in the array, is the criterio
|
|
|
404
416
|
|
|
405
417
|
A typical use for an `expect` property is checking the correctness of a Testaro test. Thus, the validation jobs in the `validation/tests/jobs` directory all contain `test` acts with `expect` properties. See the “Validation” section below.
|
|
406
418
|
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
### Tools
|
|
419
|
+
### Tool details
|
|
410
420
|
|
|
411
421
|
The tools whose tests Testaro performs have particularities described below.
|
|
412
422
|
|
|
@@ -467,7 +477,7 @@ In `node_modules/accessibility-checker/lib/reporters/ACReporterJSON.js`, add the
|
|
|
467
477
|
results.label = results.label.replace(/:/g, '-');
|
|
468
478
|
```
|
|
469
479
|
|
|
470
|
-
These changes were proposed as pull requests 1333 and 1334
|
|
480
|
+
These changes were proposed as [pull requests 1333 and 1334](https://github.com/IBMa/equal-access/pulls).
|
|
471
481
|
|
|
472
482
|
The `ibm` tool is one of two tools (`testaro` is the other) with a `withItems` property. If you set `withItems` to `false`, the result includes the counts of “violations” and “recommendations”, but no information about the rules that gave rise to them.
|
|
473
483
|
|
|
@@ -490,6 +500,7 @@ QualWeb allows specification of rules for 3 modules: `act-rules`, `wcag-techniqu
|
|
|
490
500
|
```
|
|
491
501
|
|
|
492
502
|
In that format:
|
|
503
|
+
|
|
493
504
|
- Replace `mod` with `act`, `wcag`, or `best`.
|
|
494
505
|
- Replace `m`, `n`, `o`, `p`, etc. with the 0 or more integers that identify rules.
|
|
495
506
|
|
|
@@ -505,23 +516,23 @@ The target can be provided to QualWeb either as an existing page or as a URL. Ex
|
|
|
505
516
|
|
|
506
517
|
#### Testaro
|
|
507
518
|
|
|
508
|
-
|
|
519
|
+
The rules that Testaro can test for are implemented in files within the `testaro` directory.
|
|
509
520
|
|
|
510
|
-
|
|
521
|
+
If you do not specify rules when using the `testaro` tool, Testaro will test for its default rules, namely the rules that have `true` values on the `defaultOn` property in the `allRules` array defined in the `tests/testaro.js` file. It will test for these rules in the order in which they appear in the array.
|
|
522
|
+
|
|
523
|
+
The optional `rules` argument for a `testaro` test act is an array whose first item is either `'y'` or `'n'` and whose remaining items are rule IDs. If `'y'`, then only the specified rules’ tests are performed. If `'n'`, then all the default rules are tested for, **except** for the specified rules.
|
|
511
524
|
|
|
512
525
|
The `testaro` tool (like the `ibm` tool) has a `withItems` property. If you set it to `false`, the `standardResult` object will contain an `instances` property with summaries that identify issues and instance counts. If you set it to `true`, some of the instances will be itemized.
|
|
513
526
|
|
|
514
527
|
Unlike any other tool, the `testaro` tool requires a `stopOnFail` property, which specifies whether a failure to conform to any rule (i.e. any value of `totals` other than `[0, 0, 0, 0]`) should terminate the execution of tests for the remaining rules.
|
|
515
528
|
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
Several Testaro tests make use of the `init()` function in the `procs/testaro` module. That function samples elements if the population of elements to be tested is larger than 100. The purpose is to achieve reasonable performance. The sampling overweights elements near the beginning of a page, because of the tendency of that location to have important and atypical elements.
|
|
529
|
+
Some Testaro tests make use of the `init()` function in the `procs/testaro` module. That function samples elements if the population of elements to be tested is larger than 100. The purpose is to achieve reasonable performance. The sampling overweights elements near the beginning of a page, because of the tendency of that location to have important and atypical elements.
|
|
519
530
|
|
|
520
|
-
You can add custom rules to the rules of any tool. Testaro provides a template, `data/template.js`, for the definition of a rule to be added. Once you have created a copy of the template with revisions, you can move the copy into the `testaro` directory and add an entry for your custom rule to the `
|
|
531
|
+
You can add custom rules to the rules of any tool. Testaro provides a template, `data/template.js`, for the definition of a rule to be added. Once you have created a copy of the template with revisions, you can move the copy into the `testaro` directory and add an entry for your custom rule to the `allRules` object in the `tests/testaro.js` file. Then your custom rule will act as a Testaro rule. Some `testaro` rules are simple enough to be fully specified in JSON files. You can use any of those as a template if you want to create a sufficiently simple custom rule, namely a rule whose prohibited elements are all and only the elements matching a CSS selector. More details about rule creation are in the `CONTRIBUTING.md` file.
|
|
521
532
|
|
|
522
533
|
#### WallyAX
|
|
523
534
|
|
|
524
|
-
If a `wax` test act is included in the job, an environment variable named `WAX_KEY` must exist, with your WallyAX API key as its value. You can
|
|
535
|
+
If a `wax` test act is included in the job, an environment variable named `WAX_KEY` must exist, with your WallyAX API key as its value. You can obtain it from [WallyAX](https://account.wallyax.com/?ref_app=Developer&app_type=npm).
|
|
525
536
|
|
|
526
537
|
The `wax` tool imposes a limit on the size of a page to be tested. If the page exceeds the limit, Testaro treats the page as preventing `wax` from performing its tests. The limit is less than 500,000 characters.
|
|
527
538
|
|
|
@@ -535,8 +546,6 @@ If you want the stand-alone API to perform the tests, you need to have that API
|
|
|
535
546
|
|
|
536
547
|
### Browser types
|
|
537
548
|
|
|
538
|
-
The warning comments in the `testaro/hover.js` and `testaro/motion.js` files state that those tests operate correctly only with the `webkit` browser type. The warning comment in the `testaro/focInd.js` file states that that test operates incorrectly with the `firefox` browser type.
|
|
539
|
-
|
|
540
549
|
When you want to run some tests of a tool with one browser type and other tests of the same tool with another browser type, you can do so by splitting the rules into two test acts. For example, one test act can specify the rules as
|
|
541
550
|
|
|
542
551
|
```javascript
|
|
@@ -553,7 +562,7 @@ Together, they get all tests of the tool performed. Before each test act, you ca
|
|
|
553
562
|
|
|
554
563
|
### `actSpecs` file
|
|
555
564
|
|
|
556
|
-
#### Introduction
|
|
565
|
+
#### Introduction to the `actSpecs` file
|
|
557
566
|
|
|
558
567
|
The `actSpecs.js` file contains rules governing acts. The rules determine whether an act is valid.
|
|
559
568
|
|
|
@@ -588,12 +597,14 @@ The rule is an array with two elements: a string ('Click a link') describing the
|
|
|
588
597
|
The requirement `which: [true, 'string', 'hasLength', 'substring of the link text']` specifies what is required for the `which` property of a `link`-type act. The requirement is an array.
|
|
589
598
|
|
|
590
599
|
In most cases, the array has length 4:
|
|
591
|
-
|
|
592
|
-
-
|
|
593
|
-
-
|
|
594
|
-
-
|
|
600
|
+
|
|
601
|
+
- Item 0. Is the property (here `which`) required (`true` or `false`)? The value `true` here means that every `link`-type act **must** contain a `which` property.
|
|
602
|
+
- Item 1. What format must the property value have (`'string'`, `'array'`, `'boolean'`, `'number'`, or `'object'`)?
|
|
603
|
+
- Item 2. What other validity criterion applies (if any)? (Empty string if none.) The `hasLength` criterion means that the string must be at least 1 character long.
|
|
604
|
+
- Item 3. Description of the property. In this example, the description says that the value of `which` must be a substring of the text content of the link that is to be clicked. Thus, a `link` act tells Testaro to find the first link whose text content has this substring and click it.
|
|
595
605
|
|
|
596
606
|
The validity criterion named in item 2 may be any of these:
|
|
607
|
+
|
|
597
608
|
- `'hasLength'`: is not a blank string
|
|
598
609
|
- `'isURL`': is a string starting with `http`, `https`, or `file`, then `://`, then ending with 1 or more non-whitespace characters
|
|
599
610
|
- `'isBrowserType'`: is `'chromium'`, `'firefox'`, or `'webkit'`
|
|
@@ -605,45 +616,79 @@ The validity criterion named in item 2 may be any of these:
|
|
|
605
616
|
|
|
606
617
|
## Reports
|
|
607
618
|
|
|
608
|
-
|
|
619
|
+
Any Testaro job produces a report, which is a copy of the job with additional data produced by Testaro as it performed the job. Like a job, a report is an object that can be serialized to JSON for file storage and network transmission.
|
|
620
|
+
|
|
621
|
+
### Job-level data
|
|
622
|
+
|
|
623
|
+
The data that Testaro adds to a job to create a report include job-level data: data describing the how the job as a whole was performed. Examples: when it was completed and how long it took to run.
|
|
624
|
+
|
|
625
|
+
Properties that were in a job when it was given to Testaro remain unchanged in the report. New data produced by Testaro during its performance of a job are inserted into a new `jobData` property in the report.
|
|
609
626
|
|
|
610
|
-
|
|
627
|
+
### Act-level data
|
|
611
628
|
|
|
612
|
-
Testaro also
|
|
629
|
+
Testaro also adds act-level data to each job. The new act-level data are properties added to each `act` object.
|
|
613
630
|
|
|
614
|
-
|
|
631
|
+
#### Act-level data from `test` acts
|
|
615
632
|
|
|
616
|
-
|
|
617
|
-
- results of tests conducted by tools
|
|
618
|
-
- process data, including statistics on:
|
|
619
|
-
- latency (how long a time each tool takes to run its tests)
|
|
620
|
-
- test prevention (the failure of tools to run on particular targets)
|
|
621
|
-
- logging (browser messaging, including about document errors, during testing)
|
|
633
|
+
In a `test` act, one of the 11 tools performs tests and reports the results. Testaro manages this performance with the `reporter` function of a file located in the `tests` directory. Each tool has a corresponding file, such as `alfa.js` for the `alfa` tool.
|
|
622
634
|
|
|
623
|
-
|
|
635
|
+
The `reporter` function returns an object with this structure:
|
|
624
636
|
|
|
625
|
-
|
|
637
|
+
```js
|
|
638
|
+
{
|
|
639
|
+
data: {
|
|
640
|
+
prevented: boolean (whether the tool failed to perform its tests on the page),
|
|
641
|
+
error: string (if `prevented` is `true`, a description of the error)
|
|
642
|
+
… (other tool-specific data)
|
|
643
|
+
},
|
|
644
|
+
result: object (the results of the tests performed by the tool)
|
|
645
|
+
}
|
|
646
|
+
```
|
|
647
|
+
|
|
648
|
+
On the completion of a job, Testaro has added these properties to each `test` act to produce a report:
|
|
649
|
+
|
|
650
|
+
- `what` (string): the name of the tool
|
|
651
|
+
- `actualURL` (string): the URL of the visited page, after any redirections
|
|
652
|
+
- `startTime` (string): when the tool was started
|
|
653
|
+
- `endTime` (string): when the tool reported its results
|
|
654
|
+
- `data` (object): other tool-specific data:
|
|
655
|
+
- `prevented` (boolean): whether the tool failed to perform its tests
|
|
656
|
+
- `error` (string): a description of the failure, if any
|
|
657
|
+
- other tool-specific data, if any
|
|
658
|
+
|
|
659
|
+
Testaro may also add these properties to any `test` act:
|
|
660
|
+
|
|
661
|
+
- `expectations` (object): the results of validations specified by the act in `expect` properties
|
|
662
|
+
- `expectationFailures` (number): the count of failed validations
|
|
663
|
+
|
|
664
|
+
The value of `expectations` is an array of objects, one object per expectation. Each object includes a `property` property identifying the expectation, and a `passed` property with `true` or `false` value reporting whether the expectation was satisfied. If applicable, it also has other properties identifying what was expected and what was actually reported.
|
|
665
|
+
|
|
666
|
+
Testaro also adds one or both of these properties to each `test` act:
|
|
626
667
|
|
|
627
|
-
|
|
668
|
+
- `result` (object): the results of the tests performed by the tool, in the native format of the tool
|
|
669
|
+
- `standardResult` (object): the `result` property converted to a Testaro-standard structure
|
|
628
670
|
|
|
629
|
-
A
|
|
630
|
-
|
|
631
|
-
-
|
|
671
|
+
A job specifies whether the report should include, for each `test` act, the `result` property, the `standardResult` property, or both.
|
|
672
|
+
|
|
673
|
+
#### Act-level data from `testaro` test acts
|
|
674
|
+
|
|
675
|
+
Each Testaro rule module exports a `reporter` function, which returns an object with `data`, `totals`, and `standardInstances` properties. Testaro combines the values of those properties with the corresponding values of the same properties from the other `testaro` tests to create the `data` and `result` properties added to `testaro` test acts.
|
|
632
676
|
|
|
633
677
|
#### Standard result
|
|
634
678
|
|
|
635
679
|
##### Properties
|
|
636
680
|
|
|
637
|
-
The
|
|
638
|
-
|
|
639
|
-
- `
|
|
640
|
-
- `
|
|
681
|
+
The `standardResult` property, when added to a `test` act, includes three properties:
|
|
682
|
+
|
|
683
|
+
- `prevented` (boolean): whether the tool failed to perform its tests on the page.
|
|
684
|
+
- `totals` (array of numbers): counts of rule violations at 4 ordinal severity levels. For example, the array `[3, 0, 14, 10]` reports that there were 3 violations at level 0, 0 at level 1, 14 at level 2, and 10 at level 3.
|
|
685
|
+
- `instances` (array of objects): descriptions of rule violations reported. An instance can describe a single violation, usually by one element in the page, or can summarize multiple violations of the same rule.
|
|
641
686
|
|
|
642
687
|
If the value of `prevented` is `true`, the standard result also includes an `error` property describing the reason for the failure.
|
|
643
688
|
|
|
644
689
|
##### Instances
|
|
645
690
|
|
|
646
|
-
Here is an example of a standard
|
|
691
|
+
Here is an example of an instance in a standard result:
|
|
647
692
|
|
|
648
693
|
```javascript
|
|
649
694
|
{
|
|
@@ -664,9 +709,10 @@ Here is an example of a standard instance:
|
|
|
664
709
|
}
|
|
665
710
|
```
|
|
666
711
|
|
|
667
|
-
This instance
|
|
712
|
+
This instance says that a `button` element violates a rule named `rule01`.
|
|
668
713
|
|
|
669
714
|
The element has no `id` attribute to distinguish it from other `button` elements, but the tool describes its location. This tool uses an XPath to do that. Tools use various methods for location description, namely:
|
|
715
|
+
|
|
670
716
|
- `line` (line number in the code of the page): Nu Html Checker
|
|
671
717
|
- `selector` (CSS selector): Axe, QualWeb, WAVE
|
|
672
718
|
- `xpath`: Alfa, ASLint, Equal Access
|
|
@@ -678,12 +724,14 @@ The tool also reproduces an excerpt of the element code.
|
|
|
678
724
|
##### Element identification
|
|
679
725
|
|
|
680
726
|
While the above properties can help you find the offending element, Testaro makes this easier by adding, where practical, two standard element identifiers to each standard instance:
|
|
727
|
+
|
|
681
728
|
- `boxID`: a compact representation of the x, y, width, and height of the element bounding box, if the element can be identified and is visible.
|
|
682
729
|
- `pathID`: the XPath of the element, if the element can be identified.
|
|
683
730
|
|
|
684
731
|
These standard identifiers can help you determine whether violations reported by different tools belong to the same element or different elements. The `boxID` property can also support the making of images of the violating elements.
|
|
685
732
|
|
|
686
733
|
Some tools limit the efficacy of the current algorithm for standard identifiers:
|
|
734
|
+
|
|
687
735
|
- HTML CodeSniffer does not report element locations, and the reported code excerpts exclude all text content.
|
|
688
736
|
- Nu Html Checker reports line and column boundaries of element start tags and truncates element text content in reported code excerpts.
|
|
689
737
|
|
|
@@ -692,6 +740,7 @@ Testing can change the pages being tested, and such changes can cause a particul
|
|
|
692
740
|
##### Standardization configuration
|
|
693
741
|
|
|
694
742
|
Each job specifies how Testaro is to handle report standardization. A job contains a `standard` property, with one of the following values to determine which results the report will include:
|
|
743
|
+
|
|
695
744
|
- `'also'`: original and standard.
|
|
696
745
|
- `'only'`: standard only.
|
|
697
746
|
- `'no'`: original only.
|
|
@@ -701,12 +750,13 @@ If a tool has the option to be used without itemization and is being so used, th
|
|
|
701
750
|
##### Standardization opinionation
|
|
702
751
|
|
|
703
752
|
This standard format reflects some judgments. For example:
|
|
753
|
+
|
|
704
754
|
- The `ordinalSeverity` property of an instance involves interpretation. Tools may report severity, certainty, priority, or some combination of those. They may use ordinal or metric quantifications. If they quantify ordinally, their scales may have more or fewer than 4 ranks. Testaro coerces each tool’s severity, certainty, and/or priority classification into a 4-rank ordinal classification. This classification is deemed to express the most common pattern among the tools.
|
|
705
755
|
- The `tagName` property of an instance may not always be obvious, because in some cases the rule being tested for requires a relationship among more than one element (e.g., “An X element may not have a Y element as its parent”).
|
|
706
756
|
- The `ruleID` property of an instance is a matching rule if the tool issues a message but no rule identifier for each instance. The `nuVal` tool does this. In this case, Testaro is classifying the messages into rules.
|
|
707
757
|
- The `ruleID` property of an instance may reclassify tool rules. For example, if a tool rule covers multiple situations that are dissimilar, that rule may be split into multiple rules with distinct `ruleID` properties.
|
|
708
758
|
|
|
709
|
-
You are not dependent on the judgments incorporated into the standard format, because Testaro can give you the original reports from the tools.
|
|
759
|
+
You are not dependent on the judgments incorporated into the standard format, because Testaro can give you the original reports from the tools as the `result` property of a `test` act.
|
|
710
760
|
|
|
711
761
|
The standard format does not express opinions on issue classification. A rule ID identifies something deemed to be an issue by a tool. Useful reporting from multi-tool testing still requires the classification of tool **rules** into **issues**. If tool `A` has `alt-incomplete` as a rule ID and tool `B` has `image_alt_stub` as a rule ID, Testaro does not decide whether those are really the same issue or different issues. That decision belongs to you. The standardization of tool reports by Testaro eliminates some of the drudgery in issue classification, but not any of the judgment required for issue classification.
|
|
712
762
|
|
|
@@ -755,7 +805,7 @@ In watch mode, Testaro periodically checks for a job to run and, when a job is o
|
|
|
755
805
|
|
|
756
806
|
Testaro can watch for a job in a directory of the filesystem where Testaro or your application is located, with the `dirWatch` function.
|
|
757
807
|
|
|
758
|
-
####
|
|
808
|
+
#### Directory watching by a module
|
|
759
809
|
|
|
760
810
|
```javaScript
|
|
761
811
|
const {dirWatch} = require('testaro/dirWatch');
|
|
@@ -768,7 +818,7 @@ Testaro checks for jobs in the `todo` subdirectory of `JOBDIR`. When it has perf
|
|
|
768
818
|
|
|
769
819
|
Testaro creates a report for each job and saves the report in the `raw` subdirectory of `REPORTDIR`.
|
|
770
820
|
|
|
771
|
-
####
|
|
821
|
+
#### Directory watching by a user
|
|
772
822
|
|
|
773
823
|
```javaScript
|
|
774
824
|
node call dirWatch true 300
|
|
@@ -809,7 +859,7 @@ Network watching can be repeated or 1-job. 1-job watching stops after 1 job has
|
|
|
809
859
|
|
|
810
860
|
After checking all the URLs in succession without getting a job from any of them, Testaro waits for the prescribed time before continuing to check.
|
|
811
861
|
|
|
812
|
-
####
|
|
862
|
+
#### Network watching by a module
|
|
813
863
|
|
|
814
864
|
```javaScript
|
|
815
865
|
const {netWatch} = require('testaro/netWatch');
|
|
@@ -820,7 +870,7 @@ In this example, a module of your application asks Testaro to check the servers
|
|
|
820
870
|
|
|
821
871
|
The third argument specifies whether Testaro should be certificate-tolerant. A `true` value makes Testaro accept SSL certificates that fail verification against a list of certificate authorities. This allows testing of `https` targets that, for example, use self-signed certificates. If the third argument is omitted, the default for that argument is implemented. The default is `true`.
|
|
822
872
|
|
|
823
|
-
####
|
|
873
|
+
#### Network watching by a user
|
|
824
874
|
|
|
825
875
|
```javaScript
|
|
826
876
|
node call netWatch true 300 true
|
|
@@ -904,12 +954,14 @@ Some targets prohibit the execution of alien scripts unless the client can demon
|
|
|
904
954
|
### Tool duplicativity
|
|
905
955
|
|
|
906
956
|
Tools sometimes do redundant testing, in that two or more tools test for the same defects, although such duplications are not necessarily perfect. This fact creates problems:
|
|
957
|
+
|
|
907
958
|
- One cannot be confident in excluding some tests of some tools on the assumption that they perfectly duplicate tests of other tools.
|
|
908
959
|
- The Testaro report from a job documents each tool’s results separately, so a single defect may be documented in multiple locations within the report, making the direct consumption of the report inefficient.
|
|
909
960
|
- An effort to aggregate the results into a single score may distort the scores by inflating the weights of defects that happen to be discovered by multiple tools.
|
|
910
961
|
- It is difficult to identify duplicate instances, in part because, as described above, tools use four different methods for identifying the locations of elements that violate tool rules.
|
|
911
962
|
|
|
912
963
|
To deal with the above problems, you can:
|
|
964
|
+
|
|
913
965
|
- configure `test` acts for tools to exclude tests that you consider duplicative
|
|
914
966
|
- create derivative reports that organize results by defect types rather than by tool
|
|
915
967
|
- take duplication into account when defining scoring rules
|
|
@@ -935,6 +987,7 @@ The files in the `temp` directory are presumed ephemeral and are not tracked by
|
|
|
935
987
|
### Testilo
|
|
936
988
|
|
|
937
989
|
[Testilo](https://www.npmjs.com/package/testilo) is an application that:
|
|
990
|
+
|
|
938
991
|
- converts lists of targets and lists of issues into jobs
|
|
939
992
|
- produces scores and adds them to the raw reports of Testaro
|
|
940
993
|
- produces human-oriented HTML digests from scored reports
|
package/UPGRADES.md
CHANGED
|
@@ -2912,3 +2912,56 @@ Prevents the async cleanup from running after force-kill
|
|
|
2912
2912
|
Works consistently on both macOS and Ubuntu
|
|
2913
2913
|
|
|
2914
2914
|
The above recommendation seems complex. Meanwhile the error message has been suppressed on the basis that context closure is not necessarily an error.
|
|
2915
|
+
|
|
2916
|
+
## Heap memory exhaustion
|
|
2917
|
+
|
|
2918
|
+
Exhaustion of heap memory occurred when PNG images were added to reports by the `shoot0` and `shoot1` tests. This issue was resolved with the storage of PNG images as buffers in temporary files. A recommendation generated by Claude Sonnet 4.5 was to add a safety net to `doTestAct` to remove any very large properties from the report. This has not been implemented yet, partly out of concern that it might hide defects that create unduly large properties, inhibiting the rectification of such defects. The recommended code changes were:
|
|
2919
|
+
|
|
2920
|
+
```javascript
|
|
2921
|
+
// ...existing code...
|
|
2922
|
+
|
|
2923
|
+
// Remove oversized properties from report to prevent serialization failures.
|
|
2924
|
+
const sanitizeReport = report => {
|
|
2925
|
+
const MAX_PROPERTY_SIZE = 10 * 1024 * 1024; // 10MB threshold.
|
|
2926
|
+
|
|
2927
|
+
for (const act of report.acts) {
|
|
2928
|
+
if (act.result) {
|
|
2929
|
+
for (const [key, value] of Object.entries(act.result)) {
|
|
2930
|
+
if (value && typeof value === 'object') {
|
|
2931
|
+
const size = JSON.stringify(value).length;
|
|
2932
|
+
if (size > MAX_PROPERTY_SIZE) {
|
|
2933
|
+
console.log(
|
|
2934
|
+
`WARNING: Removing oversized property '${key}' from ${act.which} result (${Math.round(size / 1024 / 1024)}MB)`
|
|
2935
|
+
);
|
|
2936
|
+
delete act.result[key];
|
|
2937
|
+
}
|
|
2938
|
+
}
|
|
2939
|
+
}
|
|
2940
|
+
}
|
|
2941
|
+
}
|
|
2942
|
+
};
|
|
2943
|
+
|
|
2944
|
+
const doTestAct = async () => {
|
|
2945
|
+
// ...existing code...
|
|
2946
|
+
|
|
2947
|
+
try {
|
|
2948
|
+
const actReport = await require(`../tests/${which}`).reporter(page, report, actIndex, 65);
|
|
2949
|
+
act.data = actReport.data;
|
|
2950
|
+
act.result = actReport.result;
|
|
2951
|
+
|
|
2952
|
+
if (act.data && act.data.prevented) {
|
|
2953
|
+
report.jobData.preventions[which] = act.data.error;
|
|
2954
|
+
}
|
|
2955
|
+
|
|
2956
|
+
await browserClose();
|
|
2957
|
+
|
|
2958
|
+
// Sanitize report before serialization.
|
|
2959
|
+
sanitizeReport(report);
|
|
2960
|
+
|
|
2961
|
+
const reportJSON = JSON.stringify(report);
|
|
2962
|
+
await fs.writeFile(reportPath, reportJSON);
|
|
2963
|
+
process.send('Act completed');
|
|
2964
|
+
}
|
|
2965
|
+
// ...existing code...
|
|
2966
|
+
};
|
|
2967
|
+
```
|
package/package.json
CHANGED
package/procs/doTestAct.js
CHANGED
|
@@ -32,7 +32,8 @@
|
|
|
32
32
|
|
|
33
33
|
// Module to perform file operations.
|
|
34
34
|
const fs = require('fs/promises');
|
|
35
|
-
|
|
35
|
+
//Modules to close and launch browsers.
|
|
36
|
+
const {browserClose, launch} = require(`${__dirname}/../run`);
|
|
36
37
|
// Module to set operating-system constants.
|
|
37
38
|
const os = require('os');
|
|
38
39
|
|
|
@@ -60,7 +61,7 @@ const doTestAct = async () => {
|
|
|
60
61
|
const act = report.acts[actIndex];
|
|
61
62
|
// Get the tool name.
|
|
62
63
|
const {which} = act;
|
|
63
|
-
// Launch a browser, navigate to the URL, and
|
|
64
|
+
// Launch a browser, navigate to the URL, and update the page export of the run module.
|
|
64
65
|
await launch(
|
|
65
66
|
report,
|
|
66
67
|
debug,
|
|
@@ -70,6 +71,8 @@ const doTestAct = async () => {
|
|
|
70
71
|
);
|
|
71
72
|
// If the launch aborted the job:
|
|
72
73
|
if (report.jobData && report.jobData.aborted) {
|
|
74
|
+
// Close any existing browser.
|
|
75
|
+
await browserClose();
|
|
73
76
|
// Save the revised report.
|
|
74
77
|
const reportJSON = JSON.stringify(report);
|
|
75
78
|
await fs.writeFile(reportPath, reportJSON);
|
|
@@ -78,37 +81,33 @@ const doTestAct = async () => {
|
|
|
78
81
|
}
|
|
79
82
|
// Otherwise, i.e. if the launch did not abort the job:
|
|
80
83
|
else {
|
|
81
|
-
// Get the
|
|
84
|
+
// Get the updated page.
|
|
82
85
|
const {page} = require('../run');
|
|
83
86
|
// If it exists:
|
|
84
87
|
if (page) {
|
|
85
88
|
try {
|
|
86
|
-
//
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
//
|
|
92
|
-
|
|
93
|
-
//
|
|
94
|
-
|
|
95
|
-
// Add the data and result to the act.
|
|
96
|
-
act.data = actReport.data;
|
|
97
|
-
act.result = actReport.result;
|
|
98
|
-
// If the tool reported that the page prevented testing:
|
|
99
|
-
if (act.data && act.data.prevented) {
|
|
100
|
-
// Add prevention data to the job data.
|
|
101
|
-
report.jobData.preventions[which] = act.data.error;
|
|
102
|
-
}
|
|
103
|
-
const reportJSON = JSON.stringify(report);
|
|
104
|
-
// Save the revised report.
|
|
105
|
-
await fs.writeFile(reportPath, reportJSON);
|
|
106
|
-
// Send a completion message.
|
|
107
|
-
process.send('Act completed');
|
|
89
|
+
// Make the act reporter perform the specified tests of the tool.
|
|
90
|
+
const actReport = await require(`../tests/${which}`).reporter(page, report, actIndex, 65);
|
|
91
|
+
// Add the data and result to the act.
|
|
92
|
+
act.data = actReport.data;
|
|
93
|
+
act.result = actReport.result;
|
|
94
|
+
// If the tool reported that the page prevented testing:
|
|
95
|
+
if (act.data && act.data.prevented) {
|
|
96
|
+
// Add prevention data to the job data.
|
|
97
|
+
report.jobData.preventions[which] = act.data.error;
|
|
108
98
|
}
|
|
99
|
+
// Close any existing browser.
|
|
100
|
+
await browserClose();
|
|
101
|
+
const reportJSON = JSON.stringify(report);
|
|
102
|
+
// Save the revised report.
|
|
103
|
+
await fs.writeFile(reportPath, reportJSON);
|
|
104
|
+
// Send a completion message.
|
|
105
|
+
process.send('Act completed');
|
|
109
106
|
}
|
|
110
107
|
// If the tool invocation failed:
|
|
111
108
|
catch(error) {
|
|
109
|
+
// Close any existing browser.
|
|
110
|
+
await browserClose();
|
|
112
111
|
// Save the revised report.
|
|
113
112
|
const reportJSON = JSON.stringify(report);
|
|
114
113
|
await fs.writeFile(reportPath, reportJSON);
|
|
@@ -126,8 +125,10 @@ const doTestAct = async () => {
|
|
|
126
125
|
act.data.error = 'No page';
|
|
127
126
|
// Add prevention data to the job data.
|
|
128
127
|
report.jobData.preventions[which] = act.data.error;
|
|
129
|
-
//
|
|
128
|
+
// Close any existing browser.
|
|
129
|
+
await browserClose();
|
|
130
130
|
const reportJSON = JSON.stringify(report);
|
|
131
|
+
// Save the revised report.
|
|
131
132
|
await fs.writeFile(reportPath, reportJSON);
|
|
132
133
|
// Report this.
|
|
133
134
|
const message = 'ERROR: No page';
|
package/procs/shoot.js
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
/*
|
|
2
|
+
© 2025 Jonathan Robert Pool. All rights reserved.
|
|
3
|
+
Licensed under the MIT License. See LICENSE file for details.
|
|
4
|
+
*/
|
|
5
|
+
|
|
6
|
+
/*
|
|
7
|
+
shoot
|
|
8
|
+
Makes and saves as a PNG buffer file a full-page screenshot and returns the file path.
|
|
9
|
+
*/
|
|
10
|
+
|
|
11
|
+
// IMPORTS
|
|
12
|
+
|
|
13
|
+
const fs = require('fs/promises');
|
|
14
|
+
const os = require('os');
|
|
15
|
+
const path = require('path');
|
|
16
|
+
const {PNG} = require('pngjs');
|
|
17
|
+
const {screenShot} = require('./screenShot');
|
|
18
|
+
|
|
19
|
+
// CONSTANTS
|
|
20
|
+
|
|
21
|
+
const tmpDir = os.tmpdir();
|
|
22
|
+
|
|
23
|
+
// FUNCTIONS
|
|
24
|
+
|
|
25
|
+
exports.shoot = async (page, index) => {
|
|
26
|
+
// Make and get a screenshot as a buffer.
|
|
27
|
+
let shot = await screenShot(page);
|
|
28
|
+
// If it succeeded:
|
|
29
|
+
if (shot.length) {
|
|
30
|
+
// Get the screenshot as an object representation of a PNG image.
|
|
31
|
+
let png = PNG.sync.read(shot);
|
|
32
|
+
shot = null;
|
|
33
|
+
const pngBuffer = PNG.sync.write(png);
|
|
34
|
+
png = null;
|
|
35
|
+
// Force garbage collection if available and if --expose-gc was a node option.
|
|
36
|
+
if (global.gc) {
|
|
37
|
+
global.gc();
|
|
38
|
+
}
|
|
39
|
+
const fileName = `testaro-shoot-${index}.png`;
|
|
40
|
+
const pngPath = path.join(tmpDir, fileName);
|
|
41
|
+
// Save the PNG buffer.
|
|
42
|
+
await fs.writeFile(pngPath, pngBuffer);
|
|
43
|
+
// Return the result.
|
|
44
|
+
return pngPath;
|
|
45
|
+
}
|
|
46
|
+
// Otherwise, i.e. if it failed:
|
|
47
|
+
else {
|
|
48
|
+
// Return this.
|
|
49
|
+
return '';
|
|
50
|
+
}
|
|
51
|
+
};
|
package/procs/visChange.js
CHANGED
|
@@ -70,7 +70,7 @@ exports.visChange = async (page, options = {}) => {
|
|
|
70
70
|
}
|
|
71
71
|
catch(error) {
|
|
72
72
|
return {
|
|
73
|
-
|
|
73
|
+
prevented: true,
|
|
74
74
|
error: 'Hovering failed'
|
|
75
75
|
};
|
|
76
76
|
}
|
|
@@ -91,7 +91,7 @@ exports.visChange = async (page, options = {}) => {
|
|
|
91
91
|
const changePercent = 100 * pixelChanges / (width * height);
|
|
92
92
|
// Return this.
|
|
93
93
|
return {
|
|
94
|
-
|
|
94
|
+
prevented: false,
|
|
95
95
|
width,
|
|
96
96
|
height,
|
|
97
97
|
pixelChanges,
|
|
@@ -102,7 +102,6 @@ exports.visChange = async (page, options = {}) => {
|
|
|
102
102
|
else {
|
|
103
103
|
// Return this.
|
|
104
104
|
return {
|
|
105
|
-
success: false,
|
|
106
105
|
prevented: true,
|
|
107
106
|
error: 'Second screenshot failed'
|
|
108
107
|
};
|
|
@@ -112,7 +111,6 @@ exports.visChange = async (page, options = {}) => {
|
|
|
112
111
|
else {
|
|
113
112
|
// Return this.
|
|
114
113
|
return {
|
|
115
|
-
success: false,
|
|
116
114
|
prevented: true,
|
|
117
115
|
error: 'First screenshot failed'
|
|
118
116
|
};
|
package/run.js
CHANGED
|
@@ -259,7 +259,6 @@ const addError = (alsoLog, alsoAbort, report, actIndex, message) => {
|
|
|
259
259
|
act.result.error ??= message;
|
|
260
260
|
if (act.type === 'test') {
|
|
261
261
|
act.data ??= {};
|
|
262
|
-
act.data.success = false;
|
|
263
262
|
act.data.prevented = true;
|
|
264
263
|
act.data.error = message;
|
|
265
264
|
// Add prevention data to the job data.
|
|
@@ -272,7 +271,7 @@ const addError = (alsoLog, alsoAbort, report, actIndex, message) => {
|
|
|
272
271
|
}
|
|
273
272
|
};
|
|
274
273
|
// Closes any current browser.
|
|
275
|
-
const browserClose = async () => {
|
|
274
|
+
const browserClose = exports.browserClose = async () => {
|
|
276
275
|
// If a browser exists:
|
|
277
276
|
if (browser) {
|
|
278
277
|
browserCloseIntentional = true;
|
|
@@ -281,7 +280,8 @@ const browserClose = async () => {
|
|
|
281
280
|
try {
|
|
282
281
|
await context.close();
|
|
283
282
|
}
|
|
284
|
-
catch(error) {
|
|
283
|
+
catch(error) {
|
|
284
|
+
}
|
|
285
285
|
}
|
|
286
286
|
// Close the browser.
|
|
287
287
|
await browser.close();
|
|
@@ -415,7 +415,7 @@ const launch = exports.launch = async (
|
|
|
415
415
|
}
|
|
416
416
|
});
|
|
417
417
|
});
|
|
418
|
-
//
|
|
418
|
+
// Reassign the page variable to a new page (tab) of the context (window).
|
|
419
419
|
page = await browserContext.newPage();
|
|
420
420
|
// Wait until it is stable.
|
|
421
421
|
await page.waitForLoadState('domcontentloaded', {timeout: 5000});
|
|
@@ -771,7 +771,7 @@ const doActs = async (report, opts = {}) => {
|
|
|
771
771
|
actLaunchSpecs[1]
|
|
772
772
|
);
|
|
773
773
|
// If this failed:
|
|
774
|
-
if (page
|
|
774
|
+
if (! page) {
|
|
775
775
|
// Add this to the act.
|
|
776
776
|
act.data ??= {};
|
|
777
777
|
act.data.prevented = true;
|
|
@@ -1513,7 +1513,7 @@ const doActs = async (report, opts = {}) => {
|
|
|
1513
1513
|
// For each browser ID/target URL class:
|
|
1514
1514
|
for (const specString of Object.keys(launchSpecActs)) {
|
|
1515
1515
|
const specs = specString.split('>');
|
|
1516
|
-
//
|
|
1516
|
+
// Replace the browser and navigate to the URL.
|
|
1517
1517
|
await launch(
|
|
1518
1518
|
report,
|
|
1519
1519
|
debug,
|
|
@@ -1521,18 +1521,18 @@ const doActs = async (report, opts = {}) => {
|
|
|
1521
1521
|
specs[0],
|
|
1522
1522
|
specs[1]
|
|
1523
1523
|
);
|
|
1524
|
-
//
|
|
1525
|
-
|
|
1526
|
-
|
|
1527
|
-
|
|
1528
|
-
|
|
1529
|
-
|
|
1530
|
-
|
|
1531
|
-
|
|
1532
|
-
|
|
1533
|
-
|
|
1534
|
-
|
|
1535
|
-
|
|
1524
|
+
// If the launch and navigation succeeded:
|
|
1525
|
+
if (page) {
|
|
1526
|
+
// For each test act in the class:
|
|
1527
|
+
for (const specActIndex of launchSpecActs[specString]) {
|
|
1528
|
+
const act = report.acts[specActIndex];
|
|
1529
|
+
// Initialize the standard result.
|
|
1530
|
+
act.standardResult = {
|
|
1531
|
+
totals: [0, 0, 0, 0],
|
|
1532
|
+
instances: []
|
|
1533
|
+
};
|
|
1534
|
+
// Populate it.
|
|
1535
|
+
standardize(act);
|
|
1536
1536
|
// Add a box ID and a path ID to each of its standard instances if missing.
|
|
1537
1537
|
for (const instance of act.standardResult.instances) {
|
|
1538
1538
|
const elementID = await identify(instance, page);
|
|
@@ -1543,18 +1543,22 @@ const doActs = async (report, opts = {}) => {
|
|
|
1543
1543
|
instance.pathID = elementID ? elementID.pathID : '';
|
|
1544
1544
|
}
|
|
1545
1545
|
};
|
|
1546
|
-
|
|
1547
|
-
|
|
1548
|
-
|
|
1549
|
-
|
|
1550
|
-
|
|
1551
|
-
}
|
|
1552
|
-
}
|
|
1546
|
+
// If the original-format result is not to be included in the report:
|
|
1547
|
+
if (standard === 'only') {
|
|
1548
|
+
// Remove it.
|
|
1549
|
+
delete act.result;
|
|
1550
|
+
}
|
|
1551
|
+
};
|
|
1552
|
+
}
|
|
1553
|
+
// Otherwise, i.e. if the launch or navigation failed:
|
|
1554
|
+
else {
|
|
1555
|
+
console.log(`ERROR: Launch or navigation to standardize ${specString} acts failed`);
|
|
1556
|
+
}
|
|
1553
1557
|
};
|
|
1554
|
-
|
|
1558
|
+
// Close the last browser launched for standardization.
|
|
1559
|
+
await browserClose();
|
|
1560
|
+
console.log('Standardization completed');
|
|
1555
1561
|
}
|
|
1556
|
-
// Close the browser.
|
|
1557
|
-
await browserClose();
|
|
1558
1562
|
// Delete the temporary report file.
|
|
1559
1563
|
await fs.rm(reportPath, {force: true});
|
|
1560
1564
|
return report;
|
package/testaro/motion.js
CHANGED
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
/*
|
|
7
7
|
motion
|
|
8
8
|
This test reports motion in a page by comparing the first and last of the screenshots previously
|
|
9
|
-
made by the
|
|
9
|
+
made by the shoot0 and shoot1 tests.
|
|
10
10
|
|
|
11
11
|
For minimal accessibility, standards require motion to be brief, or else stoppable by the user.
|
|
12
12
|
But stopping motion can be difficult or impossible, and, by the time a user manages to stop
|
|
@@ -21,9 +21,18 @@
|
|
|
21
21
|
|
|
22
22
|
// IMPORTS
|
|
23
23
|
|
|
24
|
-
// Module to
|
|
24
|
+
// Module to process files.
|
|
25
|
+
const fs = require('fs/promises');
|
|
26
|
+
// Module to get operating-system properties.
|
|
27
|
+
const os = require('os');
|
|
28
|
+
// Module to compare screenshots.
|
|
25
29
|
const pixelmatch = require('pixelmatch').default;
|
|
26
|
-
|
|
30
|
+
// Module to parse PNGs.
|
|
31
|
+
const {PNG} = require('pngjs');
|
|
32
|
+
|
|
33
|
+
// CONSTANTS
|
|
34
|
+
|
|
35
|
+
const tmpDir = os.tmpdir();
|
|
27
36
|
|
|
28
37
|
// FUNCTIONS
|
|
29
38
|
|
|
@@ -33,29 +42,32 @@ exports.reporter = async page => {
|
|
|
33
42
|
const data = {};
|
|
34
43
|
const totals = [0, 0, 0, 0];
|
|
35
44
|
const standardInstances = [];
|
|
36
|
-
// Get the
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
+
// Get the screenshot PNG buffers made by the shoot0 and shoot1 tests.
|
|
46
|
+
let shoot0PNGBuffer = await fs.readFile(`${tmpDir}/testaro-shoot-0.png`);
|
|
47
|
+
let shoot1PNGBuffer = await fs.readFile(`${tmpDir}/testaro-shoot-1.png`);
|
|
48
|
+
// Delete the buffers files.
|
|
49
|
+
await fs.unlink(`${tmpDir}/testaro-shoot-0.png`);
|
|
50
|
+
await fs.unlink(`${tmpDir}/testaro-shoot-1.png`);
|
|
51
|
+
// If both buffers exist:
|
|
52
|
+
if (shoot0PNGBuffer && shoot1PNGBuffer) {
|
|
53
|
+
// Parse them into PNG objects.
|
|
54
|
+
let shoot0PNG = PNG.sync.read(shoot0PNGBuffer);
|
|
55
|
+
let shoot1PNG = PNG.sync.read(shoot1PNGBuffer);
|
|
45
56
|
// If their dimensions differ:
|
|
46
|
-
if (width !==
|
|
57
|
+
if (shoot1PNG.width !== shoot0PNG.width || shoot1PNG.height !== shoot0PNG.height) {
|
|
47
58
|
// Report this.
|
|
48
59
|
data.prevented = true;
|
|
49
|
-
data.error = '
|
|
60
|
+
data.error = 'Screenshot dimensions differ';
|
|
50
61
|
}
|
|
51
62
|
// Otherwise, i.e. if their dimensions are identical:
|
|
52
63
|
else {
|
|
64
|
+
const {width, height} = shoot0PNG;
|
|
53
65
|
// Get the count of differing pixels between the shots.
|
|
54
|
-
const pixelChanges = pixelmatch(
|
|
66
|
+
const pixelChanges = pixelmatch(shoot0PNG.data, shoot1PNG.data, null, width, height);
|
|
55
67
|
// Get the ratio of differing to all pixels as a percentage.
|
|
56
|
-
const changePercent = 100 * pixelChanges / (width * height);
|
|
68
|
+
const changePercent = Math.round(100 * pixelChanges / (width * height));
|
|
57
69
|
// Free the memory used by screenshots.
|
|
58
|
-
|
|
70
|
+
shoot0PNG = shoot1PNG = shoot0PNGBuffer = shoot1PNGBuffer = null;
|
|
59
71
|
// If any pixels were changed:
|
|
60
72
|
if (pixelChanges) {
|
|
61
73
|
// Get the ordinal severity from the fractional pixel change.
|
|
@@ -65,26 +77,31 @@ exports.reporter = async page => {
|
|
|
65
77
|
// Get a summary standard instance.
|
|
66
78
|
standardInstances.push({
|
|
67
79
|
ruleID: 'motion',
|
|
68
|
-
what:
|
|
80
|
+
what: `Content moves or changes spontaneously (${changePercent}% of pixels changed)`,
|
|
69
81
|
count: 1,
|
|
70
82
|
ordinalSeverity,
|
|
71
83
|
tagName: 'HTML',
|
|
72
84
|
id: '',
|
|
73
85
|
location: {
|
|
74
|
-
doc: '',
|
|
75
|
-
type: '',
|
|
76
|
-
spec:
|
|
86
|
+
doc: 'dom',
|
|
87
|
+
type: 'box',
|
|
88
|
+
spec: {
|
|
89
|
+
x: 0,
|
|
90
|
+
y: 0,
|
|
91
|
+
width,
|
|
92
|
+
height
|
|
93
|
+
}
|
|
77
94
|
},
|
|
78
|
-
excerpt: ''
|
|
95
|
+
excerpt: '<html>…</html>'
|
|
79
96
|
});
|
|
80
97
|
}
|
|
81
98
|
}
|
|
82
99
|
}
|
|
83
|
-
// Otherwise, i.e. if
|
|
100
|
+
// Otherwise, i.e. if they do not both exist:
|
|
84
101
|
else {
|
|
85
102
|
// Report this.
|
|
86
103
|
data.prevented = true;
|
|
87
|
-
data.error = '
|
|
104
|
+
data.error = 'At least 1 screenshot missing';
|
|
88
105
|
}
|
|
89
106
|
// Return the result.
|
|
90
107
|
return {
|
package/testaro/motionSolo.js
CHANGED
|
@@ -61,7 +61,7 @@ exports.reporter = async page => {
|
|
|
61
61
|
delayBetween: 3000
|
|
62
62
|
});
|
|
63
63
|
// If the screenshots succeeded:
|
|
64
|
-
if (data.
|
|
64
|
+
if (! data.prevented) {
|
|
65
65
|
// If any pixels were changed:
|
|
66
66
|
if (data.pixelChanges) {
|
|
67
67
|
// Get the ordinal severity from the fractional pixel change.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
/*
|
|
2
|
+
© 2025 Jonathan Robert Pool. All rights reserved.
|
|
3
|
+
Licensed under the MIT License. See LICENSE file for details.
|
|
4
|
+
*/
|
|
5
|
+
|
|
6
|
+
/*
|
|
7
|
+
shoot0
|
|
8
|
+
This test makes and saves the first of two screenshots.
|
|
9
|
+
*/
|
|
10
|
+
|
|
11
|
+
// IMPORTS
|
|
12
|
+
|
|
13
|
+
const {shoot} = require('../procs/shoot');
|
|
14
|
+
|
|
15
|
+
// FUNCTIONS
|
|
16
|
+
|
|
17
|
+
exports.reporter = async page => {
|
|
18
|
+
// Make and save the first screenshot.
|
|
19
|
+
const pngPath = await shoot(page, 0);
|
|
20
|
+
// Return the file path or a failure result.
|
|
21
|
+
return {
|
|
22
|
+
data: {
|
|
23
|
+
prevented: ! pngPath
|
|
24
|
+
}
|
|
25
|
+
};
|
|
26
|
+
};
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
/*
|
|
2
|
+
© 2025 Jonathan Robert Pool. All rights reserved.
|
|
3
|
+
Licensed under the MIT License. See LICENSE file for details.
|
|
4
|
+
*/
|
|
5
|
+
|
|
6
|
+
/*
|
|
7
|
+
shoot1
|
|
8
|
+
This test makes and saves the second of two screenshots.
|
|
9
|
+
*/
|
|
10
|
+
|
|
11
|
+
// IMPORTS
|
|
12
|
+
|
|
13
|
+
const {shoot} = require('../procs/shoot');
|
|
14
|
+
|
|
15
|
+
// FUNCTIONS
|
|
16
|
+
|
|
17
|
+
exports.reporter = async page => {
|
|
18
|
+
// Make and save the second screenshot.
|
|
19
|
+
const pngPath = await shoot(page, 1);
|
|
20
|
+
// Return the file path or a failure result.
|
|
21
|
+
return {
|
|
22
|
+
data: {
|
|
23
|
+
prevented: ! pngPath
|
|
24
|
+
}
|
|
25
|
+
};
|
|
26
|
+
};
|
package/tests/testaro.js
CHANGED
|
@@ -42,8 +42,8 @@ const fs = require('fs/promises');
|
|
|
42
42
|
// Metadata of all rules in default execution order.
|
|
43
43
|
const allRules = [
|
|
44
44
|
{
|
|
45
|
-
id: '
|
|
46
|
-
what: 'page screenshot',
|
|
45
|
+
id: 'shoot0',
|
|
46
|
+
what: 'first page screenshot',
|
|
47
47
|
contaminator: false,
|
|
48
48
|
timeOut: 5,
|
|
49
49
|
defaultOn: true
|
|
@@ -350,8 +350,8 @@ const allRules = [
|
|
|
350
350
|
defaultOn: true
|
|
351
351
|
},
|
|
352
352
|
{
|
|
353
|
-
id: '
|
|
354
|
-
what: 'page screenshot',
|
|
353
|
+
id: 'shoot1',
|
|
354
|
+
what: 'second page screenshot',
|
|
355
355
|
contaminator: false,
|
|
356
356
|
timeOut: 5,
|
|
357
357
|
defaultOn: true
|
|
@@ -455,7 +455,7 @@ process.on('unhandledRejection', reason => {
|
|
|
455
455
|
console.error(`ERROR: Unhandled Promise Rejection (${reason})`);
|
|
456
456
|
});
|
|
457
457
|
|
|
458
|
-
//
|
|
458
|
+
// FUNCTIONS
|
|
459
459
|
|
|
460
460
|
// Conducts a JSON-defined test.
|
|
461
461
|
const jsonTest = async (ruleID, ruleArgs) => {
|
|
@@ -503,7 +503,7 @@ exports.reporter = async (page, report, actIndex) => {
|
|
|
503
503
|
rulesInvalid: [],
|
|
504
504
|
ruleTestTimes: {}
|
|
505
505
|
};
|
|
506
|
-
const result =
|
|
506
|
+
const result = {};
|
|
507
507
|
const allRuleIDs = allRules.map(rule => rule.id);
|
|
508
508
|
// If the rule specification is invalid:
|
|
509
509
|
if (! (
|
|
@@ -559,6 +559,7 @@ exports.reporter = async (page, report, actIndex) => {
|
|
|
559
559
|
// Report crashes and disconnections during this test.
|
|
560
560
|
let crashHandler;
|
|
561
561
|
let disconnectHandler;
|
|
562
|
+
// Get the current browser.
|
|
562
563
|
const {browser} = require('../run');
|
|
563
564
|
if (page && ! page.isClosed()) {
|
|
564
565
|
crashHandler = () => {
|
|
@@ -631,7 +632,10 @@ exports.reporter = async (page, report, actIndex) => {
|
|
|
631
632
|
// Prevent a retry of the test.
|
|
632
633
|
testSuccess = true;
|
|
633
634
|
// If testing is to stop after a failure and the page failed the test:
|
|
634
|
-
if (
|
|
635
|
+
if (
|
|
636
|
+
stopOnFail
|
|
637
|
+
&& ruleOrTimeoutReport.totals
|
|
638
|
+
&& ruleOrTimeoutReport.totals.some(total => total)) {
|
|
635
639
|
// Stop testing.
|
|
636
640
|
break;
|
|
637
641
|
}
|
|
@@ -693,9 +697,11 @@ exports.reporter = async (page, report, actIndex) => {
|
|
|
693
697
|
// Clear the error listeners.
|
|
694
698
|
if (page && ! page.isClosed() && crashHandler) {
|
|
695
699
|
page.off('crash', crashHandler);
|
|
700
|
+
crashHandler = null;
|
|
696
701
|
}
|
|
697
702
|
if (browser && disconnectHandler) {
|
|
698
703
|
browser.off('disconnected', disconnectHandler);
|
|
704
|
+
disconnectHandler = null;
|
|
699
705
|
}
|
|
700
706
|
}
|
|
701
707
|
// Otherwise, i.e. if the rule is undefined or doubly defined:
|
|
@@ -704,10 +710,22 @@ exports.reporter = async (page, report, actIndex) => {
|
|
|
704
710
|
data.rulesInvalid.push(rule);
|
|
705
711
|
console.log(`ERROR: Rule ${rule.id} not validly defined`);
|
|
706
712
|
// Clear the crash listener.
|
|
707
|
-
if (page && ! page.isClosed()) {
|
|
713
|
+
if (page && ! page.isClosed() && crashHandler) {
|
|
708
714
|
page.off('crash', crashHandler);
|
|
715
|
+
crashHandler = null;
|
|
716
|
+
}
|
|
717
|
+
if (browser && disconnectHandler) {
|
|
718
|
+
browser.off('disconnected', disconnectHandler);
|
|
719
|
+
disconnectHandler = null;
|
|
720
|
+
}
|
|
721
|
+
}
|
|
722
|
+
// Force a garbage collection.
|
|
723
|
+
try {
|
|
724
|
+
if (global.gc) {
|
|
725
|
+
global.gc();
|
|
709
726
|
}
|
|
710
727
|
}
|
|
728
|
+
catch(error) {}
|
|
711
729
|
};
|
|
712
730
|
// Record the test times in descending order.
|
|
713
731
|
testTimes.sort((a, b) => b[1] - a[1]).forEach(pair => {
|
package/tests/wave.js
CHANGED
|
@@ -75,9 +75,10 @@ exports.reporter = async (page, report, actIndex) => {
|
|
|
75
75
|
});
|
|
76
76
|
// When they arrive:
|
|
77
77
|
response.on('end', async () => {
|
|
78
|
+
let actResult = {};
|
|
78
79
|
// Delete unnecessary properties.
|
|
79
80
|
try {
|
|
80
|
-
|
|
81
|
+
actResult = JSON.parse(rawReport);
|
|
81
82
|
const {categories, statistics} = actResult;
|
|
82
83
|
delete categories.feature;
|
|
83
84
|
delete categories.structure;
|
|
@@ -141,7 +142,7 @@ exports.reporter = async (page, report, actIndex) => {
|
|
|
141
142
|
catch(error) {
|
|
142
143
|
data.prevented = true;
|
|
143
144
|
data.error = error.message;
|
|
144
|
-
resolve(
|
|
145
|
+
resolve(actResult);
|
|
145
146
|
};
|
|
146
147
|
});
|
|
147
148
|
}
|
package/testaro/shoot.js
DELETED
|
@@ -1,57 +0,0 @@
|
|
|
1
|
-
/*
|
|
2
|
-
© 2025 Jonathan Robert Pool. All rights reserved.
|
|
3
|
-
Licensed under the MIT License. See LICENSE file for details.
|
|
4
|
-
*/
|
|
5
|
-
|
|
6
|
-
/*
|
|
7
|
-
shoot
|
|
8
|
-
This test makes a full-page screenshot and returns it as a PNG object.
|
|
9
|
-
*/
|
|
10
|
-
|
|
11
|
-
// IMPORTS
|
|
12
|
-
|
|
13
|
-
const {PNG} = require('pngjs');
|
|
14
|
-
const {screenShot} = require('../procs/screenShot');
|
|
15
|
-
const {result} = require('../tests/testaro');
|
|
16
|
-
|
|
17
|
-
// FUNCTIONS
|
|
18
|
-
|
|
19
|
-
exports.reporter = async page => {
|
|
20
|
-
// Get the result of the previous iterations of this test, if any.
|
|
21
|
-
const shootResult = result ? result.shoot || {} : {};
|
|
22
|
-
// If any previous failure occurred:
|
|
23
|
-
if (shootResult && shootResult.success === false) {
|
|
24
|
-
// Return a failure without making a screenshot.
|
|
25
|
-
return {
|
|
26
|
-
success: false,
|
|
27
|
-
prevented: true,
|
|
28
|
-
error: 'At least 1 screenshot failed'
|
|
29
|
-
};
|
|
30
|
-
}
|
|
31
|
-
// Otherwise, make and get a screenshot.
|
|
32
|
-
const shot = await screenShot(page);
|
|
33
|
-
// If it succeeded:
|
|
34
|
-
if (shot.length) {
|
|
35
|
-
// Get the screenshot as an object representation of a PNG image.
|
|
36
|
-
const png = PNG.sync.read(shot);
|
|
37
|
-
shootResult.pngs ??= [];
|
|
38
|
-
const {pngs} = shootResult;
|
|
39
|
-
// Add it to the test result.
|
|
40
|
-
pngs.push(png);
|
|
41
|
-
// Return the updated result.
|
|
42
|
-
return {
|
|
43
|
-
success: true,
|
|
44
|
-
prevented: false,
|
|
45
|
-
pngs
|
|
46
|
-
};
|
|
47
|
-
}
|
|
48
|
-
// Otherwise, i.e. if it failed:
|
|
49
|
-
else {
|
|
50
|
-
// Return this.
|
|
51
|
-
return {
|
|
52
|
-
success: false,
|
|
53
|
-
prevented: true,
|
|
54
|
-
error: 'Screenshot failed'
|
|
55
|
-
};
|
|
56
|
-
}
|
|
57
|
-
};
|