testaro 60.4.1 → 60.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -1,11 +1,7 @@
1
- /*
2
- © 2025 Jonathan Robert Pool. All rights reserved.
3
- Licensed under the MIT License. See LICENSE file for details.
4
- */
5
-
6
1
  # Testaro Agent Guidelines
7
2
 
8
3
  ## Commands
4
+
9
5
  - **Run all tests**: `npm run tests`
10
6
  - **Run single test**: `npm test <testname>` (e.g., `npm test hover`)
11
7
  - **Run job**: `npm run run [jobID]`
@@ -14,6 +10,7 @@
14
10
  - **Lint**: `npx eslint <file>` (follows `.eslintrc.json`)
15
11
 
16
12
  ## Architecture
13
+
17
14
  - **Main dirs**: `/tests` (tool test definitions), `/procs` (shared procedures), `/validation` (test validators), root (core modules)
18
15
  - **Key files**: `run.js` (main executor), `actSpecs.js` (act specifications), `call.js` (CLI entry), `tests/testaro.js` (Testaro tool rules)
19
16
  - **Tools**: Integrates 11 a11y tools (Axe, Alfa, IBM Checker, QualWeb, ASLint, WAVE, WallyAX, Ed11y, HTML CodeSniffer, Nu Validator, Testaro)
@@ -21,8 +18,14 @@
21
18
  - **Env vars**: Required for WallyAX (`WAX_KEY`), WAVE (`WAVE_KEY`); optional `DEBUG`, `WAITS`, `JOBDIR`, `REPORTDIR`, `AGENT`
22
19
 
23
20
  ## Code Style
21
+
24
22
  - **ESLint**: 2-space indent, single quotes, semicolons required, Stroustrup brace style, no use-before-define
25
23
  - **Imports**: CommonJS (`require`/`module.exports`), not ES modules
26
24
  - **Naming**: camelCase for vars/functions, descriptive names, rule IDs in lowercase
27
25
  - **Error handling**: Try-catch blocks, report failures via `prevented: true` in results
28
26
  - **Comments**: Explain complex logic, but keep concise
27
+
28
+ ## License
29
+
30
+ © 2025 Jonathan Robert Pool. All rights reserved.
31
+ Licensed under the MIT License. See LICENSE file for details.
package/README.md CHANGED
@@ -7,10 +7,12 @@ Ensemble testing for web accessibility
7
7
  Testaro is an application for automated web accessibility testing.
8
8
 
9
9
  The purposes of Testaro are to:
10
+
10
11
  - provide programmatic access to accessibility tests defined by several tools
11
12
  - facilitate the integration of the reports of the tools into a unified report
12
13
 
13
14
  Testaro is described in two papers:
15
+
14
16
  - [How to run a thousand accessibility tests](https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3)
15
17
  - [Testaro: Efficient Ensemble Testing for Web Accessibility](https://arxiv.org/abs/2309.10167)
16
18
 
@@ -23,6 +25,7 @@ Testaro can be installed under a MacOS, Windows, Debian, or Ubuntu operating sys
23
25
  Testaro accepts _jobs_, performs them, and returns _reports_.
24
26
 
25
27
  Other software, located on the same or a different host, can make use of Testaro, performing functions such as:
28
+
26
29
  - Job preparation
27
30
  - Converting user specifications into jobs
28
31
  - Job scheduling
@@ -41,12 +44,14 @@ One software product that performs some such functions is [Testilo](https://www.
41
44
  ## Dependencies
42
45
 
43
46
  Testaro uses:
47
+
44
48
  - [Playwright](https://playwright.dev/) to launch browsers, perform user actions in them, and perform tests
45
49
  - [playwright-extra](https://www.npmjs.com/package/playwright-extra) and [puppeteer-extra-plugin-stealth](https://www.npmjs.com/package/puppeteer-extra-plugin-stealth) to make a Playwright-controlled browser more indistinguishable from a human-operated browser and thus make their requests more likely to succeed
46
50
  - [playwright-dompath](https://www.npmjs.com/package/playwright-dompath) to retrieve XPaths of elements
47
51
  - [pixelmatch](https://www.npmjs.com/package/pixelmatch) to measure motion
48
52
 
49
53
  Testaro performs tests of these _tools_:
54
+
50
55
  - [Accessibility Checker](https://www.npmjs.com/package/accessibility-checker) (IBM)
51
56
  - [Alfa](https://alfa.siteimprove.com/) (Siteimprove)
52
57
  - [ASLint](https://www.npmjs.com/package/@essentialaccessibility/aslint) (eSSENTIAL Accessibility)
@@ -65,7 +70,7 @@ Some of the tests of Testaro are designed to act as approximate alternatives to
65
70
 
66
71
  Each tool accessed with Testaro defines _rules_ and tests _targets_ for compliance with its rules. Testilo has classified the rules into _issues_ and deprecated some rules as poorly implemented. If the deprecated rules are excluded, the counts are:
67
72
 
68
- ```
73
+ ```text
69
74
  Accessibility Checker: 93
70
75
  Alfa: 64
71
76
  ASLint: 129
@@ -85,6 +90,7 @@ This tabulation may not be exact, because some of the tools are under active dev
85
90
  ## Code organization
86
91
 
87
92
  The main directories containing code files are:
93
+
88
94
  - package root: main code files
89
95
  - `tests`: files containing the code defining particular tests
90
96
  - `procs`: shared procedures
@@ -134,9 +140,9 @@ All of the tests that Testaro can perform are free of cost, except those perform
134
140
 
135
141
  ## Jobs
136
142
 
137
- A _job_ is an object that specifies what Testaro is to do. As Testaro performs a job, Testaro reports results by adding data to the job.
143
+ A _job_ is an object that specifies what Testaro is to do. As Testaro performs a job, Testaro reports results by adding data to the job and making that enhanced object available as a _report_.
138
144
 
139
- ### Example
145
+ ### Example of a job
140
146
 
141
147
  Here is an example of a job:
142
148
 
@@ -201,6 +207,7 @@ This job tells Testaro to perform two _acts_. One performs one test of the Axe t
201
207
  Each act includes a `launch` property with a default value. That instructs Testaro, before performing those tests, to launch a new Webkit browser, open a context (window) with some properties of an iPhone 8 and without a reduced-motion setting, create a page (tab), and navigate to a particular page of the `abccorp.com` website.
202
208
 
203
209
  Job properties:
210
+
204
211
  - `id`: a string uniquely identifying the job.
205
212
  - `what`: a description of the job.
206
213
  - `strict`: `true` or `false`, indicating whether _substantive redirections_ should be treated as failures. These are redirections that do more than add or subtract a final slash.
@@ -216,13 +223,16 @@ Job properties:
216
223
 
217
224
  ## Acts
218
225
 
219
- ### Introduction
226
+ As shown above, `acts` is a property of a job and has an array value. Each item in the array is an object that specifies an _act_.
227
+
228
+ ### Introduction to acts
220
229
 
221
230
  Each act object has a `type` property and optionally has a `name` property (used in branching, described below). It must or may have other properties, depending on the value of `type`.
222
231
 
223
232
  ### Act types
224
233
 
225
234
  The acts can tell Testaro to perform any of:
235
+
226
236
  - _navigations_ (browser launches, visits to URLs, waits for page conditions, etc.)
227
237
  - _moves_ (clicks, text inputs, hovers, etc.)
228
238
  - _alterations_ (changes to the page)
@@ -294,9 +304,9 @@ This act checks the result of the previous act to determine whether its `result.
294
304
 
295
305
  A `next` act can use a `next` property instead of a `jump` property. The value of the `next` property is an act name. It tells Testaro to continue performing acts starting with the act having that value as the value of its `name` property.
296
306
 
297
- #### Tests
307
+ #### Tools
298
308
 
299
- ##### Introduction
309
+ ##### Introduction to tools
300
310
 
301
311
  An act of type `test` performs the tests of a tool and reports a result. The result may indicate that a page passes or fails requirements. Typically, accessibility tests report successes and failures. But a test in Testaro is defined less restrictively, so it can report any result. As one example, the Testaro `elements` test reports facts about certain elements on a page, without asserting that those facts are successes or failures.
302
312
 
@@ -305,6 +315,7 @@ The `which` property of a `test` act identifies a tool, such as `alfa` or `testa
305
315
  ##### Configuration
306
316
 
307
317
  Every tool invoked by Testaro must have:
318
+
308
319
  - a property in the `tests` object defined in the `run.js` file, where the property name is the ID representing the tool and the value is the name of the tool
309
320
  - a `.js` file, defining the operation of the tool, in the `tests` directory, whose name base is the name of the tool
310
321
 
@@ -330,7 +341,7 @@ When you include a `rules` property, you limit the tests of the tool that are pe
330
341
 
331
342
  The `nuVal`, `qualWeb`, and `testaro` tools require specific formats for the `rules` property. Those formats are described below in the sections about those tools.
332
343
 
333
- ##### Examples
344
+ ##### Examples of test acts
334
345
 
335
346
  An example of a `test` act is:
336
347
 
@@ -393,6 +404,7 @@ The first item in each array is an identifier of a property of the act. The item
393
404
  If there is only 1 item in an array, it states the expectation that the specified property does not exist. Otherwise, there are 3 items in the array.
394
405
 
395
406
  The second item in each array, if there are 3 items, is an operator, drawn from:
407
+
396
408
  - `<`: less than
397
409
  - `=`: equal to
398
410
  - `>`: greater than
@@ -404,9 +416,7 @@ The third item in each array, if there are 3 items in the array, is the criterio
404
416
 
405
417
  A typical use for an `expect` property is checking the correctness of a Testaro test. Thus, the validation jobs in the `validation/tests/jobs` directory all contain `test` acts with `expect` properties. See the “Validation” section below.
406
418
 
407
- When a `test` act has an `expect` property, the result for that act has an `expectations` property reporting whether the expectations were satisfied. The value of `expectations` is an array of objects, one object per expectation. Each object includes a `property` property identifying the expectation, and a `passed` property with `true` or `false` value reporting whether the expectation was satisfied. If applicable, it also has other properties identifying what was expected and what was actually reported.
408
-
409
- ### Tools
419
+ ### Tool details
410
420
 
411
421
  The tools whose tests Testaro performs have particularities described below.
412
422
 
@@ -467,7 +477,7 @@ In `node_modules/accessibility-checker/lib/reporters/ACReporterJSON.js`, add the
467
477
  results.label = results.label.replace(/:/g, '-');
468
478
  ```
469
479
 
470
- These changes were proposed as pull requests 1333 and 1334 (https://github.com/IBMa/equal-access/pulls).
480
+ These changes were proposed as [pull requests 1333 and 1334](https://github.com/IBMa/equal-access/pulls).
471
481
 
472
482
  The `ibm` tool is one of two tools (`testaro` is the other) with a `withItems` property. If you set `withItems` to `false`, the result includes the counts of “violations” and “recommendations”, but no information about the rules that gave rise to them.
473
483
 
@@ -490,6 +500,7 @@ QualWeb allows specification of rules for 3 modules: `act-rules`, `wcag-techniqu
490
500
  ```
491
501
 
492
502
  In that format:
503
+
493
504
  - Replace `mod` with `act`, `wcag`, or `best`.
494
505
  - Replace `m`, `n`, `o`, `p`, etc. with the 0 or more integers that identify rules.
495
506
 
@@ -505,23 +516,23 @@ The target can be provided to QualWeb either as an existing page or as a URL. Ex
505
516
 
506
517
  #### Testaro
507
518
 
508
- If you do not specify rules when using the `testaro` tool, Testaro will test for the rules listed in the `evalRules` object of the `tests/testaro.js` file.
519
+ The rules that Testaro can test for are implemented in files within the `testaro` directory.
509
520
 
510
- The `rules` argument for a `testaro` test act is an array whose first item is either `'y'` or `'n'` and whose remaining items are rule IDs. If `'y'`, then only the specified rules’ tests are performed. If `'n'`, then all the evaluative tests are performed, **except** for the specified rules.
521
+ If you do not specify rules when using the `testaro` tool, Testaro will test for its default rules, namely the rules that have `true` values on the `defaultOn` property in the `allRules` array defined in the `tests/testaro.js` file. It will test for these rules in the order in which they appear in the array.
522
+
523
+ The optional `rules` argument for a `testaro` test act is an array whose first item is either `'y'` or `'n'` and whose remaining items are rule IDs. If `'y'`, then only the specified rules’ tests are performed. If `'n'`, then all the default rules are tested for, **except** for the specified rules.
511
524
 
512
525
  The `testaro` tool (like the `ibm` tool) has a `withItems` property. If you set it to `false`, the `standardResult` object will contain an `instances` property with summaries that identify issues and instance counts. If you set it to `true`, some of the instances will be itemized.
513
526
 
514
527
  Unlike any other tool, the `testaro` tool requires a `stopOnFail` property, which specifies whether a failure to conform to any rule (i.e. any value of `totals` other than `[0, 0, 0, 0]`) should terminate the execution of tests for the remaining rules.
515
528
 
516
- Warnings in the `testaro/hover.js`, `testaro/motion.js`, and `procs/visChange.js` files advise you to avoid launching particular browser types for the performance of particular Testaro tests.
517
-
518
- Several Testaro tests make use of the `init()` function in the `procs/testaro` module. That function samples elements if the population of elements to be tested is larger than 100. The purpose is to achieve reasonable performance. The sampling overweights elements near the beginning of a page, because of the tendency of that location to have important and atypical elements.
529
+ Some Testaro tests make use of the `init()` function in the `procs/testaro` module. That function samples elements if the population of elements to be tested is larger than 100. The purpose is to achieve reasonable performance. The sampling overweights elements near the beginning of a page, because of the tendency of that location to have important and atypical elements.
519
530
 
520
- You can add custom rules to the rules of any tool. Testaro provides a template, `data/template.js`, for the definition of a rule to be added. Once you have created a copy of the template with revisions, you can move the copy into the `testaro` directory and add an entry for your custom rule to the `evalRules` object in the `tests/testaro.js` file. Then your custom rule will act as a Testaro rule. Some `testaro` rules are simple enough to be fully specified in JSON files. You can use any of those as a template if you want to create a sufficiently simple custom rule, namely a rule whose prohibited elements are all and only the elements matching a CSS selector. More details about rule creation are in the `CONTRIBUTING.md` file.
531
+ You can add custom rules to the rules of any tool. Testaro provides a template, `data/template.js`, for the definition of a rule to be added. Once you have created a copy of the template with revisions, you can move the copy into the `testaro` directory and add an entry for your custom rule to the `allRules` object in the `tests/testaro.js` file. Then your custom rule will act as a Testaro rule. Some `testaro` rules are simple enough to be fully specified in JSON files. You can use any of those as a template if you want to create a sufficiently simple custom rule, namely a rule whose prohibited elements are all and only the elements matching a CSS selector. More details about rule creation are in the `CONTRIBUTING.md` file.
521
532
 
522
533
  #### WallyAX
523
534
 
524
- If a `wax` test act is included in the job, an environment variable named `WAX_KEY` must exist, with your WallyAX API key as its value. You can request it from [WallyAX](mailto:technology@wallyax.com).
535
+ If a `wax` test act is included in the job, an environment variable named `WAX_KEY` must exist, with your WallyAX API key as its value. You can obtain it from [WallyAX](https://account.wallyax.com/?ref_app=Developer&app_type=npm).
525
536
 
526
537
  The `wax` tool imposes a limit on the size of a page to be tested. If the page exceeds the limit, Testaro treats the page as preventing `wax` from performing its tests. The limit is less than 500,000 characters.
527
538
 
@@ -535,8 +546,6 @@ If you want the stand-alone API to perform the tests, you need to have that API
535
546
 
536
547
  ### Browser types
537
548
 
538
- The warning comments in the `testaro/hover.js` and `testaro/motion.js` files state that those tests operate correctly only with the `webkit` browser type. The warning comment in the `testaro/focInd.js` file states that that test operates incorrectly with the `firefox` browser type.
539
-
540
549
  When you want to run some tests of a tool with one browser type and other tests of the same tool with another browser type, you can do so by splitting the rules into two test acts. For example, one test act can specify the rules as
541
550
 
542
551
  ```javascript
@@ -553,7 +562,7 @@ Together, they get all tests of the tool performed. Before each test act, you ca
553
562
 
554
563
  ### `actSpecs` file
555
564
 
556
- #### Introduction
565
+ #### Introduction to the `actSpecs` file
557
566
 
558
567
  The `actSpecs.js` file contains rules governing acts. The rules determine whether an act is valid.
559
568
 
@@ -588,12 +597,14 @@ The rule is an array with two elements: a string ('Click a link') describing the
588
597
  The requirement `which: [true, 'string', 'hasLength', 'substring of the link text']` specifies what is required for the `which` property of a `link`-type act. The requirement is an array.
589
598
 
590
599
  In most cases, the array has length 4:
591
- - 0. Is the property (here `which`) required (`true` or `false`)? The value `true` here means that every `link`-type act **must** contain a `which` property.
592
- - 1. What format must the property value have (`'string'`, `'array'`, `'boolean'`, `'number'`, or `'object'`)?
593
- - 2. What other validity criterion applies (if any)? (Empty string if none.) The `hasLength` criterion means that the string must be at least 1 character long.
594
- - 3. Description of the property. In this example, the description says that the value of `which` must be a substring of the text content of the link that is to be clicked. Thus, a `link` act tells Testaro to find the first link whose text content has this substring and click it.
600
+
601
+ - Item 0. Is the property (here `which`) required (`true` or `false`)? The value `true` here means that every `link`-type act **must** contain a `which` property.
602
+ - Item 1. What format must the property value have (`'string'`, `'array'`, `'boolean'`, `'number'`, or `'object'`)?
603
+ - Item 2. What other validity criterion applies (if any)? (Empty string if none.) The `hasLength` criterion means that the string must be at least 1 character long.
604
+ - Item 3. Description of the property. In this example, the description says that the value of `which` must be a substring of the text content of the link that is to be clicked. Thus, a `link` act tells Testaro to find the first link whose text content has this substring and click it.
595
605
 
596
606
  The validity criterion named in item 2 may be any of these:
607
+
597
608
  - `'hasLength'`: is not a blank string
598
609
  - `'isURL`': is a string starting with `http`, `https`, or `file`, then `://`, then ending with 1 or more non-whitespace characters
599
610
  - `'isBrowserType'`: is `'chromium'`, `'firefox'`, or `'webkit'`
@@ -605,45 +616,79 @@ The validity criterion named in item 2 may be any of these:
605
616
 
606
617
  ## Reports
607
618
 
608
- ### Introduction
619
+ Any Testaro job produces a report, which is a copy of the job with additional data produced by Testaro as it performed the job. Like a job, a report is an object that can be serialized to JSON for file storage and network transmission.
620
+
621
+ ### Job-level data
622
+
623
+ The data that Testaro adds to a job to create a report include job-level data: data describing the how the job as a whole was performed. Examples: when it was completed and how long it took to run.
624
+
625
+ Properties that were in a job when it was given to Testaro remain unchanged in the report. New data produced by Testaro during its performance of a job are inserted into a new `jobData` property in the report.
609
626
 
610
- Each tool produces a _tool report_ of the results of its tests. Testaro prunes the tool reports for brevity, removing content that is judged unlikely to be useful. Testaro then appends each tool report to the test act that invoked the tool.
627
+ ### Act-level data
611
628
 
612
- Testaro also generates some data about the job and adds those data to the job, in a `jobData` property.
629
+ Testaro also adds act-level data to each job. The new act-level data are properties added to each `act` object.
613
630
 
614
- ### Contents
631
+ #### Act-level data from `test` acts
615
632
 
616
- A report discloses:
617
- - results of tests conducted by tools
618
- - process data, including statistics on:
619
- - latency (how long a time each tool takes to run its tests)
620
- - test prevention (the failure of tools to run on particular targets)
621
- - logging (browser messaging, including about document errors, during testing)
633
+ In a `test` act, one of the 11 tools performs tests and reports the results. Testaro manages this performance with the `reporter` function of a file located in the `tests` directory. Each tool has a corresponding file, such as `alfa.js` for the `alfa` tool.
622
634
 
623
- ### Formats
635
+ The `reporter` function returns an object with this structure:
624
636
 
625
- #### Tool-report formats
637
+ ```js
638
+ {
639
+ data: {
640
+ prevented: boolean (whether the tool failed to perform its tests on the page),
641
+ error: string (if `prevented` is `true`, a description of the error)
642
+ … (other tool-specific data)
643
+ },
644
+ result: object (the results of the tests performed by the tool)
645
+ }
646
+ ```
647
+
648
+ On the completion of a job, Testaro has added these properties to each `test` act to produce a report:
649
+
650
+ - `what` (string): the name of the tool
651
+ - `actualURL` (string): the URL of the visited page, after any redirections
652
+ - `startTime` (string): when the tool was started
653
+ - `endTime` (string): when the tool reported its results
654
+ - `data` (object): other tool-specific data:
655
+ - `prevented` (boolean): whether the tool failed to perform its tests
656
+ - `error` (string): a description of the failure, if any
657
+ - other tool-specific data, if any
658
+
659
+ Testaro may also add these properties to any `test` act:
660
+
661
+ - `expectations` (object): the results of validations specified by the act in `expect` properties
662
+ - `expectationFailures` (number): the count of failed validations
663
+
664
+ The value of `expectations` is an array of objects, one object per expectation. Each object includes a `property` property identifying the expectation, and a `passed` property with `true` or `false` value reporting whether the expectation was satisfied. If applicable, it also has other properties identifying what was expected and what was actually reported.
665
+
666
+ Testaro also adds one or both of these properties to each `test` act:
626
667
 
627
- The tools listed above as dependencies write their tool reports in various formats. They differ in how they organize multiple instances of the same problem, how they classify severity and certainty, how they point to the locations of problems, how they name problems, etc.
668
+ - `result` (object): the results of the tests performed by the tool, in the native format of the tool
669
+ - `standardResult` (object): the `result` property converted to a Testaro-standard structure
628
670
 
629
- A Testaro report can include, for each tool, either or both of these properties:
630
- - `result`: the result in the native tool format.
631
- - `standardResult`: the result in a standard format identical for all tools.
671
+ A job specifies whether the report should include, for each `test` act, the `result` property, the `standardResult` property, or both.
672
+
673
+ #### Act-level data from `testaro` test acts
674
+
675
+ Each Testaro rule module exports a `reporter` function, which returns an object with `data`, `totals`, and `standardInstances` properties. Testaro combines the values of those properties with the corresponding values of the same properties from the other `testaro` tests to create the `data` and `result` properties added to `testaro` test acts.
632
676
 
633
677
  #### Standard result
634
678
 
635
679
  ##### Properties
636
680
 
637
- The standard result includes three properties:
638
- - `prevented`: a boolean (`true` or `false`) value, stating whether the page prevented the tool from performing its tests.
639
- - `totals`: an array of numbers representing how many instances of rule violations at each level of severity the tool reported. There are 4 ordinal severity levels. For example, the array `[3, 0, 14, 10]` would report that there were 3 violations at level 0, 0 at level 1, 14 at level 2, and 10 at level 3.
640
- - `instances`: an array of objects describing the rule violations. An instance can describe a single violation, usually by one element in the page, or can summarize multiple violations of the same rule.
681
+ The `standardResult` property, when added to a `test` act, includes three properties:
682
+
683
+ - `prevented` (boolean): whether the tool failed to perform its tests on the page.
684
+ - `totals` (array of numbers): counts of rule violations at 4 ordinal severity levels. For example, the array `[3, 0, 14, 10]` reports that there were 3 violations at level 0, 0 at level 1, 14 at level 2, and 10 at level 3.
685
+ - `instances` (array of objects): descriptions of rule violations reported. An instance can describe a single violation, usually by one element in the page, or can summarize multiple violations of the same rule.
641
686
 
642
687
  If the value of `prevented` is `true`, the standard result also includes an `error` property describing the reason for the failure.
643
688
 
644
689
  ##### Instances
645
690
 
646
- Here is an example of a standard instance:
691
+ Here is an example of an instance in a standard result:
647
692
 
648
693
  ```javascript
649
694
  {
@@ -664,9 +709,10 @@ Here is an example of a standard instance:
664
709
  }
665
710
  ```
666
711
 
667
- This instance describes a violation of a rule named `rule01` by a `button` element.
712
+ This instance says that a `button` element violates a rule named `rule01`.
668
713
 
669
714
  The element has no `id` attribute to distinguish it from other `button` elements, but the tool describes its location. This tool uses an XPath to do that. Tools use various methods for location description, namely:
715
+
670
716
  - `line` (line number in the code of the page): Nu Html Checker
671
717
  - `selector` (CSS selector): Axe, QualWeb, WAVE
672
718
  - `xpath`: Alfa, ASLint, Equal Access
@@ -678,12 +724,14 @@ The tool also reproduces an excerpt of the element code.
678
724
  ##### Element identification
679
725
 
680
726
  While the above properties can help you find the offending element, Testaro makes this easier by adding, where practical, two standard element identifiers to each standard instance:
727
+
681
728
  - `boxID`: a compact representation of the x, y, width, and height of the element bounding box, if the element can be identified and is visible.
682
729
  - `pathID`: the XPath of the element, if the element can be identified.
683
730
 
684
731
  These standard identifiers can help you determine whether violations reported by different tools belong to the same element or different elements. The `boxID` property can also support the making of images of the violating elements.
685
732
 
686
733
  Some tools limit the efficacy of the current algorithm for standard identifiers:
734
+
687
735
  - HTML CodeSniffer does not report element locations, and the reported code excerpts exclude all text content.
688
736
  - Nu Html Checker reports line and column boundaries of element start tags and truncates element text content in reported code excerpts.
689
737
 
@@ -692,6 +740,7 @@ Testing can change the pages being tested, and such changes can cause a particul
692
740
  ##### Standardization configuration
693
741
 
694
742
  Each job specifies how Testaro is to handle report standardization. A job contains a `standard` property, with one of the following values to determine which results the report will include:
743
+
695
744
  - `'also'`: original and standard.
696
745
  - `'only'`: standard only.
697
746
  - `'no'`: original only.
@@ -701,12 +750,13 @@ If a tool has the option to be used without itemization and is being so used, th
701
750
  ##### Standardization opinionation
702
751
 
703
752
  This standard format reflects some judgments. For example:
753
+
704
754
  - The `ordinalSeverity` property of an instance involves interpretation. Tools may report severity, certainty, priority, or some combination of those. They may use ordinal or metric quantifications. If they quantify ordinally, their scales may have more or fewer than 4 ranks. Testaro coerces each tool’s severity, certainty, and/or priority classification into a 4-rank ordinal classification. This classification is deemed to express the most common pattern among the tools.
705
755
  - The `tagName` property of an instance may not always be obvious, because in some cases the rule being tested for requires a relationship among more than one element (e.g., “An X element may not have a Y element as its parent”).
706
756
  - The `ruleID` property of an instance is a matching rule if the tool issues a message but no rule identifier for each instance. The `nuVal` tool does this. In this case, Testaro is classifying the messages into rules.
707
757
  - The `ruleID` property of an instance may reclassify tool rules. For example, if a tool rule covers multiple situations that are dissimilar, that rule may be split into multiple rules with distinct `ruleID` properties.
708
758
 
709
- You are not dependent on the judgments incorporated into the standard format, because Testaro can give you the original reports from the tools.
759
+ You are not dependent on the judgments incorporated into the standard format, because Testaro can give you the original reports from the tools as the `result` property of a `test` act.
710
760
 
711
761
  The standard format does not express opinions on issue classification. A rule ID identifies something deemed to be an issue by a tool. Useful reporting from multi-tool testing still requires the classification of tool **rules** into **issues**. If tool `A` has `alt-incomplete` as a rule ID and tool `B` has `image_alt_stub` as a rule ID, Testaro does not decide whether those are really the same issue or different issues. That decision belongs to you. The standardization of tool reports by Testaro eliminates some of the drudgery in issue classification, but not any of the judgment required for issue classification.
712
762
 
@@ -755,7 +805,7 @@ In watch mode, Testaro periodically checks for a job to run and, when a job is o
755
805
 
756
806
  Testaro can watch for a job in a directory of the filesystem where Testaro or your application is located, with the `dirWatch` function.
757
807
 
758
- #### By a module
808
+ #### Directory watching by a module
759
809
 
760
810
  ```javaScript
761
811
  const {dirWatch} = require('testaro/dirWatch');
@@ -768,7 +818,7 @@ Testaro checks for jobs in the `todo` subdirectory of `JOBDIR`. When it has perf
768
818
 
769
819
  Testaro creates a report for each job and saves the report in the `raw` subdirectory of `REPORTDIR`.
770
820
 
771
- #### By a user
821
+ #### Directory watching by a user
772
822
 
773
823
  ```javaScript
774
824
  node call dirWatch true 300
@@ -809,7 +859,7 @@ Network watching can be repeated or 1-job. 1-job watching stops after 1 job has
809
859
 
810
860
  After checking all the URLs in succession without getting a job from any of them, Testaro waits for the prescribed time before continuing to check.
811
861
 
812
- #### By a module
862
+ #### Network watching by a module
813
863
 
814
864
  ```javaScript
815
865
  const {netWatch} = require('testaro/netWatch');
@@ -820,7 +870,7 @@ In this example, a module of your application asks Testaro to check the servers
820
870
 
821
871
  The third argument specifies whether Testaro should be certificate-tolerant. A `true` value makes Testaro accept SSL certificates that fail verification against a list of certificate authorities. This allows testing of `https` targets that, for example, use self-signed certificates. If the third argument is omitted, the default for that argument is implemented. The default is `true`.
822
872
 
823
- #### By a user
873
+ #### Network watching by a user
824
874
 
825
875
  ```javaScript
826
876
  node call netWatch true 300 true
@@ -904,12 +954,14 @@ Some targets prohibit the execution of alien scripts unless the client can demon
904
954
  ### Tool duplicativity
905
955
 
906
956
  Tools sometimes do redundant testing, in that two or more tools test for the same defects, although such duplications are not necessarily perfect. This fact creates problems:
957
+
907
958
  - One cannot be confident in excluding some tests of some tools on the assumption that they perfectly duplicate tests of other tools.
908
959
  - The Testaro report from a job documents each tool’s results separately, so a single defect may be documented in multiple locations within the report, making the direct consumption of the report inefficient.
909
960
  - An effort to aggregate the results into a single score may distort the scores by inflating the weights of defects that happen to be discovered by multiple tools.
910
961
  - It is difficult to identify duplicate instances, in part because, as described above, tools use four different methods for identifying the locations of elements that violate tool rules.
911
962
 
912
963
  To deal with the above problems, you can:
964
+
913
965
  - configure `test` acts for tools to exclude tests that you consider duplicative
914
966
  - create derivative reports that organize results by defect types rather than by tool
915
967
  - take duplication into account when defining scoring rules
@@ -935,6 +987,7 @@ The files in the `temp` directory are presumed ephemeral and are not tracked by
935
987
  ### Testilo
936
988
 
937
989
  [Testilo](https://www.npmjs.com/package/testilo) is an application that:
990
+
938
991
  - converts lists of targets and lists of issues into jobs
939
992
  - produces scores and adds them to the raw reports of Testaro
940
993
  - produces human-oriented HTML digests from scored reports
package/UPGRADES.md CHANGED
@@ -2912,3 +2912,56 @@ Prevents the async cleanup from running after force-kill
2912
2912
  Works consistently on both macOS and Ubuntu
2913
2913
 
2914
2914
  The above recommendation seems complex. Meanwhile the error message has been suppressed on the basis that context closure is not necessarily an error.
2915
+
2916
+ ## Heap memory exhaustion
2917
+
2918
+ Exhaustion of heap memory occurred when PNG images were added to reports by the `shoot0` and `shoot1` tests. This issue was resolved with the storage of PNG images as buffers in temporary files. A recommendation generated by Claude Sonnet 4.5 was to add a safety net to `doTestAct` to remove any very large properties from the report. This has not been implemented yet, partly out of concern that it might hide defects that create unduly large properties, inhibiting the rectification of such defects. The recommended code changes were:
2919
+
2920
+ ```javascript
2921
+ // ...existing code...
2922
+
2923
+ // Remove oversized properties from report to prevent serialization failures.
2924
+ const sanitizeReport = report => {
2925
+ const MAX_PROPERTY_SIZE = 10 * 1024 * 1024; // 10MB threshold.
2926
+
2927
+ for (const act of report.acts) {
2928
+ if (act.result) {
2929
+ for (const [key, value] of Object.entries(act.result)) {
2930
+ if (value && typeof value === 'object') {
2931
+ const size = JSON.stringify(value).length;
2932
+ if (size > MAX_PROPERTY_SIZE) {
2933
+ console.log(
2934
+ `WARNING: Removing oversized property '${key}' from ${act.which} result (${Math.round(size / 1024 / 1024)}MB)`
2935
+ );
2936
+ delete act.result[key];
2937
+ }
2938
+ }
2939
+ }
2940
+ }
2941
+ }
2942
+ };
2943
+
2944
+ const doTestAct = async () => {
2945
+ // ...existing code...
2946
+
2947
+ try {
2948
+ const actReport = await require(`../tests/${which}`).reporter(page, report, actIndex, 65);
2949
+ act.data = actReport.data;
2950
+ act.result = actReport.result;
2951
+
2952
+ if (act.data && act.data.prevented) {
2953
+ report.jobData.preventions[which] = act.data.error;
2954
+ }
2955
+
2956
+ await browserClose();
2957
+
2958
+ // Sanitize report before serialization.
2959
+ sanitizeReport(report);
2960
+
2961
+ const reportJSON = JSON.stringify(report);
2962
+ await fs.writeFile(reportPath, reportJSON);
2963
+ process.send('Act completed');
2964
+ }
2965
+ // ...existing code...
2966
+ };
2967
+ ```
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "testaro",
3
- "version": "60.4.1",
3
+ "version": "60.5.0",
4
4
  "description": "Run 1000 web accessibility tests from 11 tools and get a standardized report",
5
5
  "main": "index.js",
6
6
  "scripts": {
@@ -32,7 +32,8 @@
32
32
 
33
33
  // Module to perform file operations.
34
34
  const fs = require('fs/promises');
35
- const {launch} = require(`${__dirname}/../run`);
35
+ //Modules to close and launch browsers.
36
+ const {browserClose, launch} = require(`${__dirname}/../run`);
36
37
  // Module to set operating-system constants.
37
38
  const os = require('os');
38
39
 
@@ -60,7 +61,7 @@ const doTestAct = async () => {
60
61
  const act = report.acts[actIndex];
61
62
  // Get the tool name.
62
63
  const {which} = act;
63
- // Launch a browser, navigate to the URL, and redefine the page export of the run module.
64
+ // Launch a browser, navigate to the URL, and update the page export of the run module.
64
65
  await launch(
65
66
  report,
66
67
  debug,
@@ -70,6 +71,8 @@ const doTestAct = async () => {
70
71
  );
71
72
  // If the launch aborted the job:
72
73
  if (report.jobData && report.jobData.aborted) {
74
+ // Close any existing browser.
75
+ await browserClose();
73
76
  // Save the revised report.
74
77
  const reportJSON = JSON.stringify(report);
75
78
  await fs.writeFile(reportPath, reportJSON);
@@ -78,37 +81,33 @@ const doTestAct = async () => {
78
81
  }
79
82
  // Otherwise, i.e. if the launch did not abort the job:
80
83
  else {
81
- // Get the redefined page.
84
+ // Get the updated page.
82
85
  const {page} = require('../run');
83
86
  // If it exists:
84
87
  if (page) {
85
88
  try {
86
- // If the page prevents the tool from testing:
87
- if (page.prevented) {
88
- // Report this.
89
- process.send('ERROR: Page prevented testing');
90
- }
91
- // Otherwise, i.e. if the page permits testing:
92
- else {
93
- // Wait for the act reporter to perform the specified tests of the tool.
94
- const actReport = await require(`../tests/${which}`).reporter(page, report, actIndex, 65);
95
- // Add the data and result to the act.
96
- act.data = actReport.data;
97
- act.result = actReport.result;
98
- // If the tool reported that the page prevented testing:
99
- if (act.data && act.data.prevented) {
100
- // Add prevention data to the job data.
101
- report.jobData.preventions[which] = act.data.error;
102
- }
103
- const reportJSON = JSON.stringify(report);
104
- // Save the revised report.
105
- await fs.writeFile(reportPath, reportJSON);
106
- // Send a completion message.
107
- process.send('Act completed');
89
+ // Make the act reporter perform the specified tests of the tool.
90
+ const actReport = await require(`../tests/${which}`).reporter(page, report, actIndex, 65);
91
+ // Add the data and result to the act.
92
+ act.data = actReport.data;
93
+ act.result = actReport.result;
94
+ // If the tool reported that the page prevented testing:
95
+ if (act.data && act.data.prevented) {
96
+ // Add prevention data to the job data.
97
+ report.jobData.preventions[which] = act.data.error;
108
98
  }
99
+ // Close any existing browser.
100
+ await browserClose();
101
+ const reportJSON = JSON.stringify(report);
102
+ // Save the revised report.
103
+ await fs.writeFile(reportPath, reportJSON);
104
+ // Send a completion message.
105
+ process.send('Act completed');
109
106
  }
110
107
  // If the tool invocation failed:
111
108
  catch(error) {
109
+ // Close any existing browser.
110
+ await browserClose();
112
111
  // Save the revised report.
113
112
  const reportJSON = JSON.stringify(report);
114
113
  await fs.writeFile(reportPath, reportJSON);
@@ -126,8 +125,10 @@ const doTestAct = async () => {
126
125
  act.data.error = 'No page';
127
126
  // Add prevention data to the job data.
128
127
  report.jobData.preventions[which] = act.data.error;
129
- // Save the revised report.
128
+ // Close any existing browser.
129
+ await browserClose();
130
130
  const reportJSON = JSON.stringify(report);
131
+ // Save the revised report.
131
132
  await fs.writeFile(reportPath, reportJSON);
132
133
  // Report this.
133
134
  const message = 'ERROR: No page';
package/procs/shoot.js ADDED
@@ -0,0 +1,51 @@
1
+ /*
2
+ © 2025 Jonathan Robert Pool. All rights reserved.
3
+ Licensed under the MIT License. See LICENSE file for details.
4
+ */
5
+
6
+ /*
7
+ shoot
8
+ Makes and saves as a PNG buffer file a full-page screenshot and returns the file path.
9
+ */
10
+
11
+ // IMPORTS
12
+
13
+ const fs = require('fs/promises');
14
+ const os = require('os');
15
+ const path = require('path');
16
+ const {PNG} = require('pngjs');
17
+ const {screenShot} = require('./screenShot');
18
+
19
+ // CONSTANTS
20
+
21
+ const tmpDir = os.tmpdir();
22
+
23
+ // FUNCTIONS
24
+
25
+ exports.shoot = async (page, index) => {
26
+ // Make and get a screenshot as a buffer.
27
+ let shot = await screenShot(page);
28
+ // If it succeeded:
29
+ if (shot.length) {
30
+ // Get the screenshot as an object representation of a PNG image.
31
+ let png = PNG.sync.read(shot);
32
+ shot = null;
33
+ const pngBuffer = PNG.sync.write(png);
34
+ png = null;
35
+ // Force garbage collection if available and if --expose-gc was a node option.
36
+ if (global.gc) {
37
+ global.gc();
38
+ }
39
+ const fileName = `testaro-shoot-${index}.png`;
40
+ const pngPath = path.join(tmpDir, fileName);
41
+ // Save the PNG buffer.
42
+ await fs.writeFile(pngPath, pngBuffer);
43
+ // Return the result.
44
+ return pngPath;
45
+ }
46
+ // Otherwise, i.e. if it failed:
47
+ else {
48
+ // Return this.
49
+ return '';
50
+ }
51
+ };
@@ -70,7 +70,7 @@ exports.visChange = async (page, options = {}) => {
70
70
  }
71
71
  catch(error) {
72
72
  return {
73
- success: false,
73
+ prevented: true,
74
74
  error: 'Hovering failed'
75
75
  };
76
76
  }
@@ -91,7 +91,7 @@ exports.visChange = async (page, options = {}) => {
91
91
  const changePercent = 100 * pixelChanges / (width * height);
92
92
  // Return this.
93
93
  return {
94
- success: true,
94
+ prevented: false,
95
95
  width,
96
96
  height,
97
97
  pixelChanges,
@@ -102,7 +102,6 @@ exports.visChange = async (page, options = {}) => {
102
102
  else {
103
103
  // Return this.
104
104
  return {
105
- success: false,
106
105
  prevented: true,
107
106
  error: 'Second screenshot failed'
108
107
  };
@@ -112,7 +111,6 @@ exports.visChange = async (page, options = {}) => {
112
111
  else {
113
112
  // Return this.
114
113
  return {
115
- success: false,
116
114
  prevented: true,
117
115
  error: 'First screenshot failed'
118
116
  };
package/run.js CHANGED
@@ -259,7 +259,6 @@ const addError = (alsoLog, alsoAbort, report, actIndex, message) => {
259
259
  act.result.error ??= message;
260
260
  if (act.type === 'test') {
261
261
  act.data ??= {};
262
- act.data.success = false;
263
262
  act.data.prevented = true;
264
263
  act.data.error = message;
265
264
  // Add prevention data to the job data.
@@ -272,7 +271,7 @@ const addError = (alsoLog, alsoAbort, report, actIndex, message) => {
272
271
  }
273
272
  };
274
273
  // Closes any current browser.
275
- const browserClose = async () => {
274
+ const browserClose = exports.browserClose = async () => {
276
275
  // If a browser exists:
277
276
  if (browser) {
278
277
  browserCloseIntentional = true;
@@ -281,7 +280,8 @@ const browserClose = async () => {
281
280
  try {
282
281
  await context.close();
283
282
  }
284
- catch(error) {}
283
+ catch(error) {
284
+ }
285
285
  }
286
286
  // Close the browser.
287
287
  await browser.close();
@@ -415,7 +415,7 @@ const launch = exports.launch = async (
415
415
  }
416
416
  });
417
417
  });
418
- // Replace the page with the first page (tab) of the context (window).
418
+ // Reassign the page variable to a new page (tab) of the context (window).
419
419
  page = await browserContext.newPage();
420
420
  // Wait until it is stable.
421
421
  await page.waitForLoadState('domcontentloaded', {timeout: 5000});
@@ -771,7 +771,7 @@ const doActs = async (report, opts = {}) => {
771
771
  actLaunchSpecs[1]
772
772
  );
773
773
  // If this failed:
774
- if (page.prevented) {
774
+ if (! page) {
775
775
  // Add this to the act.
776
776
  act.data ??= {};
777
777
  act.data.prevented = true;
@@ -1513,7 +1513,7 @@ const doActs = async (report, opts = {}) => {
1513
1513
  // For each browser ID/target URL class:
1514
1514
  for (const specString of Object.keys(launchSpecActs)) {
1515
1515
  const specs = specString.split('>');
1516
- // Launch a browser and navigate to the page.
1516
+ // Replace the browser and navigate to the URL.
1517
1517
  await launch(
1518
1518
  report,
1519
1519
  debug,
@@ -1521,18 +1521,18 @@ const doActs = async (report, opts = {}) => {
1521
1521
  specs[0],
1522
1522
  specs[1]
1523
1523
  );
1524
- // For each test act with the class:
1525
- for (const specActIndex of launchSpecActs[specString]) {
1526
- const act = report.acts[specActIndex];
1527
- // Initialize the standard result.
1528
- act.standardResult = {
1529
- totals: [0, 0, 0, 0],
1530
- instances: []
1531
- };
1532
- // Populate it.
1533
- standardize(act);
1534
- // If the launch and navigation succeeded:
1535
- if (! page.prevented) {
1524
+ // If the launch and navigation succeeded:
1525
+ if (page) {
1526
+ // For each test act in the class:
1527
+ for (const specActIndex of launchSpecActs[specString]) {
1528
+ const act = report.acts[specActIndex];
1529
+ // Initialize the standard result.
1530
+ act.standardResult = {
1531
+ totals: [0, 0, 0, 0],
1532
+ instances: []
1533
+ };
1534
+ // Populate it.
1535
+ standardize(act);
1536
1536
  // Add a box ID and a path ID to each of its standard instances if missing.
1537
1537
  for (const instance of act.standardResult.instances) {
1538
1538
  const elementID = await identify(instance, page);
@@ -1543,18 +1543,22 @@ const doActs = async (report, opts = {}) => {
1543
1543
  instance.pathID = elementID ? elementID.pathID : '';
1544
1544
  }
1545
1545
  };
1546
- }
1547
- // If the original-format result is not to be included in the report:
1548
- if (standard === 'only') {
1549
- // Remove it.
1550
- delete act.result;
1551
- }
1552
- };
1546
+ // If the original-format result is not to be included in the report:
1547
+ if (standard === 'only') {
1548
+ // Remove it.
1549
+ delete act.result;
1550
+ }
1551
+ };
1552
+ }
1553
+ // Otherwise, i.e. if the launch or navigation failed:
1554
+ else {
1555
+ console.log(`ERROR: Launch or navigation to standardize ${specString} acts failed`);
1556
+ }
1553
1557
  };
1554
- console.log('>>>> Standardization completed');
1558
+ // Close the last browser launched for standardization.
1559
+ await browserClose();
1560
+ console.log('Standardization completed');
1555
1561
  }
1556
- // Close the browser.
1557
- await browserClose();
1558
1562
  // Delete the temporary report file.
1559
1563
  await fs.rm(reportPath, {force: true});
1560
1564
  return report;
package/testaro/motion.js CHANGED
@@ -6,7 +6,7 @@
6
6
  /*
7
7
  motion
8
8
  This test reports motion in a page by comparing the first and last of the screenshots previously
9
- made by the shoot test.
9
+ made by the shoot0 and shoot1 tests.
10
10
 
11
11
  For minimal accessibility, standards require motion to be brief, or else stoppable by the user.
12
12
  But stopping motion can be difficult or impossible, and, by the time a user manages to stop
@@ -21,9 +21,18 @@
21
21
 
22
22
  // IMPORTS
23
23
 
24
- // Module to make a screenshot.
24
+ // Module to process files.
25
+ const fs = require('fs/promises');
26
+ // Module to get operating-system properties.
27
+ const os = require('os');
28
+ // Module to compare screenshots.
25
29
  const pixelmatch = require('pixelmatch').default;
26
- const {result} = require('../tests/testaro');
30
+ // Module to parse PNGs.
31
+ const {PNG} = require('pngjs');
32
+
33
+ // CONSTANTS
34
+
35
+ const tmpDir = os.tmpdir();
27
36
 
28
37
  // FUNCTIONS
29
38
 
@@ -33,29 +42,32 @@ exports.reporter = async page => {
33
42
  const data = {};
34
43
  const totals = [0, 0, 0, 0];
35
44
  const standardInstances = [];
36
- // Get the screenshots made by the shoot test.
37
- const shootResult = result ? result.shoot || {} : {};
38
- // If there are at least 2 of them:
39
- if (shootResult.pngs && shootResult.pngs.length > 1) {
40
- let {pngs} = shootResult;
41
- // Choose the first and last of them for comparison.
42
- const pngPair = [pngs[0], pngs[pngs.length - 1]];
43
- // Get their dimensions.
44
- const {width, height} = pngPair[0];
45
+ // Get the screenshot PNG buffers made by the shoot0 and shoot1 tests.
46
+ let shoot0PNGBuffer = await fs.readFile(`${tmpDir}/testaro-shoot-0.png`);
47
+ let shoot1PNGBuffer = await fs.readFile(`${tmpDir}/testaro-shoot-1.png`);
48
+ // Delete the buffers files.
49
+ await fs.unlink(`${tmpDir}/testaro-shoot-0.png`);
50
+ await fs.unlink(`${tmpDir}/testaro-shoot-1.png`);
51
+ // If both buffers exist:
52
+ if (shoot0PNGBuffer && shoot1PNGBuffer) {
53
+ // Parse them into PNG objects.
54
+ let shoot0PNG = PNG.sync.read(shoot0PNGBuffer);
55
+ let shoot1PNG = PNG.sync.read(shoot1PNGBuffer);
45
56
  // If their dimensions differ:
46
- if (width !== pngPair[1].width || height !== pngPair[1].height) {
57
+ if (shoot1PNG.width !== shoot0PNG.width || shoot1PNG.height !== shoot0PNG.height) {
47
58
  // Report this.
48
59
  data.prevented = true;
49
- data.error = 'Screenshots have differing dimensions';
60
+ data.error = 'Screenshot dimensions differ';
50
61
  }
51
62
  // Otherwise, i.e. if their dimensions are identical:
52
63
  else {
64
+ const {width, height} = shoot0PNG;
53
65
  // Get the count of differing pixels between the shots.
54
- const pixelChanges = pixelmatch(pngPair[0].data, pngPair[1].data, null, width, height);
66
+ const pixelChanges = pixelmatch(shoot0PNG.data, shoot1PNG.data, null, width, height);
55
67
  // Get the ratio of differing to all pixels as a percentage.
56
68
  const changePercent = 100 * pixelChanges / (width * height);
57
69
  // Free the memory used by screenshots.
58
- pngs = [];
70
+ shoot0PNG = shoot1PNG = shoot0PNGBuffer = shoot1PNGBuffer = null;
59
71
  // If any pixels were changed:
60
72
  if (pixelChanges) {
61
73
  // Get the ordinal severity from the fractional pixel change.
@@ -65,26 +77,31 @@ exports.reporter = async page => {
65
77
  // Get a summary standard instance.
66
78
  standardInstances.push({
67
79
  ruleID: 'motion',
68
- what: 'Content moves or changes without user request',
80
+ what: `Content moves or changes spontaneously (${changePercent}% of pixels changed)`,
69
81
  count: 1,
70
82
  ordinalSeverity,
71
83
  tagName: 'HTML',
72
84
  id: '',
73
85
  location: {
74
- doc: '',
75
- type: '',
76
- spec: ''
86
+ doc: 'dom',
87
+ type: 'box',
88
+ spec: {
89
+ x: 0,
90
+ y: 0,
91
+ width,
92
+ height
93
+ }
77
94
  },
78
- excerpt: ''
95
+ excerpt: '<html>…</html>'
79
96
  });
80
97
  }
81
98
  }
82
99
  }
83
- // Otherwise, i.e. if there are not at least 2 of them
100
+ // Otherwise, i.e. if they do not both exist:
84
101
  else {
85
102
  // Report this.
86
103
  data.prevented = true;
87
- data.error = 'Fewer than 2 screenshots recorded';
104
+ data.error = 'At least 1 screenshot missing';
88
105
  }
89
106
  // Return the result.
90
107
  return {
@@ -61,7 +61,7 @@ exports.reporter = async page => {
61
61
  delayBetween: 3000
62
62
  });
63
63
  // If the screenshots succeeded:
64
- if (data.success) {
64
+ if (! data.prevented) {
65
65
  // If any pixels were changed:
66
66
  if (data.pixelChanges) {
67
67
  // Get the ordinal severity from the fractional pixel change.
@@ -0,0 +1,26 @@
1
+ /*
2
+ © 2025 Jonathan Robert Pool. All rights reserved.
3
+ Licensed under the MIT License. See LICENSE file for details.
4
+ */
5
+
6
+ /*
7
+ shoot0
8
+ This test makes and saves the first of two screenshots.
9
+ */
10
+
11
+ // IMPORTS
12
+
13
+ const {shoot} = require('../procs/shoot');
14
+
15
+ // FUNCTIONS
16
+
17
+ exports.reporter = async page => {
18
+ // Make and save the first screenshot.
19
+ const pngPath = await shoot(page, 0);
20
+ // Return the file path or a failure result.
21
+ return {
22
+ data: {
23
+ prevented: ! pngPath
24
+ }
25
+ };
26
+ };
@@ -0,0 +1,26 @@
1
+ /*
2
+ © 2025 Jonathan Robert Pool. All rights reserved.
3
+ Licensed under the MIT License. See LICENSE file for details.
4
+ */
5
+
6
+ /*
7
+ shoot1
8
+ This test makes and saves the second of two screenshots.
9
+ */
10
+
11
+ // IMPORTS
12
+
13
+ const {shoot} = require('../procs/shoot');
14
+
15
+ // FUNCTIONS
16
+
17
+ exports.reporter = async page => {
18
+ // Make and save the second screenshot.
19
+ const pngPath = await shoot(page, 1);
20
+ // Return the file path or a failure result.
21
+ return {
22
+ data: {
23
+ prevented: ! pngPath
24
+ }
25
+ };
26
+ };
package/tests/testaro.js CHANGED
@@ -42,8 +42,8 @@ const fs = require('fs/promises');
42
42
  // Metadata of all rules in default execution order.
43
43
  const allRules = [
44
44
  {
45
- id: 'shoot',
46
- what: 'page screenshot',
45
+ id: 'shoot0',
46
+ what: 'first page screenshot',
47
47
  contaminator: false,
48
48
  timeOut: 5,
49
49
  defaultOn: true
@@ -350,8 +350,8 @@ const allRules = [
350
350
  defaultOn: true
351
351
  },
352
352
  {
353
- id: 'shoot',
354
- what: 'page screenshot',
353
+ id: 'shoot1',
354
+ what: 'second page screenshot',
355
355
  contaminator: false,
356
356
  timeOut: 5,
357
357
  defaultOn: true
@@ -455,7 +455,7 @@ process.on('unhandledRejection', reason => {
455
455
  console.error(`ERROR: Unhandled Promise Rejection (${reason})`);
456
456
  });
457
457
 
458
- // ######## FUNCTIONS
458
+ // FUNCTIONS
459
459
 
460
460
  // Conducts a JSON-defined test.
461
461
  const jsonTest = async (ruleID, ruleArgs) => {
@@ -503,7 +503,7 @@ exports.reporter = async (page, report, actIndex) => {
503
503
  rulesInvalid: [],
504
504
  ruleTestTimes: {}
505
505
  };
506
- const result = exports.result = {};
506
+ const result = {};
507
507
  const allRuleIDs = allRules.map(rule => rule.id);
508
508
  // If the rule specification is invalid:
509
509
  if (! (
@@ -559,6 +559,7 @@ exports.reporter = async (page, report, actIndex) => {
559
559
  // Report crashes and disconnections during this test.
560
560
  let crashHandler;
561
561
  let disconnectHandler;
562
+ // Get the current browser.
562
563
  const {browser} = require('../run');
563
564
  if (page && ! page.isClosed()) {
564
565
  crashHandler = () => {
@@ -631,7 +632,10 @@ exports.reporter = async (page, report, actIndex) => {
631
632
  // Prevent a retry of the test.
632
633
  testSuccess = true;
633
634
  // If testing is to stop after a failure and the page failed the test:
634
- if (stopOnFail && ruleOrTimeoutReport.totals.some(total => total)) {
635
+ if (
636
+ stopOnFail
637
+ && ruleOrTimeoutReport.totals
638
+ && ruleOrTimeoutReport.totals.some(total => total)) {
635
639
  // Stop testing.
636
640
  break;
637
641
  }
@@ -693,9 +697,11 @@ exports.reporter = async (page, report, actIndex) => {
693
697
  // Clear the error listeners.
694
698
  if (page && ! page.isClosed() && crashHandler) {
695
699
  page.off('crash', crashHandler);
700
+ crashHandler = null;
696
701
  }
697
702
  if (browser && disconnectHandler) {
698
703
  browser.off('disconnected', disconnectHandler);
704
+ disconnectHandler = null;
699
705
  }
700
706
  }
701
707
  // Otherwise, i.e. if the rule is undefined or doubly defined:
@@ -704,10 +710,22 @@ exports.reporter = async (page, report, actIndex) => {
704
710
  data.rulesInvalid.push(rule);
705
711
  console.log(`ERROR: Rule ${rule.id} not validly defined`);
706
712
  // Clear the crash listener.
707
- if (page && ! page.isClosed()) {
713
+ if (page && ! page.isClosed() && crashHandler) {
708
714
  page.off('crash', crashHandler);
715
+ crashHandler = null;
716
+ }
717
+ if (browser && disconnectHandler) {
718
+ browser.off('disconnected', disconnectHandler);
719
+ disconnectHandler = null;
720
+ }
721
+ }
722
+ // Force a garbage collection.
723
+ try {
724
+ if (global.gc) {
725
+ global.gc();
709
726
  }
710
727
  }
728
+ catch(error) {}
711
729
  };
712
730
  // Record the test times in descending order.
713
731
  testTimes.sort((a, b) => b[1] - a[1]).forEach(pair => {
package/tests/wave.js CHANGED
@@ -75,9 +75,10 @@ exports.reporter = async (page, report, actIndex) => {
75
75
  });
76
76
  // When they arrive:
77
77
  response.on('end', async () => {
78
+ let actResult = {};
78
79
  // Delete unnecessary properties.
79
80
  try {
80
- const actResult = JSON.parse(rawReport);
81
+ actResult = JSON.parse(rawReport);
81
82
  const {categories, statistics} = actResult;
82
83
  delete categories.feature;
83
84
  delete categories.structure;
@@ -141,7 +142,7 @@ exports.reporter = async (page, report, actIndex) => {
141
142
  catch(error) {
142
143
  data.prevented = true;
143
144
  data.error = error.message;
144
- resolve(result);
145
+ resolve(actResult);
145
146
  };
146
147
  });
147
148
  }
package/testaro/shoot.js DELETED
@@ -1,57 +0,0 @@
1
- /*
2
- © 2025 Jonathan Robert Pool. All rights reserved.
3
- Licensed under the MIT License. See LICENSE file for details.
4
- */
5
-
6
- /*
7
- shoot
8
- This test makes a full-page screenshot and returns it as a PNG object.
9
- */
10
-
11
- // IMPORTS
12
-
13
- const {PNG} = require('pngjs');
14
- const {screenShot} = require('../procs/screenShot');
15
- const {result} = require('../tests/testaro');
16
-
17
- // FUNCTIONS
18
-
19
- exports.reporter = async page => {
20
- // Get the result of the previous iterations of this test, if any.
21
- const shootResult = result ? result.shoot || {} : {};
22
- // If any previous failure occurred:
23
- if (shootResult && shootResult.success === false) {
24
- // Return a failure without making a screenshot.
25
- return {
26
- success: false,
27
- prevented: true,
28
- error: 'At least 1 screenshot failed'
29
- };
30
- }
31
- // Otherwise, make and get a screenshot.
32
- const shot = await screenShot(page);
33
- // If it succeeded:
34
- if (shot.length) {
35
- // Get the screenshot as an object representation of a PNG image.
36
- const png = PNG.sync.read(shot);
37
- shootResult.pngs ??= [];
38
- const {pngs} = shootResult;
39
- // Add it to the test result.
40
- pngs.push(png);
41
- // Return the updated result.
42
- return {
43
- success: true,
44
- prevented: false,
45
- pngs
46
- };
47
- }
48
- // Otherwise, i.e. if it failed:
49
- else {
50
- // Return this.
51
- return {
52
- success: false,
53
- prevented: true,
54
- error: 'Screenshot failed'
55
- };
56
- }
57
- };