testilo 35.1.1 → 36.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +132 -134
- package/batch.js +7 -15
- package/call.js +8 -12
- package/merge.js +44 -78
- package/package.json +1 -1
- package/procs/digest/tdp43e/index.html +10 -2
- package/procs/digest/tdp43e/index.js +13 -10
- package/procs/track/ttp40/index.js +1 -1
- package/script.js +24 -7
package/README.md
CHANGED
|
@@ -77,11 +77,11 @@ For example, a target list might be:
|
|
|
77
77
|
]
|
|
78
78
|
```
|
|
79
79
|
|
|
80
|
-
A target list can be represented by a text file, in which each target is specified on a line with a
|
|
80
|
+
A target list can be represented by a text file, in which each target is specified on a line with a Vertical Line character (`|`) delimiting its description and its URL, which are not quoted. Such a file representing the above target list would have this content:
|
|
81
81
|
|
|
82
82
|
```text
|
|
83
|
-
World Wide Web Consortium
|
|
84
|
-
Mozilla Foundation
|
|
83
|
+
World Wide Web Consortium|https://www.w3.org/
|
|
84
|
+
Mozilla Foundation|https://foundation.mozilla.org/en/
|
|
85
85
|
```
|
|
86
86
|
|
|
87
87
|
### Batches
|
|
@@ -92,17 +92,15 @@ Targets can be specified in a more complex way, too. That allows you to create j
|
|
|
92
92
|
{
|
|
93
93
|
id: 'clothing-stores',
|
|
94
94
|
what: 'clothing stores',
|
|
95
|
-
targets:
|
|
96
|
-
{
|
|
97
|
-
id: 'acme',
|
|
95
|
+
targets: {
|
|
96
|
+
acme: {
|
|
98
97
|
what: 'Acme Clothes',
|
|
99
|
-
|
|
100
|
-
|
|
98
|
+
url: 'https://acmeclothes.com/',
|
|
99
|
+
actGroups: {
|
|
101
100
|
public: [
|
|
102
101
|
{
|
|
103
102
|
type: 'launch',
|
|
104
103
|
what: 'Acme Clothes home page',
|
|
105
|
-
url: 'https://acmeclothes.com/'
|
|
106
104
|
}
|
|
107
105
|
],
|
|
108
106
|
private: [
|
|
@@ -139,10 +137,54 @@ Targets can be specified in a more complex way, too. That allows you to create j
|
|
|
139
137
|
}
|
|
140
138
|
```
|
|
141
139
|
|
|
142
|
-
As shown, a batch, unlike a target list, defines named
|
|
140
|
+
As shown, a batch, unlike a target list, defines named groups of acts. They can be substituted for script placeholders, so various complex operations can be performed on each target.
|
|
141
|
+
|
|
142
|
+
In this example, the `public` act group contains only 1 act, of type `launch`. A `launch` act in an act group is permitted to have only two properties, `what` and `url`. If either of these is omitted, its value is inherited from the corresponding property of the target. In the `public` act group in this case, the `what` value is specified, but the `url` value will be `https://acmeclothes.com/`, inherited from the target.
|
|
143
143
|
|
|
144
144
|
A batch is a JavaScript object. It can be converted to JSON and stored in a file.
|
|
145
145
|
|
|
146
|
+
### Target list to batch
|
|
147
|
+
|
|
148
|
+
If you have a target list, the `batch` module of Testilo can convert it to a simple batch. The batch will contain, for each target, only one act group, named `main`, containing only a `launch` act. The target’s entry in the target list will determine the `what` and `url` properties of the target, and the `launch` acts will not override the values of those properties.
|
|
149
|
+
|
|
150
|
+
#### Invocation
|
|
151
|
+
|
|
152
|
+
There are two ways to use the `batch` module.
|
|
153
|
+
|
|
154
|
+
##### By a module
|
|
155
|
+
|
|
156
|
+
A module can invoke `batch()` in this way:
|
|
157
|
+
|
|
158
|
+
```javaScript
|
|
159
|
+
const {batch} = require('testilo/batch');
|
|
160
|
+
const id = 'divns';
|
|
161
|
+
const what = 'divisions';
|
|
162
|
+
const targets = [
|
|
163
|
+
['Energy', 'https://abc.com/energy'],
|
|
164
|
+
['Water', 'https://abc.com/water']
|
|
165
|
+
];
|
|
166
|
+
const batchObj = batch(id, what, targets);
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
The `id` argument to `batch()` is an identifier for the target list. The `what` variable describes the target list. The `targets` variable is an array of arrays, with each array containing the 2 items (description and URL) defining one target.
|
|
170
|
+
|
|
171
|
+
The `batch()` function of the `batch` module generates a batch and returns it as an object. Within the batch, each target is given a sequential (base-62 alphanumeric) string as an ID.
|
|
172
|
+
|
|
173
|
+
The invoking module can further dispose of the batch as needed.
|
|
174
|
+
|
|
175
|
+
##### By a user
|
|
176
|
+
|
|
177
|
+
A user can invoke `batch()` in this way:
|
|
178
|
+
|
|
179
|
+
- Create a target list and save it as a text file (with Vertical-Line-delimited items in Newline-delimited lines) in the `targetLists` subdirectory of the `SPECDIR` directory. Name the file `x.txt`, where `x` is the list ID.
|
|
180
|
+
- In the Testilo project directory, execute the statement `node call batch id what`.
|
|
181
|
+
|
|
182
|
+
In this statement, replace `id` with the list ID and `what` with a string describing the batch. Example: `node call batch divns 'ABC company divisions'`.
|
|
183
|
+
|
|
184
|
+
The `call` module will retrieve the named target list.
|
|
185
|
+
The `batch` module will convert the target list to a batch.
|
|
186
|
+
The `call` module will save the batch as a JSON file named `x.json` (replacing `x` with the list ID) in the `batches` subdirectory of the `SPECDIR` directory.
|
|
187
|
+
|
|
146
188
|
### Scripts
|
|
147
189
|
|
|
148
190
|
The generic, target-independent description of a job is _script_. A script can contain _placeholders_ that Testilo replaces with acts from a batch, creating one job per target. Thus, one script plus a batch containing _n_ targets will generate _n_ jobs.
|
|
@@ -155,12 +197,32 @@ Here is a script:
|
|
|
155
197
|
what: 'aside mislocation',
|
|
156
198
|
strict: true,
|
|
157
199
|
isolate: true,
|
|
158
|
-
|
|
200
|
+
standard: 'also',
|
|
201
|
+
observe: false,
|
|
202
|
+
deviceID: 'Kindle Fire HDX',
|
|
203
|
+
browserID: 'webkit',
|
|
204
|
+
lowMotion: false,
|
|
205
|
+
timeLimit: 80,
|
|
206
|
+
creationTimeStamp: ''
|
|
207
|
+
executionTimeStamp: '',
|
|
208
|
+
sources: {
|
|
209
|
+
script: 'ts99',
|
|
210
|
+
batch: '',
|
|
211
|
+
target: {
|
|
212
|
+
what: '',
|
|
213
|
+
url: ''
|
|
214
|
+
},
|
|
215
|
+
lastTarget: false,
|
|
216
|
+
mergeID: '',
|
|
217
|
+
sendReportTo: '',
|
|
218
|
+
requester: ''
|
|
219
|
+
},
|
|
159
220
|
acts: [
|
|
160
221
|
{
|
|
161
222
|
type: 'placeholder',
|
|
162
223
|
which: 'private',
|
|
163
|
-
|
|
224
|
+
deviceID: 'default',
|
|
225
|
+
browserID: 'chromium'
|
|
164
226
|
},
|
|
165
227
|
{
|
|
166
228
|
type: 'test',
|
|
@@ -181,58 +243,22 @@ Here is a script:
|
|
|
181
243
|
```
|
|
182
244
|
|
|
183
245
|
A script has several properties that specify facts about the jobs to be created. They include:
|
|
184
|
-
- `id`: an ID. A script can be converted from a JavaScript object to JSON and saved in a file in the `SPECDIR` directory, where it will be named by its ID (e.g., if the ID is `ts99`, the file name will be `ts99.json`).
|
|
246
|
+
- `id`: an ID. A script can be converted from a JavaScript object to JSON and saved in a file in the `SPECDIR` directory, where it will be named by its ID (e.g., if the ID is `ts99`, the file name will be `ts99.json`). Each script needs an `id` with a unique value composed of alphanumeric ASCII characters.
|
|
185
247
|
- `what`: a description of the script.
|
|
186
248
|
- `strict`: `true` if Testaro is to abort jobs when a target redirects a request to a URL differing substantially from the one specified. If `false` Testaro is to allow redirection. All differences are considered substantial unless the URLs differ only in the presence and absence of a trailing slash.
|
|
187
249
|
- `isolate`: If `true`, Testilo, before creating a job, will isolate test acts, as needed, from effects of previous test acts, by inserting a copy of the latest placeholder after each target-modifying test act other than the final act. If `false`, placeholders will not be duplicated.
|
|
250
|
+
- `standard`: When Testaro performs a job, every tool produces its own report. Testaro can convert the test results of each tool report to standard results. The `standard` property specifies how to handle standardization. If `also`, Testaro will include in its reports both the original results of the tests of tools and the Testaro-standardized results. If `only`, reports will include only the standardized test results. If `no`, reports will include only the original results, without standardization.
|
|
251
|
+
- `observe`: Testaro jobs can allow granular observation. If `true`, the job will do so. If `false`, Testaro will not report job progress, but will send a report to the server only when the report is completed. It is generally user-friendly to allow granular observation, and for user applications to implement it, if they make users wait while jobs are assigned and performed, since that process typically takes a few minutes.
|
|
188
252
|
- `timeLimit`: This specifies the maximum duration, in seconds, of a job. Testaro will abort jobs that are not completed within that time.
|
|
253
|
+
- `deviceID`: This specifies the default device type of the job.
|
|
254
|
+
- `browserID`: This specifies the default browser type (`'chromium'`, `'firefox'`, or `'webkit'`) of the job.
|
|
255
|
+
- `lowMotion`: This is true if the browser is to create tabs with the `reduce-motion` option set to `reduce` instead of `no-preference`. This makes the browser act as if the user has chosen a [motion-reduction option in the settings of the operating system or browser](https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-reduced-motion#user_preferences). Motions on pages are, however, often immune to this setting.
|
|
256
|
+
- `creationTimeStamp`, `executionTimeStamp`, and `sources`: These properties will have values assigned to them when jobs are created from the script, except for the `sources.script` property, which will preserve the ID of the script after the `id` property has been replaced with a job ID.
|
|
189
257
|
- `acts`: an array of acts.
|
|
190
258
|
|
|
191
|
-
|
|
259
|
+
In this example, the script contains 3 acts, of which the first is a placeholder. If the above batch were merged with this script, in each job the placeholder would be replaced with the acts in the `private` act group of a target. For example, the first act of the first job would launch a Chromium browser on a default device, navigate to the Acme login page, complete and submit the login form, wait for the account page to load, run the Axe tests, and then run the QualWeb tests. If the batch contained additional targets, additional jobs would be created, with the acts for each target specified by the `private` property of the `actGroups` object of that target.
|
|
192
260
|
|
|
193
|
-
As shown in this example,
|
|
194
|
-
|
|
195
|
-
### Target list to batch
|
|
196
|
-
|
|
197
|
-
If you have a target list, the `batch` module of Testilo can convert it to a simple batch. The batch will contain, for each target, only one array of acts, named `main`, containing only a `launch` act (depending on the script to specify the browser type and depending on the target to specify the URL).
|
|
198
|
-
|
|
199
|
-
#### Invocation
|
|
200
|
-
|
|
201
|
-
There are two ways to use the `batch` module.
|
|
202
|
-
|
|
203
|
-
##### By a module
|
|
204
|
-
|
|
205
|
-
A module can invoke `batch()` in this way:
|
|
206
|
-
|
|
207
|
-
```javaScript
|
|
208
|
-
const {batch} = require('testilo/batch');
|
|
209
|
-
const id = 'divns';
|
|
210
|
-
const what = 'divisions';
|
|
211
|
-
const targets = [
|
|
212
|
-
['Energy', 'https://abc.com/energy'],
|
|
213
|
-
['Water', 'https://abc.com/water']
|
|
214
|
-
];
|
|
215
|
-
const batchObj = batch(id, what, targets);
|
|
216
|
-
```
|
|
217
|
-
|
|
218
|
-
The `id` argument to `batch()` is a unique identifier for the target list. The `what` variable describes the target list. The `targets` variable is an array of arrays, with each array containing the 2 items (description and URL) defining one target.
|
|
219
|
-
|
|
220
|
-
The `batch()` function of the `batch` module generates a batch and returns it as an object. Within the batch, each target is given a sequential (base-62 alphanumeric) string as an ID.
|
|
221
|
-
|
|
222
|
-
The invoking module can further dispose of the batch as needed.
|
|
223
|
-
|
|
224
|
-
##### By a user
|
|
225
|
-
|
|
226
|
-
A user can invoke `batch()` in this way:
|
|
227
|
-
|
|
228
|
-
- Create a target list and save it as a text file (with tab-delimited items in newline-delimited lines) in the `targetLists` subdirectory of the `SPECDIR` directory. Name the file `x.txt`, where `x` is the list ID.
|
|
229
|
-
- In the Testilo project directory, execute the statement `node call batch id what`.
|
|
230
|
-
|
|
231
|
-
In this statement, replace `id` with the list ID and `what` with a string describing the batch. Example: `node call batch divns 'ABC company divisions'`.
|
|
232
|
-
|
|
233
|
-
The `call` module will retrieve the named target list.
|
|
234
|
-
The `batch` module will convert the target list to a batch.
|
|
235
|
-
The `call` module will save the batch as a JSON file named `x.json` (replacing `x` with the list ID) in the `batches` subdirectory of the `SPECDIR` directory.
|
|
261
|
+
As shown in this example, it is possible for any particular placeholder to override the default device type and/or browser type of the script by having its own optional `deviceID` and/or `browserID` property. Some rules are particularly relevant to some device types and/or can be successfully tested only with particular browser types. Overriding the default device and browser types lets you handle such constraints.
|
|
236
262
|
|
|
237
263
|
### Script creation
|
|
238
264
|
|
|
@@ -277,9 +303,9 @@ For an issue restriction, it has this structure:
|
|
|
277
303
|
|
|
278
304
|
If you specify tool options, the script will prescribe the tests for all evaluation rules of the tools that you specify.
|
|
279
305
|
|
|
280
|
-
If you specify issue options, the script will prescribe the tests for all evaluation rules that are classified
|
|
306
|
+
If you specify issue options, the script will prescribe the tests for all evaluation rules that are classified into the issues whose IDs you specify. Any tools that do not have any of those rules will be omitted. The value of `specs.issues` is an issue classification object, with a structure like the one in `procs/score/tic43.js`. That one classifies about 1000 rules into about 300 issues.
|
|
281
307
|
|
|
282
|
-
For example, one issue in the `
|
|
308
|
+
For example, one issue in the `tic43.js` file is `mainNot1`. Four rules are classified as belonging to that issue: rule `main_element_only_one` of the `aslint` tool and 3 more rules defined by 3 other tools. You can also create custom classifications and save them in a `score` subdirectory of the `FUNCTIONDIR` directory.
|
|
283
309
|
|
|
284
310
|
#### Invocation
|
|
285
311
|
|
|
@@ -318,19 +344,34 @@ The first form will create a script with no restrictions.
|
|
|
318
344
|
|
|
319
345
|
The second form will create a script that prescribes tests for all the rules of the specified tools.
|
|
320
346
|
|
|
321
|
-
The third form will create a script that prescribes tests for all the rules classified by the
|
|
347
|
+
The third form will create a script that prescribes tests for all the rules classified by the named issue-classification file into any of the specified issues.
|
|
322
348
|
|
|
323
|
-
In this statement, replace `id` with an ID for the script, such as `headings1`, consisting of ASCII alphanumeric characters.
|
|
349
|
+
In this statement, replace `id` with an ID for the script, such as `headings1`, consisting of ASCII alphanumeric characters, or with `''` if you want Testilo to create an ID. Replace `what` with a string describing the script, or with `''` if you want Testilo to create a generic description.
|
|
324
350
|
|
|
325
351
|
The `call` module will retrieve the named classification, if any.
|
|
326
352
|
The `script` module will create a script.
|
|
327
353
|
The `call` module will save the script as a JSON file in the `scripts` subdirectory of the `SPECDIR` directory, using the `id` value as the base of the file name.
|
|
328
354
|
|
|
329
|
-
####
|
|
355
|
+
#### Properties
|
|
330
356
|
|
|
331
|
-
|
|
357
|
+
When the `script` module creates a script for you, it does not ask you for all of the property values that the script may require. Instead, it chooses these default values:
|
|
358
|
+
- `strict`: `false`
|
|
359
|
+
- `isolate`: `true`
|
|
360
|
+
- `standard`: `'only'`
|
|
361
|
+
- `observe`: `false`
|
|
362
|
+
- `deviceID`: `'default'`
|
|
363
|
+
- `browserID`: `'webkit'`
|
|
364
|
+
- `lowMotion`: `false`
|
|
365
|
+
- `timeLimit`: 50 plus 30 per tool
|
|
366
|
+
- `axe` test act: `detailLevel` = 2
|
|
367
|
+
- `ibm` test act: `withItems` = `true`, `withNewContent` = `false`
|
|
368
|
+
- `qualWeb` test act: `withNewContent` = `false`
|
|
369
|
+
- `testaro` test act: `withItems` = true, `stopOnFail` = `false`
|
|
370
|
+
- `wave` test act: `reportType` = 4
|
|
332
371
|
|
|
333
|
-
|
|
372
|
+
The `webkit` browser type is selected because the other browser types corrupt some tests. The `ibm` test is performed on the existing page content because some targets cause HTTP2 protocol errors when the `ibm` tool tries to visit them.
|
|
373
|
+
|
|
374
|
+
After you invoke `script`, you can edit the script that it creates to revise any of these options.
|
|
334
375
|
|
|
335
376
|
### Merge
|
|
336
377
|
|
|
@@ -348,26 +389,13 @@ A module can invoke `merge()` in this way:
|
|
|
348
389
|
const {merge} = require('testilo/merge');
|
|
349
390
|
const script = …;
|
|
350
391
|
const batch = …;
|
|
351
|
-
const standard = 'only';
|
|
352
|
-
const observe = false;
|
|
353
|
-
const requester = 'me@mydomain.tld';
|
|
354
392
|
const timeStamp = '241215T1200';
|
|
355
|
-
const jobs = merge(script, batch,
|
|
393
|
+
const jobs = merge(script, batch, timeStamp);
|
|
356
394
|
```
|
|
357
395
|
|
|
358
396
|
The first two arguments are a script and a batch obtained from files or from prior calls to `script()` and `batch()`.
|
|
359
397
|
|
|
360
|
-
The `
|
|
361
|
-
|
|
362
|
-
The `observe` argument tells Testaro whether the jobs should allow granular observation. If `true`, it will. If `false`, Testaro will not report job progress, but will send reports to the server only when the reports are completed. It is generally user-friendly to allow granular observation, and for user applications to implement it, if they make users wait while jobs are assigned and performed, since that process typically takes a few minutes.
|
|
363
|
-
|
|
364
|
-
The `requester` argument is an email address to which any notices about the job are to be sent.
|
|
365
|
-
|
|
366
|
-
The `timeStamp` argument specifies the earliest UTC date and time when the jobs may be assigned, or it may be an empty string if now.
|
|
367
|
-
|
|
368
|
-
The `deviceID` argument specifies the ID of the test device. It must be either `'default'` or the ID of one of the test devices recognized by Playwright, published at `https://github.com/microsoft/playwright/blob/main/packages/playwright-core/src/server/deviceDescriptorsSource.json`.
|
|
369
|
-
|
|
370
|
-
The `merge()` function returns the jobs in an array. The invoking module can further dispose of the jobs as needed.
|
|
398
|
+
The `merge()` function returns an array of jobs, one job per target in the batch. The invoking module can further dispose of the jobs as needed.
|
|
371
399
|
|
|
372
400
|
##### By a user
|
|
373
401
|
|
|
@@ -378,13 +406,13 @@ A user can invoke `merge()` in this way:
|
|
|
378
406
|
- In the Testilo project directory, execute the statement:
|
|
379
407
|
|
|
380
408
|
```javascript
|
|
381
|
-
node call merge scriptID batchID
|
|
409
|
+
node call merge scriptID batchID executionTimeStamp todoDir
|
|
382
410
|
```
|
|
383
411
|
|
|
384
412
|
In this statement, replace:
|
|
385
413
|
- `scriptID` with the ID (which is also the base of the file name) of the script.
|
|
386
414
|
- `batchID` with the ID (which is also the base of the file name) of the batch.
|
|
387
|
-
- `
|
|
415
|
+
- `executionTimeStamp` with a time stamp in format `yymmddThhMM` representing the UTC date and time before which the jobs are not to be executed, or `''` if it is now.
|
|
388
416
|
- `todoDir`: `true` if the jobs are to be saved in the `todo` subdirectory, or `false` if they are to be saved in the `pending` subdirectory, of the `JOBDIR` directory.
|
|
389
417
|
|
|
390
418
|
The `call` module will retrieve the named script and batch from their respective directories.
|
|
@@ -393,61 +421,30 @@ The `call` module will save the jobs as JSON files in the `todo` or `pending` su
|
|
|
393
421
|
|
|
394
422
|
#### Output
|
|
395
423
|
|
|
396
|
-
A Testaro job produced by `merge`
|
|
424
|
+
A Testaro job produced by `merge` will be identical to the script from which it was derived (see the example above), except that:
|
|
425
|
+
- The `id` property of the job will be revised to uniquely identify the job.
|
|
426
|
+
- The originally empty properties will be populated, as in this example:
|
|
397
427
|
|
|
398
|
-
```javaScript
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
deviceID: 'Galaxy S8',
|
|
414
|
-
url: 'https://acmeclothes.com/'
|
|
415
|
-
},
|
|
416
|
-
{
|
|
417
|
-
type: 'test',
|
|
418
|
-
which: 'axe',
|
|
419
|
-
detailLevel: 2,
|
|
420
|
-
rules: ['landmark-complementary-is-top-level'],
|
|
421
|
-
what: 'Axe'
|
|
422
|
-
},
|
|
423
|
-
{
|
|
424
|
-
type: 'launch',
|
|
425
|
-
which: 'webkit',
|
|
426
|
-
what: 'Acme Clothes',
|
|
427
|
-
deviceID: 'Galaxy S8',
|
|
428
|
-
url: 'https://acmeclothes.com/'
|
|
429
|
-
},
|
|
430
|
-
{
|
|
431
|
-
type: 'test',
|
|
432
|
-
which: 'qualWeb',
|
|
433
|
-
withNewContent: false,
|
|
434
|
-
rules: ['QW-BP25', 'QW-BP26']
|
|
435
|
-
what: 'QualWeb'
|
|
436
|
-
}
|
|
437
|
-
],
|
|
438
|
-
sources: {
|
|
439
|
-
script: 'ts99',
|
|
440
|
-
batch: 'clothing-stores',
|
|
441
|
-
lastTarget: false,
|
|
442
|
-
target: {
|
|
443
|
-
id: 'acme',
|
|
444
|
-
what: 'Acme Clothes'
|
|
428
|
+
```javaScript
|
|
429
|
+
…
|
|
430
|
+
creationTimeStamp: '241229T0537',
|
|
431
|
+
executionTimeStamp: '250110T1200',
|
|
432
|
+
sources: {
|
|
433
|
+
script: 'ts99',
|
|
434
|
+
batch: 'departments',
|
|
435
|
+
mergeID: '7f',
|
|
436
|
+
sendReportTo: 'https://abccorp.com/api/report',
|
|
437
|
+
requester: 'malavu@abccorp.com'
|
|
438
|
+
target: {
|
|
439
|
+
what: 'Real Estate Management',
|
|
440
|
+
url: 'https://abccorp.com/mgmt/realproperty.html'
|
|
441
|
+
},
|
|
442
|
+
lastTarget: false,
|
|
445
443
|
},
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
```
|
|
444
|
+
…
|
|
445
|
+
```
|
|
446
|
+
|
|
447
|
+
The `merge()` function will populate the `sources.sendReportTo` property with the value, if any, of the environment variable `SEND_REPORT_TO`, and populate the `sources.requester` property with the value, if any, of the environment variable `REQUESTER`.
|
|
451
448
|
|
|
452
449
|
#### Validation
|
|
453
450
|
|
|
@@ -472,6 +469,7 @@ Thus, a report produced by Testaro contains these properties:
|
|
|
472
469
|
- `timeStamp`
|
|
473
470
|
- `acts`
|
|
474
471
|
- `sources`
|
|
472
|
+
- `timeStamp`
|
|
475
473
|
- `creationTimeStamp`
|
|
476
474
|
- `jobData`
|
|
477
475
|
|
package/batch.js
CHANGED
|
@@ -25,10 +25,6 @@
|
|
|
25
25
|
Converts a target list to a batch.
|
|
26
26
|
*/
|
|
27
27
|
|
|
28
|
-
// IMPORTS
|
|
29
|
-
|
|
30
|
-
const {alphaNumOf} = require('./procs/util');
|
|
31
|
-
|
|
32
28
|
// FUNCTIONS
|
|
33
29
|
|
|
34
30
|
// Converts a target list to a batch and returns the batch.
|
|
@@ -51,25 +47,21 @@ exports.batch = (id, what, targetList) => {
|
|
|
51
47
|
const batch = {
|
|
52
48
|
id,
|
|
53
49
|
what,
|
|
54
|
-
targets:
|
|
50
|
+
targets: {}
|
|
55
51
|
};
|
|
56
52
|
// For each target:
|
|
57
|
-
targetList.forEach(
|
|
53
|
+
targetList.forEach(target => {
|
|
58
54
|
// Add it to the batch.
|
|
59
|
-
batch.targets
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
which: target[1],
|
|
63
|
-
acts: {
|
|
55
|
+
batch.targets[target[0]] = {
|
|
56
|
+
url: target[1],
|
|
57
|
+
actGroups: {
|
|
64
58
|
main: [
|
|
65
59
|
{
|
|
66
|
-
type: 'launch'
|
|
67
|
-
what: target[0],
|
|
68
|
-
url: target[1]
|
|
60
|
+
type: 'launch'
|
|
69
61
|
}
|
|
70
62
|
]
|
|
71
63
|
}
|
|
72
|
-
}
|
|
64
|
+
};
|
|
73
65
|
});
|
|
74
66
|
// Return the batch.
|
|
75
67
|
return batch;
|
package/call.js
CHANGED
|
@@ -186,8 +186,8 @@ const callScript = async (scriptID, what, optionType, ... optionDetails) => {
|
|
|
186
186
|
return;
|
|
187
187
|
}
|
|
188
188
|
}
|
|
189
|
-
// Create a script.
|
|
190
|
-
const scriptObj = script(scriptID, what, optionArg);
|
|
189
|
+
// Create a script with default device and browser IDs.
|
|
190
|
+
const scriptObj = script(scriptID, undefined, undefined, what, optionArg);
|
|
191
191
|
try {
|
|
192
192
|
// Save it.
|
|
193
193
|
const scriptJSON = JSON.stringify(scriptObj, null, 2);
|
|
@@ -203,18 +203,14 @@ const callScript = async (scriptID, what, optionType, ... optionDetails) => {
|
|
|
203
203
|
const callMerge = async (
|
|
204
204
|
scriptID,
|
|
205
205
|
batchID,
|
|
206
|
-
|
|
207
|
-
observe,
|
|
208
|
-
requester,
|
|
209
|
-
timeStamp,
|
|
210
|
-
deviceID,
|
|
206
|
+
executionTimeStamp,
|
|
211
207
|
todoDir
|
|
212
208
|
) => {
|
|
213
209
|
try {
|
|
214
210
|
// If the todoDir argument is invalid:
|
|
215
211
|
if (! ['true', 'false'].includes(todoDir)) {
|
|
216
212
|
// Report this.
|
|
217
|
-
throw new Error('Invalid todoDir
|
|
213
|
+
throw new Error('Invalid todoDir argument to merge');
|
|
218
214
|
}
|
|
219
215
|
// Get the script and the batch.
|
|
220
216
|
const scriptJSON = await fs.readFile(`${specDir}/scripts/${scriptID}.json`, 'utf8');
|
|
@@ -222,18 +218,18 @@ const callMerge = async (
|
|
|
222
218
|
const batchJSON = await fs.readFile(`${specDir}/batches/${batchID}.json`, 'utf8');
|
|
223
219
|
const batch = JSON.parse(batchJSON);
|
|
224
220
|
// Merge them into an array of jobs.
|
|
225
|
-
const jobs = merge(script, batch,
|
|
221
|
+
const jobs = merge(script, batch, executionTimeStamp);
|
|
226
222
|
// Save the jobs.
|
|
227
223
|
const subdir = `${jobDir}/${todoDir === 'true' ? 'todo' : 'pending'}`;
|
|
228
224
|
for (const job of jobs) {
|
|
229
225
|
const jobJSON = JSON.stringify(job, null, 2);
|
|
230
226
|
await fs.writeFile(`${subdir}/${job.id}.json`, `${jobJSON}\n`);
|
|
231
227
|
}
|
|
232
|
-
const truncatedID = `${jobs[0].
|
|
228
|
+
const truncatedID = `${jobs[0].id.replace(/[^-]+$/, '')}…`;
|
|
233
229
|
console.log(`Script ${scriptID} and batch ${batchID} merged as ${truncatedID} in ${subdir}`);
|
|
234
230
|
}
|
|
235
231
|
catch(error) {
|
|
236
|
-
console.log(`ERROR merging script and batch (${error.message})`);
|
|
232
|
+
console.log(`ERROR merging script ${scriptID} and batch ${batchID} (${error.message})`);
|
|
237
233
|
}
|
|
238
234
|
};
|
|
239
235
|
// Gets the file base names (equal to the IDs) of the selected reports.
|
|
@@ -499,7 +495,7 @@ else if (fn === 'script' && (fnArgs.length === 2 || fnArgs.length > 3)) {
|
|
|
499
495
|
console.log('Execution completed');
|
|
500
496
|
});
|
|
501
497
|
}
|
|
502
|
-
else if (fn === 'merge' && fnArgs.length ===
|
|
498
|
+
else if (fn === 'merge' && fnArgs.length === 4) {
|
|
503
499
|
callMerge(... fnArgs)
|
|
504
500
|
.then(() => {
|
|
505
501
|
console.log('Execution completed');
|
package/merge.js
CHANGED
|
@@ -30,7 +30,7 @@
|
|
|
30
30
|
// Module to keep secrets.
|
|
31
31
|
require('dotenv').config();
|
|
32
32
|
// Utility module.
|
|
33
|
-
const {alphaNumOf, dateOf, getRandomString, getNowStamp
|
|
33
|
+
const {alphaNumOf, dateOf, getRandomString, getNowStamp} = require('./procs/util');
|
|
34
34
|
|
|
35
35
|
// ########## CONSTANTS
|
|
36
36
|
|
|
@@ -49,60 +49,34 @@ const mergeIDLength = 2;
|
|
|
49
49
|
// ########## FUNCTIONS
|
|
50
50
|
|
|
51
51
|
// Merges a script and a batch and returns jobs.
|
|
52
|
-
exports.merge = (script, batch,
|
|
53
|
-
// If standard is invalid:
|
|
54
|
-
if (! ['also', 'only', 'no'].includes(standard)) {
|
|
55
|
-
// Report this and quit.
|
|
56
|
-
console.log('ERROR: Invalid standard treatment specified');
|
|
57
|
-
return [];
|
|
58
|
-
}
|
|
59
|
-
// If observe is invalid:
|
|
60
|
-
if (! [true, false].includes(observe)) {
|
|
61
|
-
// Report this and quit.
|
|
62
|
-
console.log('ERROR: Invalid observe configuration specified');
|
|
63
|
-
return [];
|
|
64
|
-
}
|
|
52
|
+
exports.merge = (script, batch, executionTimeStamp) => {
|
|
65
53
|
// If a time stamp was specified:
|
|
66
|
-
if (
|
|
54
|
+
if (executionTimeStamp) {
|
|
67
55
|
// If it is invalid:
|
|
68
|
-
if (! dateOf(
|
|
56
|
+
if (! dateOf(executionTimeStamp)) {
|
|
69
57
|
// Report this and quit.
|
|
70
|
-
console.log('ERROR:
|
|
58
|
+
console.log('ERROR: Execution time stamp invalid');
|
|
71
59
|
return [];
|
|
72
60
|
}
|
|
73
61
|
}
|
|
74
62
|
// Otherwise, i.e. if no time stamp was specified:
|
|
75
63
|
else {
|
|
76
64
|
// Create one for the job.
|
|
77
|
-
|
|
65
|
+
executionTimeStamp = getNowStamp();
|
|
78
66
|
}
|
|
79
|
-
//
|
|
80
|
-
if (! isValidDeviceID(deviceID)) {
|
|
81
|
-
// Report this and quit.
|
|
82
|
-
console.log('ERROR: Device ID invalid');
|
|
83
|
-
return [];
|
|
84
|
-
}
|
|
85
|
-
// Initialize a job as a copy of the script.
|
|
67
|
+
// Initialize a job template as a copy of the script.
|
|
86
68
|
const protoJob = JSON.parse(JSON.stringify(script));
|
|
87
|
-
//
|
|
88
|
-
protoJob.sources = {
|
|
89
|
-
script: script.id,
|
|
90
|
-
batch: batch.id,
|
|
91
|
-
lastTarget: false,
|
|
92
|
-
target: {
|
|
93
|
-
id: '',
|
|
94
|
-
what: '',
|
|
95
|
-
which: ''
|
|
96
|
-
},
|
|
97
|
-
requester
|
|
98
|
-
};
|
|
99
|
-
// Add properties to the job.
|
|
100
|
-
protoJob.standard = standard;
|
|
101
|
-
protoJob.observe = observe;
|
|
102
|
-
protoJob.timeStamp = timeStamp;
|
|
69
|
+
// Populate empty properties of the template.
|
|
103
70
|
protoJob.creationTimeStamp = getNowStamp();
|
|
71
|
+
protoJob.executionTimeStamp = executionTimeStamp;
|
|
72
|
+
const {sources} = protoJob;
|
|
73
|
+
sources.script = script.id;
|
|
74
|
+
sources.batch = batch.id;
|
|
75
|
+
sources.mergeID = getRandomString(mergeIDLength);
|
|
76
|
+
sources.sendReportTo = process.env.SEND_REPORT_TO || '';
|
|
77
|
+
sources.requester = process.env.REQUESTER || '';
|
|
104
78
|
// If isolation was requested:
|
|
105
|
-
if (
|
|
79
|
+
if (protoJob.isolate) {
|
|
106
80
|
// For each act:
|
|
107
81
|
let {acts} = protoJob;
|
|
108
82
|
let lastPlaceholder = {};
|
|
@@ -124,63 +98,55 @@ exports.merge = (script, batch, standard, observe, requester, timeStamp, deviceI
|
|
|
124
98
|
acts[actIndex] = JSON.parse(JSON.stringify([act, lastPlaceholder]));
|
|
125
99
|
}
|
|
126
100
|
};
|
|
127
|
-
// Flatten the acts.
|
|
101
|
+
// Flatten the acts, causing insertion of placeholder copies before acts needing them.
|
|
128
102
|
protoJob.acts = acts.flat();
|
|
129
103
|
}
|
|
130
|
-
// Delete the no-longer-necessary job property.
|
|
131
|
-
delete protoJob.isolate;
|
|
132
104
|
// Initialize an array of jobs.
|
|
133
105
|
const jobs = [];
|
|
134
|
-
// Get an ID for the merger.
|
|
135
|
-
const mergeID = getRandomString(mergeIDLength);
|
|
136
106
|
// For each target in the batch:
|
|
137
107
|
const {targets} = batch;
|
|
138
|
-
|
|
108
|
+
const targetIDs = Object.keys(targets);
|
|
109
|
+
targetIDs.forEach((what, index) => {
|
|
139
110
|
// If the target has the required identifiers:
|
|
140
|
-
const {
|
|
141
|
-
if (
|
|
111
|
+
const {actGroups, url} = targets[what];
|
|
112
|
+
if (actGroups && url) {
|
|
142
113
|
// Initialize a job.
|
|
143
114
|
const job = JSON.parse(JSON.stringify(protoJob));
|
|
115
|
+
const {sources} = job;
|
|
144
116
|
// Make the job ID unique.
|
|
145
|
-
const
|
|
146
|
-
job.id = `${
|
|
147
|
-
// Add other properties to the job.
|
|
148
|
-
job.mergeID = mergeID;
|
|
149
|
-
job.sendReportTo = process.env.SEND_REPORT_TO || '';
|
|
117
|
+
const targetSuffix = alphaNumOf(index);
|
|
118
|
+
job.id = `${executionTimeStamp}-${sources.mergeID}-${targetSuffix}`;
|
|
150
119
|
// If the target is the last one:
|
|
151
|
-
if (index ===
|
|
120
|
+
if (index === targetIDs.length - 1) {
|
|
152
121
|
// Add that fact to the sources property of the job.
|
|
153
|
-
|
|
122
|
+
sources.lastTarget = true;
|
|
154
123
|
}
|
|
155
|
-
//
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
job
|
|
159
|
-
// Replace each placeholder object in the job with the named replacer array of the target.
|
|
124
|
+
// Populate the target-specific properties of the job.
|
|
125
|
+
sources.target.what = what;
|
|
126
|
+
sources.target.url = url;
|
|
127
|
+
// Replace each placeholder object in the job with the named act group of the target.
|
|
160
128
|
let {acts} = job;
|
|
161
129
|
for (const actIndex in acts) {
|
|
162
130
|
const act = acts[actIndex];
|
|
163
131
|
if (act.type === 'placeholder') {
|
|
164
132
|
const replacerName = act.which;
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
133
|
+
const replacerActs = actGroups[replacerName];
|
|
134
|
+
if (replacerName && actGroups && replacerActs) {
|
|
135
|
+
// Add properties to any launch act in the replacer.
|
|
136
|
+
for (const replacerAct of replacerActs) {
|
|
137
|
+
if (replacerAct.type === 'launch') {
|
|
138
|
+
if (act.deviceID) {
|
|
139
|
+
replacerAct.deviceID = act.deviceID;
|
|
140
|
+
}
|
|
141
|
+
if (act.browserID) {
|
|
142
|
+
replacerAct.browserID = act.browserID;
|
|
174
143
|
}
|
|
175
144
|
}
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
else {
|
|
179
|
-
console.log(`ERROR: Target ${target.id} has no ${replacerName} replacer`);
|
|
180
|
-
}
|
|
145
|
+
};
|
|
146
|
+
acts[actIndex] = replacerActs;
|
|
181
147
|
}
|
|
182
148
|
else {
|
|
183
|
-
console.log(`ERROR: Placeholder for target ${
|
|
149
|
+
console.log(`ERROR: Placeholder for target ${what} not replaceable`);
|
|
184
150
|
}
|
|
185
151
|
}
|
|
186
152
|
}
|
|
@@ -190,7 +156,7 @@ exports.merge = (script, batch, standard, observe, requester, timeStamp, deviceI
|
|
|
190
156
|
jobs.push(job);
|
|
191
157
|
}
|
|
192
158
|
else {
|
|
193
|
-
console.log(
|
|
159
|
+
console.log(`ERROR: Target ${what} in batch missing actGroups or url property`);
|
|
194
160
|
}
|
|
195
161
|
});
|
|
196
162
|
return jobs;
|
package/package.json
CHANGED
|
@@ -20,10 +20,18 @@
|
|
|
20
20
|
<caption>Synopsis</caption>
|
|
21
21
|
<tr><th>Page</th><td>__org__</td></tr>
|
|
22
22
|
<tr><th>URL</th><td>__url__</td></tr>
|
|
23
|
-
<tr><th>Requester</th><td>__requester__</td></tr>
|
|
24
|
-
<tr><th>Device</th><td>__device__</td></tr>
|
|
25
23
|
<tr><th>Test date</th><td>__dateSlash__</td></tr>
|
|
26
24
|
<tr><th>Score</th><td>__total__</td></tr>
|
|
25
|
+
</table>
|
|
26
|
+
<table class="allBorder">
|
|
27
|
+
<caption>Configuration</caption>
|
|
28
|
+
<tr><th>Redirections</th><td>__redirections__</td></tr>
|
|
29
|
+
<tr><th>Tool isolation</th><td>__isolation__</td></tr>
|
|
30
|
+
<tr><th>Report format(s)</th><td>__standard__</td></tr>
|
|
31
|
+
<tr><th>Requester</th><td>__requester__</td></tr>
|
|
32
|
+
<tr><th>Device</th><td>__device__</td></tr>
|
|
33
|
+
<tr><th>Browser type</th><td>__browser__</td></tr>
|
|
34
|
+
<tr><th>Reduced motion</th><td>__motion__</td></tr>
|
|
27
35
|
<tr><th>Tested by</th><td>Testaro__agent__, procedure <code>__ts__</code></td></tr>
|
|
28
36
|
<tr><th>Scored by</th><td>Testilo, procedure <code>__sp__</code></td></tr>
|
|
29
37
|
<tr><th>Digested by</th><td>Testilo, procedure <code>__dp__</code></td></tr>
|
|
@@ -57,7 +57,7 @@ const getIssueScoreRow = (issueConstants, issueDetails) => {
|
|
|
57
57
|
};
|
|
58
58
|
// Adds parameters to a query for a digest.
|
|
59
59
|
const populateQuery = (report, query) => {
|
|
60
|
-
const {
|
|
60
|
+
const {browserID, deviceID, id, isolate, lowMotion, score, sources, standard, strict} = report;
|
|
61
61
|
const {agent, script, target, requester} = sources;
|
|
62
62
|
const {scoreProcID, summary, details} = score;
|
|
63
63
|
query.ts = script;
|
|
@@ -67,16 +67,16 @@ const populateQuery = (report, query) => {
|
|
|
67
67
|
query.dateISO = getNowDate();
|
|
68
68
|
query.dateSlash = getNowDateSlash();
|
|
69
69
|
query.org = target.what;
|
|
70
|
-
query.url = target.
|
|
70
|
+
query.url = target.url;
|
|
71
|
+
query.redirections = strict ? 'prohibited' : 'permitted';
|
|
72
|
+
query.isolation = isolate ? 'yes' : 'no';
|
|
73
|
+
query.standard
|
|
74
|
+
= ['original', 'standard', 'original and standard'][['no', 'only', 'also'].indexOf(standard)];
|
|
75
|
+
query.motion = lowMotion ? 'requested' : 'not requested';
|
|
71
76
|
query.requester = requester;
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
}
|
|
76
|
-
else {
|
|
77
|
-
query.device = 'unknown';
|
|
78
|
-
}
|
|
79
|
-
query.agent = ` on agent ${agent}` || '';
|
|
77
|
+
query.device = deviceID;
|
|
78
|
+
query.browser = browserID;
|
|
79
|
+
query.agent = agent ? ` on agent ${agent}` : '';
|
|
80
80
|
query.reportURL = process.env.SCORED_REPORT_URL.replace('__id__', id);
|
|
81
81
|
// Add values for the score-summary table to the query.
|
|
82
82
|
const rows = {
|
|
@@ -134,11 +134,14 @@ const populateQuery = (report, query) => {
|
|
|
134
134
|
issueDetailRows.push(`<h5>Rule <code>${ruleID}</code></h5>`);
|
|
135
135
|
issueDetailRows.push(`<p>Description: ${ruleData.what}</p>`);
|
|
136
136
|
issueDetailRows.push(`<p>Count of instances: ${ruleData.complaints.countTotal}</p>`);
|
|
137
|
+
/*
|
|
138
|
+
This fails unless the caller handles such URLs and has compatible scored report URLs.
|
|
137
139
|
const href = `${query.reportURL}?tool=${toolID}&rule=${ruleID}`;
|
|
138
140
|
const detailLabel = `Issue ${issueSummary} tool ${toolID} rule ${ruleID} instance details`;
|
|
139
141
|
issueDetailRows.push(
|
|
140
142
|
`<p><a href="${href}" aria-label="${detailLabel}">Instance details</a></p>`
|
|
141
143
|
);
|
|
144
|
+
*/
|
|
142
145
|
});
|
|
143
146
|
});
|
|
144
147
|
});
|
|
@@ -77,7 +77,7 @@ const populateQuery = async (id, what, summaryReport, query) => {
|
|
|
77
77
|
const scoreCell = `<td><a href=${digestLinkDestination}>${result.score}</a></td>`;
|
|
78
78
|
// Create a target cell.
|
|
79
79
|
const {target} = result.sources;
|
|
80
|
-
const targetLink = `<a href="${target.
|
|
80
|
+
const targetLink = `<a href="${target.url}">${target.what}</a>`;
|
|
81
81
|
const targetCell = `<td>${targetIDs[target.what]}: ${targetLink}</td>`;
|
|
82
82
|
const row = `<tr>${[timeCell, scoreCell, targetCell].join('')}</tr>`;
|
|
83
83
|
// Add the row to the array of rows.
|
package/script.js
CHANGED
|
@@ -111,18 +111,35 @@ exports.script = (id, what, options = {}) => {
|
|
|
111
111
|
});
|
|
112
112
|
}
|
|
113
113
|
// Initialize a script.
|
|
114
|
-
const timeLimit = Math.round(50 + 30 * Object.keys(toolsRulesData).length);
|
|
115
114
|
const scriptObj = {
|
|
116
115
|
id,
|
|
117
116
|
what,
|
|
118
|
-
strict:
|
|
117
|
+
strict: false,
|
|
119
118
|
isolate: true,
|
|
120
|
-
|
|
121
|
-
|
|
119
|
+
standard: 'only',
|
|
120
|
+
observe: false,
|
|
121
|
+
deviceID: 'default',
|
|
122
|
+
browserID: 'webkit',
|
|
123
|
+
lowMotion: false,
|
|
124
|
+
timeLimit: Math.round(50 + 30 * Object.keys(toolsRulesData).length),
|
|
125
|
+
creationTimeStamp: '',
|
|
126
|
+
executionTimeStamp: '',
|
|
127
|
+
sources: {
|
|
128
|
+
script: id,
|
|
129
|
+
batch: '',
|
|
130
|
+
target: {
|
|
131
|
+
what: '',
|
|
132
|
+
url: ''
|
|
133
|
+
},
|
|
134
|
+
lastTarget: false,
|
|
135
|
+
mergeID: '',
|
|
136
|
+
sendReportTo: '',
|
|
137
|
+
requester: ''
|
|
138
|
+
},
|
|
139
|
+
acts: [
|
|
122
140
|
{
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
"launch": "webkit"
|
|
141
|
+
type: 'placeholder',
|
|
142
|
+
which: 'main'
|
|
126
143
|
}
|
|
127
144
|
]
|
|
128
145
|
};
|