testilo 22.1.0 → 23.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +143 -151
- package/batch.js +24 -21
- package/call.js +20 -52
- package/merge.js +38 -87
- package/package.json +1 -1
- package/procs/test.js +20 -0
- package/procs/util.js +63 -0
- package/script.js +1 -2
- package/series.js +0 -72
- package/validation/series/job.json +0 -37
- package/validation/series/validate.js +0 -78
package/README.md
CHANGED
|
@@ -5,7 +5,7 @@ Utilities for Testaro
|
|
|
5
5
|
|
|
6
6
|
The Testilo package contains utilities that facilitate the use of the [Testaro](https://www.npmjs.com/package/testaro) package.
|
|
7
7
|
|
|
8
|
-
Testaro performs
|
|
8
|
+
Testaro performs jobs and creates reports in JSON format. The utilities in Testilo fall into two categories:
|
|
9
9
|
- Job preparation
|
|
10
10
|
- Report enhancement
|
|
11
11
|
|
|
@@ -21,24 +21,34 @@ When Testilo is a dependency of another application, the `.env` file is not impo
|
|
|
21
21
|
|
|
22
22
|
Testilo is written in Node.js. Commands are given to Testilo in a command-line (terminal) interface or programmatically.
|
|
23
23
|
|
|
24
|
-
Shared routines
|
|
24
|
+
Shared routines, called _procs_, are located in the `procs` directory.
|
|
25
25
|
|
|
26
26
|
Testilo can be installed wherever Node.js (version 14 or later) is installed. This can be a server or the same workstation on which Testaro is installed.
|
|
27
27
|
|
|
28
|
-
The reason for Testilo being an independent package, rather than part of Testaro, is that Testilo can be installed on any host, while Testaro can run successfully only on a Windows
|
|
28
|
+
The reason for Testilo being an independent package, rather than part of Testaro, is that Testilo can be installed on any host, while Testaro can run successfully only on a Windows, Macintosh, Ubuntu, or Debian workstation. Testaro runs tests similar to those that a human accessibility tester would run, using whatever browsers, input devices, system settings, simulated and attached devices, and assistive technologies tests may require. Thus, Testaro is limited to functionalities that require workstation attributes. For maximum flexibility in the management of Testaro jobs, all other functionalities are located outside of Testaro. You could have software such as Testilo running on a server, communicating with multiple workstations running Testaro. The workstations could receive jobs from the server and return job reports to the server for further processing.
|
|
29
29
|
|
|
30
30
|
## Configuration
|
|
31
31
|
|
|
32
32
|
Environment variables for Testilo can be specified in a `.env` file. An example:
|
|
33
33
|
|
|
34
34
|
```bash
|
|
35
|
-
FUNCTIONDIR
|
|
35
|
+
FUNCTIONDIR=./procs
|
|
36
36
|
JOBDIR=../testdir/jobs
|
|
37
37
|
REPORTDIR=../testdir/reports
|
|
38
38
|
REQUESTER=a11ymgr@a11yorg.com
|
|
39
39
|
SPECDIR=../testdir/specs
|
|
40
40
|
```
|
|
41
41
|
|
|
42
|
+
The `FUNCTIONDIR` environment variable typically references the `procs` directory, but it could reference a different directory in the filesystem where Testilo resides, if you wanted to customize the procs that Testilo uses.
|
|
43
|
+
|
|
44
|
+
`JOBDIR` references a directory in the filesystem where jobs created by the `merge` proc are to be saved.
|
|
45
|
+
|
|
46
|
+
`REPORTDIR` references a directory in the filesystem where reports are saved.
|
|
47
|
+
|
|
48
|
+
`REQUESTER` is an email address that will be used as a job property if no other email address is specified for the `sources.requester` property of the job.
|
|
49
|
+
|
|
50
|
+
`SPECDIR` references a directory in the filesystem where tanrget lists, batches, and scripts can be found. Those are raw materials from which Testaro creates jobs.
|
|
51
|
+
|
|
42
52
|
## Job preparation
|
|
43
53
|
|
|
44
54
|
### Introduction
|
|
@@ -49,20 +59,30 @@ You can create a job for Testaro directly, without using Testilo.
|
|
|
49
59
|
|
|
50
60
|
Testilo can, however, make job preparation more efficient in these scenarios:
|
|
51
61
|
- You want to perform a battery of tests on multiple targets.
|
|
52
|
-
- You want to test targets for particular issues, using whichever tools happen to have tests for those issues.
|
|
62
|
+
- You want to test targets only for particular issues, using whichever tools happen to have tests for those issues.
|
|
53
63
|
|
|
54
64
|
### Target lists
|
|
55
65
|
|
|
56
|
-
The simplest version of a list of targets is a _target list_. It is an array of arrays defining 1 or more targets. It
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
-
|
|
66
|
+
The simplest version of a list of targets is a _target list_. It is an array of arrays defining 1 or more targets. It can be stored as a tab-delimited text file.
|
|
67
|
+
|
|
68
|
+
A target is defined by 2 items:
|
|
69
|
+
- A description
|
|
70
|
+
- A URL
|
|
60
71
|
|
|
61
|
-
For example, a
|
|
72
|
+
For example, a target list might be:
|
|
73
|
+
|
|
74
|
+
```javaScript
|
|
75
|
+
[
|
|
76
|
+
['World Wide Web Consortium', 'https://www.w3.org/'],
|
|
77
|
+
['Mozilla Foundation', 'https://foundation.mozilla.org/en/']
|
|
78
|
+
]
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
If this target list were stored as a file, its content would be this (with “→” representing the Tab character):
|
|
62
82
|
|
|
63
83
|
```text
|
|
64
|
-
|
|
65
|
-
|
|
84
|
+
World Wide Web Consortium→https://www.w3.org/
|
|
85
|
+
Mozilla Foundation→https://foundation.mozilla.org/en/
|
|
66
86
|
```
|
|
67
87
|
|
|
68
88
|
### Batches
|
|
@@ -76,27 +96,21 @@ Targets can be specified in a more complex way, too. That allows you to create j
|
|
|
76
96
|
targets: [
|
|
77
97
|
{
|
|
78
98
|
id: 'acme',
|
|
79
|
-
which: 'https://acmeclothes.com/',
|
|
80
99
|
what: 'Acme Clothes',
|
|
100
|
+
which: 'https://acmeclothes.com/',
|
|
81
101
|
acts: {
|
|
82
102
|
public: [
|
|
83
103
|
{
|
|
84
|
-
type: 'launch'
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
type: 'url',
|
|
88
|
-
which: 'https://acmeclothes.com/',
|
|
89
|
-
what: 'Acme Clothes home page'
|
|
104
|
+
type: 'launch',
|
|
105
|
+
what: 'Acme Clothes home page',
|
|
106
|
+
url: 'https://acmeclothes.com/'
|
|
90
107
|
}
|
|
91
108
|
],
|
|
92
109
|
private: [
|
|
93
110
|
{
|
|
94
|
-
type: 'launch'
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
type: 'url',
|
|
98
|
-
which: 'https://acmeclothes.com/login.html',
|
|
99
|
-
what: 'Acme Clothes login page'
|
|
111
|
+
type: 'launch',
|
|
112
|
+
what: 'Acme Clothes login page',
|
|
113
|
+
url: 'https://acmeclothes.com/login.html'
|
|
100
114
|
},
|
|
101
115
|
{
|
|
102
116
|
type: 'text',
|
|
@@ -128,9 +142,11 @@ Targets can be specified in a more complex way, too. That allows you to create j
|
|
|
128
142
|
|
|
129
143
|
As shown, a batch, unlike a target list, defines named sequences of acts. They can be plugged into jobs, so various complex operations can be performed on each target.
|
|
130
144
|
|
|
145
|
+
A batch is a JavaScript object. It can be converted to JSON and stored in a file.
|
|
146
|
+
|
|
131
147
|
### Scripts
|
|
132
148
|
|
|
133
|
-
The generic, target-independent description of a job is _script_. A script can contain _placeholders_ that Testilo replaces with acts from a batch, creating one job per target. Thus, one script plus
|
|
149
|
+
The generic, target-independent description of a job is _script_. A script can contain _placeholders_ that Testilo replaces with acts from a batch, creating one job per target. Thus, one script plus a batch containing _n_ targets will generate _n_ jobs.
|
|
134
150
|
|
|
135
151
|
Here is a script:
|
|
136
152
|
|
|
@@ -143,7 +159,6 @@ Here is a script:
|
|
|
143
159
|
timeLimit: 60,
|
|
144
160
|
standard: 'also',
|
|
145
161
|
observe: false,
|
|
146
|
-
timeStamp: '240115T1200',
|
|
147
162
|
acts: [
|
|
148
163
|
{
|
|
149
164
|
type: 'placeholder',
|
|
@@ -169,18 +184,18 @@ Here is a script:
|
|
|
169
184
|
```
|
|
170
185
|
|
|
171
186
|
A script has several properties that specify facts about the jobs to be created. They include:
|
|
172
|
-
- `id`: an ID
|
|
187
|
+
- `id`: an ID. A script can be converted from a JavaScript object to JSON and saved in a file in the `SPECDIR` directory, where it will be named by its ID (e.g., if the ID is `ts99`, the file name will be `ts99.json`). Thus, each script needs an `id` with a unique value.
|
|
173
188
|
- `what`: a description of the script.
|
|
174
|
-
- `strict`:
|
|
175
|
-
- `isolate`:
|
|
176
|
-
- `
|
|
177
|
-
- `
|
|
178
|
-
- `
|
|
189
|
+
- `strict`: `true` if Testaro is to abort jobs when a target redirects a request to a URL differing substantially from the one specified. If `false` Testaro is to allow redirection. All differences are considered substantial unless the URLs differ only in the presence and absence of a trailing slash.
|
|
190
|
+
- `isolate`: If `true`, Testilo, before creating a job, will isolate test acts, as needed, from effects of previous test acts, by inserting a copy of the latest placeholder after each target-modifying test act other than the final act. If `false`, placeholders will not be duplicated.
|
|
191
|
+
- `timeLimit`: This specifies the maximum duration, in seconds, of a job. Testaro will abort jobs that are not completed within that time.
|
|
192
|
+
- `standard`: If `also`, jobs will tell Testaro to include in its reports both the original results of the tests of tools and the Testaro-standardized results. If `only`, reports are to include only the standardized test results. If `no`, reports are to include only the original results, without standardization.
|
|
193
|
+
- `observe`: If `true`, jobs will tell Testaro to allow granular observation of job progress. If `false`, jobs will tell Testaro not to permit granular observation, but only to send the report to the server when the report is completed. It is generally user-friendly to allow granular observation, and for user applications to implement it, if they make users wait while jobs are assigned and performed, since that process typically takes about 3 minutes.
|
|
179
194
|
- `acts`: an array of acts.
|
|
180
195
|
|
|
181
196
|
The first act in this example script is a placeholder, whose `which` property is `'private'`. If the above batch were merged with this script, in each job the placeholder would be replaced with the `private` acts of a target. For example, the first act of the first job would launch a Chromium browser, navigate to the Acme login page, complete and submit the login form, wait for the account page to load, run the Axe tests, and then run the QualWeb tests. If the batch contained additional targets, additional jobs would be created, with the login actions for each target specified in the `private` array of the `acts` object of that target.
|
|
182
197
|
|
|
183
|
-
As shown in this example, when a browser is launched by placeholder substitution, the script can determine the browser type (`chromium`, `firefox`, or `webkit`) by assigning a value to a `launch` property of the placeholder.
|
|
198
|
+
As shown in this example, when a browser is launched by placeholder substitution, the script can determine the browser type (`chromium`, `firefox`, or `webkit`) by assigning a value to a `launch` property of the placeholder. This allows a script to ensure that the tests it requires are performed with appropriate browser types. Some browser types are incompatible with some tests.
|
|
184
199
|
|
|
185
200
|
### Target list to batch
|
|
186
201
|
|
|
@@ -196,33 +211,35 @@ A module can invoke `batch` in this way:
|
|
|
196
211
|
|
|
197
212
|
```javaScript
|
|
198
213
|
const {batch} = require('testilo/batch');
|
|
199
|
-
const batchObj = batch(
|
|
214
|
+
const batchObj = batch(id, what, targets);
|
|
200
215
|
```
|
|
201
216
|
|
|
202
|
-
This invocation references `
|
|
217
|
+
This invocation references `id`, `what`, and `targets` variables that the module must have already defined. The `id` variable is a unique identifier for the target list. The `what` variable describes the target list. The `targets` variable is an array of arrays, with each array containing the 2 items (description and URL) defining one target.
|
|
203
218
|
|
|
204
219
|
The `batch()` function of the `batch` module generates a batch and returns it as an object. The invoking module can further dispose of the batch as needed.
|
|
205
220
|
|
|
221
|
+
The ID assigned to each target by the `batch()` function is a sequential (base-62 alphanumeric) string, rather than a mnemonic like the one (`'acme'`) in the above example.
|
|
222
|
+
|
|
206
223
|
##### By a user
|
|
207
224
|
|
|
208
225
|
A user can invoke `batch` in this way:
|
|
209
226
|
|
|
210
|
-
- Create a target list and save it as a text file (with tab-delimited items in newline-delimited lines) in the `targetLists` subdirectory of the `
|
|
211
|
-
- In the Testilo project directory, execute the statement `node call batch
|
|
227
|
+
- Create a target list and save it as a text file (with tab-delimited items in newline-delimited lines) in the `targetLists` subdirectory of the `SPECDIR` directory. Name the file `x.tsv`, where `x` is the list ID.
|
|
228
|
+
- In the Testilo project directory, execute the statement `node call batch id what`.
|
|
212
229
|
|
|
213
|
-
In this statement, replace `
|
|
230
|
+
In this statement, replace `id` with the list ID and `what` with a string describing the batch.
|
|
214
231
|
|
|
215
232
|
The `call` module will retrieve the named target list.
|
|
216
233
|
The `batch` module will convert the target list to a batch.
|
|
217
|
-
The `call` module will save the batch as a JSON file in the `batches` subdirectory of the `
|
|
234
|
+
The `call` module will save the batch as a JSON file in the `batches` subdirectory of the `SPECDIR` directory.
|
|
218
235
|
|
|
219
236
|
### Issues to script
|
|
220
237
|
|
|
221
|
-
Testilo
|
|
238
|
+
Testilo contains a classification of tool rules into _issues_. It is located in the `procs/score` directory and has a file name starting with `tic` (Testilo issue classification). You can create custom classifications and save them in a `score` subdirectory of the `FUNCTIONDIR` directory.
|
|
222
239
|
|
|
223
240
|
For example, one of the issues in the `tic40.js` file is `mainNot1`. Four rules are classified as belonging to that issue: rule `main_element_only_one` of `aslint` and 3 more rules defined by 3 other tools.
|
|
224
241
|
|
|
225
|
-
If you want Testaro to test targets for only particular issues, you can
|
|
242
|
+
If you want Testaro to test targets for only particular issues, you can use the `script` module to create a script. Jobs created from that script will make Testaro test for only the issues you specify to the `script` module.
|
|
226
243
|
|
|
227
244
|
If you want Testaro to test targets for **all** the rules of all the available tools, you can use the `script` module to create a script that does not impose any issue restrictions.
|
|
228
245
|
|
|
@@ -240,9 +257,9 @@ const scriptObj = script(scriptID, issues, issueID0, issueID1, …);
|
|
|
240
257
|
```
|
|
241
258
|
|
|
242
259
|
This invocation references `scriptID`, `issues`, and `issueID` variables.
|
|
243
|
-
- The `scriptID` variable
|
|
260
|
+
- The `scriptID` variable specifies the ID that the script will have.
|
|
244
261
|
- The `issues` variable (if present) is an object that classifies issues, such as the `issues` object in a `tic` file.
|
|
245
|
-
- The `issueID` variables (if any) are strings, such as `'regionNoText'`, that name properties of the `issues` object.
|
|
262
|
+
- The `issueID` variables (if any) are strings, such as `'regionNoText'`, that name issues, i.e. properties of the `issues` object, that you want jobs from the script to test for.
|
|
246
263
|
|
|
247
264
|
The `script()` function of the `script` module generates a script and returns it as an object. The invoking module can further modify and use the script as needed.
|
|
248
265
|
|
|
@@ -255,50 +272,49 @@ const scriptObj = script(scriptID);
|
|
|
255
272
|
|
|
256
273
|
##### By a user
|
|
257
274
|
|
|
258
|
-
A user can invoke `script` in this way: In the Testilo project directory, execute the statement `node call script
|
|
275
|
+
A user can invoke `script` in this way: In the Testilo project directory, execute the statement `node call script id ticnn issuea issueb …`.
|
|
259
276
|
|
|
260
277
|
In this statement:
|
|
261
|
-
- Replace `
|
|
262
|
-
- Replace `
|
|
263
|
-
- Replace the remaining arguments (`
|
|
278
|
+
- Replace `id` with an ID for the script, such as `headings`.
|
|
279
|
+
- Replace `ticnn` with the base, such as `tic99`, of the name of an issue classification file in the `score` subdirectory of the `FUNCTIONDIR` directory.
|
|
280
|
+
- Replace the remaining arguments (`issuea` etc.) with issue names from that classification file.
|
|
264
281
|
|
|
265
282
|
The `call` module will retrieve the named classification.
|
|
266
283
|
The `script` module will create a script.
|
|
267
|
-
The `call` module will save the script as a JSON file in the `scripts` subdirectory of the `
|
|
284
|
+
The `call` module will save the script as a JSON file in the `scripts` subdirectory of the `SPECDIR` directory.
|
|
268
285
|
|
|
269
|
-
To create a script without any issue restrictions, a user can execute the statement `node call script
|
|
286
|
+
To create a script without any issue restrictions, a user can execute the statement `node call script id`.
|
|
270
287
|
|
|
271
288
|
#### Options
|
|
272
289
|
|
|
273
|
-
|
|
290
|
+
The `script` module will use the value of the `SEND_REPORT_TO` environment variable as the value of the `sendReportTo` property of the script, if that variable exists, and otherwise will leave that property with an empty-string value.
|
|
291
|
+
|
|
292
|
+
When the `script` module creates a script for you, it does not ask you for all of the options that the script may require. Instead, it chooses default options. After you invoke `script`, you can edit the script that it creates to revise options.
|
|
274
293
|
|
|
275
294
|
### Merge
|
|
276
295
|
|
|
277
296
|
Testilo merges batches with scripts, producing jobs, by means of the `merge` module.
|
|
278
297
|
|
|
279
|
-
The `merge` module needs to be given a batch and a script.
|
|
280
|
-
|
|
281
298
|
#### Output
|
|
282
299
|
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
Suppose you ask for a merger of the above batch and script, **without** the isolation option. Then the first job produced by `merge` will look like this:
|
|
300
|
+
Suppose you ask for a merger of the above batch and script. Then the first job produced by `merge` will look like this:
|
|
286
301
|
|
|
287
302
|
```javaScript
|
|
288
303
|
{
|
|
289
|
-
id: '
|
|
290
|
-
what: '
|
|
304
|
+
id: '240115T1200-4Rw-acme',
|
|
305
|
+
what: 'aside mislocation',
|
|
291
306
|
strict: true,
|
|
292
307
|
timeLimit: 60,
|
|
308
|
+
standard: 'also',
|
|
309
|
+
observe: false,
|
|
310
|
+
sendReportTo: 'https://ourdomain.com/testman/api/report'
|
|
311
|
+
timeStamp: '240115T1200',
|
|
293
312
|
acts: [
|
|
294
313
|
{
|
|
295
314
|
type: 'launch',
|
|
296
|
-
which: 'chromium'
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
type: 'url',
|
|
300
|
-
which: 'https://acmeclothes.com/login.html',
|
|
301
|
-
what: 'Acme Clothes login page'
|
|
315
|
+
which: 'chromium',
|
|
316
|
+
what: 'Acme Clothes login page',
|
|
317
|
+
url: 'https://acmeclothes.com/login.html'
|
|
302
318
|
},
|
|
303
319
|
{
|
|
304
320
|
type: 'text',
|
|
@@ -323,15 +339,42 @@ Suppose you ask for a merger of the above batch and script, **without** the isol
|
|
|
323
339
|
{
|
|
324
340
|
type: 'test',
|
|
325
341
|
which: 'axe',
|
|
326
|
-
detailLevel:
|
|
327
|
-
rules: [],
|
|
328
|
-
what: 'Axe
|
|
342
|
+
detailLevel: 2,
|
|
343
|
+
rules: ['landmark-complementary-is-top-level'],
|
|
344
|
+
what: 'Axe'
|
|
345
|
+
},
|
|
346
|
+
{
|
|
347
|
+
type: 'launch',
|
|
348
|
+
which: 'chromium',
|
|
349
|
+
what: 'Acme Clothes login page',
|
|
350
|
+
url: 'https://acmeclothes.com/login.html'
|
|
351
|
+
},
|
|
352
|
+
{
|
|
353
|
+
type: 'text',
|
|
354
|
+
which: 'User Name',
|
|
355
|
+
what: 'tester34'
|
|
356
|
+
},
|
|
357
|
+
{
|
|
358
|
+
type: 'text',
|
|
359
|
+
which: 'Password',
|
|
360
|
+
what: '34SecretTester'
|
|
361
|
+
},
|
|
362
|
+
{
|
|
363
|
+
type: 'button',
|
|
364
|
+
which: 'Submit',
|
|
365
|
+
what: 'submit the login form'
|
|
366
|
+
},
|
|
367
|
+
{
|
|
368
|
+
type: 'wait',
|
|
369
|
+
which: 'title',
|
|
370
|
+
what: 'account'
|
|
329
371
|
},
|
|
330
372
|
{
|
|
331
373
|
type: 'test',
|
|
332
374
|
which: 'qualWeb',
|
|
333
375
|
withNewContent: false,
|
|
334
|
-
|
|
376
|
+
rules: ['QW-BP25', 'QW-BP26']
|
|
377
|
+
what: 'QualWeb'
|
|
335
378
|
}
|
|
336
379
|
],
|
|
337
380
|
sources: {
|
|
@@ -343,32 +386,19 @@ Suppose you ask for a merger of the above batch and script, **without** the isol
|
|
|
343
386
|
},
|
|
344
387
|
requester: 'you@yourdomain.tld'
|
|
345
388
|
},
|
|
346
|
-
creationTime: '
|
|
347
|
-
timeStamp: '231120T155314'
|
|
389
|
+
creationTime: '241120T1550'
|
|
348
390
|
}
|
|
349
391
|
```
|
|
350
392
|
|
|
351
393
|
Testilo has substituted the `private` acts from the `acme` target of the batch for the placeholder when creating the job. Testilo also has:
|
|
394
|
+
- inserted a copy of those same acts after the `axe` test act, because `axe` is a target-modifying tool.
|
|
352
395
|
- let the script determine the browser type of the `launch` act.
|
|
353
|
-
-
|
|
354
|
-
-
|
|
355
|
-
-
|
|
356
|
-
- inserted a `sources` property into the job, recording facts about the script, the batch, the target, and the email address given by the user who requested the merger.
|
|
396
|
+
- given the job an ID that combines the time stamp with a differentiator and the batch ID.
|
|
397
|
+
- inserted a `sources` property into the job, recording facts about the script, the batch, the target, the requester, and the report URL.
|
|
398
|
+
- added a time stamp describing the creation time to the job.
|
|
357
399
|
|
|
358
400
|
This is a valid Testaro job.
|
|
359
401
|
|
|
360
|
-
##### With isolation
|
|
361
|
-
|
|
362
|
-
If, however, you requested a merger **with** isolation, then Testilo would take cognizance of the fact that an `axe` test act is a target-modifying act. Testilo would therefore act as if another instance of the placeholder had been located in the script after the `axe` test act. So, copies of the same 6 acts that precede the `axe` test act would be inserted **after** the `axe` test act, too.
|
|
363
|
-
|
|
364
|
-
Of the 9 tools providing tests for Testaro, 6 are target-modifying:
|
|
365
|
-
- `alfa`
|
|
366
|
-
- `aslint`
|
|
367
|
-
- `axe`
|
|
368
|
-
- `htmlcs`
|
|
369
|
-
- `ibm`
|
|
370
|
-
- `testaro`
|
|
371
|
-
|
|
372
402
|
#### Invocation
|
|
373
403
|
|
|
374
404
|
There are two ways to use the `merge` module.
|
|
@@ -379,10 +409,16 @@ A module can invoke `merge` in this way:
|
|
|
379
409
|
|
|
380
410
|
```javaScript
|
|
381
411
|
const {merge} = require('testilo/merge');
|
|
382
|
-
const jobs = merge(script, batch, requester,
|
|
412
|
+
const jobs = merge(script, batch, requester, timeStamp);
|
|
383
413
|
```
|
|
384
414
|
|
|
385
|
-
|
|
415
|
+
The `merge` module uses these 4 arguments to create jobs from a script and a batch.
|
|
416
|
+
|
|
417
|
+
The arguments are:
|
|
418
|
+
- `script`: a script.
|
|
419
|
+
- `batch`: a batch.
|
|
420
|
+
- `requester`: an email address.
|
|
421
|
+
- `timeStamp`: the earliest UTC date and time when the jobs may be assigned (format `240415T1230`), or an empty string if now.
|
|
386
422
|
|
|
387
423
|
The `merge()` function of the `merge` module generates jobs and returns them in an array. The invoking module can further dispose of the jobs as needed.
|
|
388
424
|
|
|
@@ -390,75 +426,23 @@ The `merge()` function of the `merge` module generates jobs and returns them in
|
|
|
390
426
|
|
|
391
427
|
A user can invoke `merge` in this way:
|
|
392
428
|
|
|
393
|
-
- Create a script and save it as a JSON file in the `scripts` subdirectory of the `
|
|
394
|
-
- Create a batch and save it as a JSON file in the `batches` subdirectory of the `
|
|
395
|
-
- In the Testilo project directory, execute
|
|
396
|
-
- `node call merge scriptName batchName email isolate standard granular todoDir pre post`
|
|
397
|
-
|
|
398
|
-
In these statements, replace:
|
|
399
|
-
- `scriptName` with the base name of the script file
|
|
400
|
-
- `batchName` with the base name of the batch file
|
|
401
|
-
- `email` with an email address, or with an empty string if the environment variable `process.env.REQUESTER` exists and you want to use it
|
|
402
|
-
- `isolate` with `true` if you want test isolation or `false` if not
|
|
403
|
-
- `standard` with `'also'`, `'only'`, or `'no'` to specify the treatment of standard-format results.
|
|
404
|
-
- `granular` with `true` if granular observation is to be permitted, or `false` if not.
|
|
405
|
-
- `todoDir` with `true` if the job is to be saved in the `todo` subdirectory or `false` if it is to be saved in the `pending` subdirectory of the `process.env.JOBDIR` directory.
|
|
406
|
-
- `pre` with the pre-ID part of the report URL.
|
|
407
|
-
- `post` with the post-ID part of the report URL.
|
|
408
|
-
|
|
409
|
-
The `call` module will retrieve the named script and batch from their respective directories.
|
|
410
|
-
The `merge` module will create an array of jobs, with or without test isolation.
|
|
411
|
-
The `call` module will save the jobs as JSON files in the `todo` or `pending` subdirectory of the `process.env.JOBDIR` directory.
|
|
412
|
-
|
|
413
|
-
#### Validation
|
|
414
|
-
|
|
415
|
-
To test the `merge` module, in the project directory you can execute the statement `node validation/merge/validate`. If `merge` is valid, all logging statements will begin with “Success” and none will begin with “ERROR”.
|
|
416
|
-
|
|
417
|
-
### Series
|
|
418
|
-
|
|
419
|
-
If you want to monitor a web resource by performing identical jobs repeatedly and comparing the results, you can use the `series` module to create a series of identical jobs.
|
|
420
|
-
|
|
421
|
-
The jobs in a series differ from one another only in the timestamp segments of their `id` properties. For example, if the first job had the `id` value `240528T1316-mon-mozilla` and the events in the series occurred at intervals of 12 hours, then the second job would have the `id` value `240529T0116-mon-mozilla`.
|
|
422
|
-
|
|
423
|
-
The `series` module adds a `sources.series` property to each job in the series. The value of that property is the `id` value of the first job in the series.
|
|
424
|
-
|
|
425
|
-
To support monitoring, a server that receives job requests from testing agents can perform a time check on the first job in the queue. If the time specified by the `id` of the first job is in the future, the server can reply that there is no job to do.
|
|
426
|
-
|
|
427
|
-
#### Invocation
|
|
428
|
-
|
|
429
|
-
There are two ways to use the `series` module.
|
|
430
|
-
|
|
431
|
-
##### By a module
|
|
432
|
-
|
|
433
|
-
A module can invoke `series` in this way:
|
|
434
|
-
|
|
435
|
-
```javaScript
|
|
436
|
-
const {series} = require('testilo/series');
|
|
437
|
-
const jobs = series(job, count, interval);
|
|
438
|
-
```
|
|
439
|
-
|
|
440
|
-
This invocation references a `job` variable, whose value is a job object. The `count` variable is an integer, 2 or greater, specifying how many events the series consists of. The `interval` variable is an integer, 1 or greater, specifying how many minutes are to elapse after each event before the next event. The `series()` function of the `series` module generates an array of job objects and returns the array. The invoking module can further dispose of the jobs as needed.
|
|
441
|
-
|
|
442
|
-
##### By a user
|
|
443
|
-
|
|
444
|
-
A user can invoke `series` in this way:
|
|
445
|
-
|
|
446
|
-
- Create a job and save it as a JSON file in the `todo` subdirectory of the `process.env.JOBDIR` directory.
|
|
447
|
-
- In the Testilo project directory, execute this statement:
|
|
448
|
-
- `node call series j c i`
|
|
429
|
+
- Create a script and save it as a JSON file in the `scripts` subdirectory of the `SPECDIR` directory.
|
|
430
|
+
- Create a batch and save it as a JSON file in the `batches` subdirectory of the `SPECDIR` directory.
|
|
431
|
+
- In the Testilo project directory, execute the statement `node call merge scriptID batchID requester timeStamp todoDir`.
|
|
449
432
|
|
|
450
433
|
In this statement, replace:
|
|
451
|
-
- `
|
|
452
|
-
- `
|
|
453
|
-
- `
|
|
434
|
+
- `scriptID` with the ID (which is also the base of the file name) of the script.
|
|
435
|
+
- `batchID` with the ID (which is also the base of the file name) of the batch.
|
|
436
|
+
- `requester` and `timeStamp` as described above.
|
|
437
|
+
- `todoDir`: `true` if the jobs are to be saved in the `todo` subdirectory, or `false` if they are to be saved in the `pending` subdirectory, of the `JOBDIR` directory.
|
|
454
438
|
|
|
455
|
-
The `call` module will retrieve the
|
|
456
|
-
The `
|
|
457
|
-
The `call` module will save the jobs as JSON files in the `todo` subdirectory of the `
|
|
439
|
+
The `call` module will retrieve the named script and batch from their respective directories.
|
|
440
|
+
The `merge` module will create an array of jobs.
|
|
441
|
+
The `call` module will save the jobs as JSON files in the `todo` or `pending` subdirectory of the `JOBDIR` directory.
|
|
458
442
|
|
|
459
443
|
#### Validation
|
|
460
444
|
|
|
461
|
-
To test the `
|
|
445
|
+
To test the `merge` module, in the project directory you can execute the statement `node validation/merge/validate`. If `merge` is valid, all logging statements will begin with “Success” and none will begin with “ERROR”.
|
|
462
446
|
|
|
463
447
|
## Report scoring
|
|
464
448
|
|
|
@@ -719,3 +703,11 @@ The third argument to `call` (`23pl` in this example) is optional. If it is omit
|
|
|
719
703
|
### Validation
|
|
720
704
|
|
|
721
705
|
To test the `compare` module, in the project directory you can execute the statement `node validation/compare/validate`. If `compare` is valid, all logging statements will begin with “Success” and none will begin with “ERROR”.
|
|
706
|
+
|
|
707
|
+
## Origin
|
|
708
|
+
|
|
709
|
+
Work on the functionalities of Testaro and Testilo began in 2017. It was named [Autotest](https://github.com/jrpool/autotest) in early 2021 and then partitioned into the more single-purpose packages Testaro and Testilo in January 2022.
|
|
710
|
+
|
|
711
|
+
## Etymology
|
|
712
|
+
|
|
713
|
+
“Testilo” means “testing tool” in Esperanto.
|
package/batch.js
CHANGED
|
@@ -3,7 +3,11 @@
|
|
|
3
3
|
Converts a target list to a batch.
|
|
4
4
|
*/
|
|
5
5
|
|
|
6
|
-
//
|
|
6
|
+
// IMPORTS
|
|
7
|
+
|
|
8
|
+
const {alphaNumOf} = require('./procs/util');
|
|
9
|
+
|
|
10
|
+
// FUNCTIONS
|
|
7
11
|
|
|
8
12
|
// Converts a target list to a batch and returns the batch.
|
|
9
13
|
exports.batch = (id, what, targetList) => {
|
|
@@ -17,9 +21,9 @@ exports.batch = (id, what, targetList) => {
|
|
|
17
21
|
&& targetList.length
|
|
18
22
|
&& targetList.every(
|
|
19
23
|
target => Array.isArray(target)
|
|
24
|
+
&& target.length === 2
|
|
20
25
|
&& target.every(item => typeof item === 'string')
|
|
21
26
|
)
|
|
22
|
-
&& targetList.some(target => target.length === 3)
|
|
23
27
|
) {
|
|
24
28
|
// Initialize the batch.
|
|
25
29
|
const batch = {
|
|
@@ -27,25 +31,23 @@ exports.batch = (id, what, targetList) => {
|
|
|
27
31
|
what,
|
|
28
32
|
targets: []
|
|
29
33
|
};
|
|
30
|
-
// For each
|
|
31
|
-
targetList.forEach(target => {
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
});
|
|
48
|
-
}
|
|
34
|
+
// For each target:
|
|
35
|
+
targetList.forEach((target, index) => {
|
|
36
|
+
// Add it to the batch.
|
|
37
|
+
batch.targets.push({
|
|
38
|
+
id: alphaNumOf(index),
|
|
39
|
+
what: target[0],
|
|
40
|
+
which: target[1],
|
|
41
|
+
acts: {
|
|
42
|
+
main: [
|
|
43
|
+
{
|
|
44
|
+
type: 'launch',
|
|
45
|
+
what: target[0],
|
|
46
|
+
url: target[1]
|
|
47
|
+
}
|
|
48
|
+
]
|
|
49
|
+
}
|
|
50
|
+
});
|
|
49
51
|
});
|
|
50
52
|
// Return the batch.
|
|
51
53
|
return batch;
|
|
@@ -53,6 +55,7 @@ exports.batch = (id, what, targetList) => {
|
|
|
53
55
|
// Otherwise, i.e. if the arguments are invalid:
|
|
54
56
|
else {
|
|
55
57
|
// Return this.
|
|
58
|
+
console.log('ERROR: information missing or invalid');
|
|
56
59
|
return null;
|
|
57
60
|
}
|
|
58
61
|
};
|
package/call.js
CHANGED
|
@@ -24,8 +24,6 @@ const {batch} = require('./batch');
|
|
|
24
24
|
const {script} = require('./script');
|
|
25
25
|
// Function to process a merger.
|
|
26
26
|
const {merge} = require('./merge');
|
|
27
|
-
// Function to generate a job series.
|
|
28
|
-
const {series} = require('./series');
|
|
29
27
|
// Function to score reports.
|
|
30
28
|
const {score} = require('./score');
|
|
31
29
|
// Function to digest reports.
|
|
@@ -45,16 +43,21 @@ const fnArgs = process.argv.slice(3);
|
|
|
45
43
|
// ########## FUNCTIONS
|
|
46
44
|
|
|
47
45
|
// Converts a target list to a batch.
|
|
48
|
-
const callBatch = async (
|
|
46
|
+
const callBatch = async (id, what) => {
|
|
49
47
|
// Get the target list.
|
|
50
|
-
const listString = await fs.readFile(`${specDir}/targetLists/${
|
|
51
|
-
const list = listString
|
|
48
|
+
const listString = await fs.readFile(`${specDir}/targetLists/${id}.tsv`, 'utf8');
|
|
49
|
+
const list = listString
|
|
50
|
+
.split('\n')
|
|
51
|
+
.filter(target => target.length)
|
|
52
|
+
.map(target => target.split('\t'));
|
|
52
53
|
// Convert it to a batch.
|
|
53
|
-
const batchObj = batch(
|
|
54
|
+
const batchObj = batch(id, what, list);
|
|
54
55
|
// Save the batch.
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
56
|
+
if (batchObj) {
|
|
57
|
+
const batchJSON = JSON.stringify(batchObj, null, 2);
|
|
58
|
+
await fs.writeFile(`${specDir}/batches/${id}.json`, `${batchJSON}\n`);
|
|
59
|
+
console.log(`Target list ${id} converted to a batch and saved in ${specDir}/batches`);
|
|
60
|
+
}
|
|
58
61
|
};
|
|
59
62
|
// Fulfills a script-creation request.
|
|
60
63
|
const callScript = async (scriptID, classificationID = null, ... issueIDs) => {
|
|
@@ -74,10 +77,8 @@ const callMerge = async (
|
|
|
74
77
|
scriptID,
|
|
75
78
|
batchID,
|
|
76
79
|
requester,
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
isGranular,
|
|
80
|
-
todo
|
|
80
|
+
timeStamp,
|
|
81
|
+
todoDir
|
|
81
82
|
) => {
|
|
82
83
|
// Get the script and the batch.
|
|
83
84
|
const scriptJSON = await fs.readFile(`${specDir}/scripts/${scriptID}.json`, 'utf8');
|
|
@@ -85,42 +86,15 @@ const callMerge = async (
|
|
|
85
86
|
const batchJSON = await fs.readFile(`${specDir}/batches/${batchID}.json`, 'utf8');
|
|
86
87
|
const batch = JSON.parse(batchJSON);
|
|
87
88
|
// Merge them into an array of jobs.
|
|
88
|
-
const jobs = merge(
|
|
89
|
-
script, batch, requester, withIsolation, standard, isGranular, null, urlPrefix, urlSuffix
|
|
90
|
-
);
|
|
89
|
+
const jobs = merge(script, batch, requester, timeStamp);
|
|
91
90
|
// Save the jobs.
|
|
92
|
-
const
|
|
91
|
+
const subdir = `${jobDir}/${todoDir === 'true' ? 'todo' : 'pending'}`;
|
|
93
92
|
for (const job of jobs) {
|
|
94
93
|
const jobJSON = JSON.stringify(job, null, 2);
|
|
95
|
-
await fs.writeFile(`${
|
|
96
|
-
}
|
|
97
|
-
const {timeStamp} = jobs[0];
|
|
98
|
-
console.log(
|
|
99
|
-
`Script ${scriptID} and batch ${batchID} merged as ${timeStamp}-… in ${jobDir}/${destination}`
|
|
100
|
-
);
|
|
101
|
-
};
|
|
102
|
-
// Fulfills a series request.
|
|
103
|
-
const callSeries = async (idStart, count, interval) => {
|
|
104
|
-
// Get the initial job.
|
|
105
|
-
const jobNames = await fs.readdir(`${jobDir}/pending`);
|
|
106
|
-
const seriesJobName = jobNames.find(jobName => jobName.startsWith(idStart));
|
|
107
|
-
// If it exists:
|
|
108
|
-
if (seriesJobName) {
|
|
109
|
-
// Generate a job series.
|
|
110
|
-
const jobJSON = await fs.readFile(`${jobDir}/todo/${seriesJobName}`, 'utf8');
|
|
111
|
-
const job = JSON.parse(jobJSON);
|
|
112
|
-
const jobSeries = series(job, Number.parseInt(count), Number.parseInt(interval));
|
|
113
|
-
// Save the jobs.
|
|
114
|
-
for (const item of jobSeries) {
|
|
115
|
-
await fs.writeFile(`${jobDir}/todo/${item.id}.json`, `${JSON.stringify(item, null, 2)}\n`);
|
|
116
|
-
}
|
|
117
|
-
console.log(`Series of ${jobSeries.length} jobs generated and saved in ${jobDir}/todo`);
|
|
118
|
-
}
|
|
119
|
-
// Otherwise, i.e. if it does not exist:
|
|
120
|
-
else {
|
|
121
|
-
// Report this.
|
|
122
|
-
console.log('ERROR: No matching to-do job found');
|
|
94
|
+
await fs.writeFile(`${subdir}/${job.id}.json`, `${jobJSON}\n`);
|
|
123
95
|
}
|
|
96
|
+
const truncatedID = `${jobs[0].timeStamp}-${jobs[0].mergeID}-…`;
|
|
97
|
+
console.log(`Script ${scriptID} and batch ${batchID} merged as ${truncatedID} in ${subdir}`);
|
|
124
98
|
};
|
|
125
99
|
// Gets selected reports.
|
|
126
100
|
const getReports = async (type, selector = '') => {
|
|
@@ -247,18 +221,12 @@ else if (fn === 'script' && fnArgs.length) {
|
|
|
247
221
|
console.log('Execution completed');
|
|
248
222
|
});
|
|
249
223
|
}
|
|
250
|
-
else if (fn === 'merge' && fnArgs.length ===
|
|
224
|
+
else if (fn === 'merge' && fnArgs.length === 5) {
|
|
251
225
|
callMerge(... fnArgs)
|
|
252
226
|
.then(() => {
|
|
253
227
|
console.log('Execution completed');
|
|
254
228
|
});
|
|
255
229
|
}
|
|
256
|
-
else if (fn === 'series' && fnArgs.length === 3) {
|
|
257
|
-
callSeries(... fnArgs)
|
|
258
|
-
.then(() => {
|
|
259
|
-
console.log('Execution completed');
|
|
260
|
-
});
|
|
261
|
-
}
|
|
262
230
|
else if (fn === 'score' && fnArgs.length > 0 && fnArgs.length < 3) {
|
|
263
231
|
callScore(... fnArgs)
|
|
264
232
|
.then(() => {
|
package/merge.js
CHANGED
|
@@ -1,83 +1,33 @@
|
|
|
1
1
|
/*
|
|
2
2
|
merge.js
|
|
3
3
|
Merges a script and a batch and returns jobs.
|
|
4
|
-
Arguments:
|
|
5
|
-
0. script
|
|
6
|
-
1. batch
|
|
7
|
-
2. requester
|
|
8
|
-
3. whether to provide test isolation
|
|
9
|
-
4. value of the standard property
|
|
10
|
-
5. whether reporting is to be granular
|
|
11
|
-
6. date and time as a compact timestamp for job execution, if not now
|
|
12
4
|
*/
|
|
13
5
|
|
|
14
6
|
// ########## IMPORTS
|
|
15
7
|
|
|
16
8
|
// Module to keep secrets.
|
|
17
9
|
require('dotenv').config();
|
|
10
|
+
// Module to perform common actions.
|
|
11
|
+
const {alphaNumOf, dateOf, getRandomString, getNowStamp} = require('./procs/util');
|
|
18
12
|
|
|
19
13
|
// ########## CONSTANTS
|
|
20
14
|
|
|
21
|
-
// Standard requester.
|
|
22
|
-
const stdRequester = process.env.REQUESTER || 'nobody@nodomain.tld';
|
|
23
|
-
// Length of the random part of a job ID, as a string.
|
|
24
|
-
const randomIDLength = process.env.RANDOM_ID_LENGTH || '3';
|
|
25
15
|
// Tools that alter the page.
|
|
26
16
|
const contaminantNames = new Set([
|
|
27
17
|
'alfa',
|
|
28
18
|
'aslint',
|
|
29
19
|
'axe',
|
|
20
|
+
'ed11y',
|
|
30
21
|
'htmlcs',
|
|
31
22
|
'testaro'
|
|
32
23
|
]);
|
|
33
|
-
const randomIDChars = (() => {
|
|
34
|
-
const digits = Array(10).fill('').map((digit, index) => index.toString());
|
|
35
|
-
const uppers = Array(26).fill('').map((letter, index) => String.fromCodePoint(65 + index));
|
|
36
|
-
const lowers = Array(26).fill('').map((letter, index) => String.fromCodePoint(97 + index));
|
|
37
|
-
return digits.concat(uppers, lowers);
|
|
38
|
-
})();
|
|
39
24
|
|
|
40
25
|
|
|
41
26
|
// ########## FUNCTIONS
|
|
42
27
|
|
|
43
|
-
// Inserts a character periodically in a string.
|
|
44
|
-
const punctuate = (string, insertion, chunkSize) => {
|
|
45
|
-
const segments = [];
|
|
46
|
-
let startIndex = 0;
|
|
47
|
-
while (startIndex < string.length) {
|
|
48
|
-
segments.push(string.slice(startIndex, startIndex + chunkSize));
|
|
49
|
-
startIndex += chunkSize;
|
|
50
|
-
}
|
|
51
|
-
return segments.join(insertion);
|
|
52
|
-
};
|
|
53
|
-
// Converts a compact timestamp to a date.
|
|
54
|
-
const dateOf = timeStamp => {
|
|
55
|
-
if (/^\d{6}T\d{4}$/.test(timeStamp)) {
|
|
56
|
-
const dateString = punctuate(timeStamp.slice(0, 6), '-', 2);
|
|
57
|
-
const timeString = punctuate(timeStamp.slice(7, 11), ':', 2);
|
|
58
|
-
return new Date(`20${dateString}T${timeString}Z`);
|
|
59
|
-
} else {
|
|
60
|
-
return null;
|
|
61
|
-
}
|
|
62
|
-
};
|
|
63
|
-
// Converts a date and time to a compact timestamp.
|
|
64
|
-
const stampTime = date => date.toISOString().replace(/[-:]/g, '').slice(2, 13);
|
|
65
|
-
// Generates a random string.
|
|
66
|
-
const getRandomID = length => {
|
|
67
|
-
const chars = [];
|
|
68
|
-
for (let i = 0; i < length; i++) {
|
|
69
|
-
chars.push(randomIDChars[Math.floor(62 * Math.random())]);
|
|
70
|
-
}
|
|
71
|
-
return chars.join('');
|
|
72
|
-
};
|
|
73
28
|
// Merges a script and a batch and returns jobs.
|
|
74
|
-
exports.merge = (
|
|
75
|
-
|
|
76
|
-
) => {
|
|
77
|
-
if (isolate === 'false') {
|
|
78
|
-
isolate = false;
|
|
79
|
-
}
|
|
80
|
-
// If a timestamp was specified:
|
|
29
|
+
exports.merge = (script, batch, requester, timeStamp) => {
|
|
30
|
+
// If a time stamp was specified:
|
|
81
31
|
if (timeStamp) {
|
|
82
32
|
// If it is invalid:
|
|
83
33
|
if (! dateOf(timeStamp)) {
|
|
@@ -86,77 +36,77 @@ exports.merge = (
|
|
|
86
36
|
return [];
|
|
87
37
|
}
|
|
88
38
|
}
|
|
89
|
-
// Otherwise, i.e. if no
|
|
39
|
+
// Otherwise, i.e. if no time stamp was specified:
|
|
90
40
|
else {
|
|
91
41
|
// Create one for the job.
|
|
92
|
-
timeStamp =
|
|
42
|
+
timeStamp = getNowStamp();
|
|
93
43
|
}
|
|
94
|
-
//
|
|
95
|
-
requester ||= stdRequester;
|
|
96
|
-
// Create a creation-time description.
|
|
97
|
-
const creationTime = (new Date()).toISOString().slice(0, 16);
|
|
98
|
-
// Initialize a target-independent job.
|
|
44
|
+
// Initialize a job as a copy of the script.
|
|
99
45
|
const protoJob = JSON.parse(JSON.stringify(script));
|
|
100
|
-
//
|
|
101
|
-
protoJob.id = `${timeStamp}-${getRandomID(Number.parseInt(randomIDLength, 10))}-`;
|
|
102
|
-
// Add a sources property to the job.
|
|
46
|
+
// Add an initialized sources property to it.
|
|
103
47
|
protoJob.sources = {
|
|
104
48
|
script: script.id,
|
|
105
49
|
batch: batch.id,
|
|
106
50
|
target: {
|
|
107
51
|
id: '',
|
|
108
|
-
|
|
109
|
-
|
|
52
|
+
what: '',
|
|
53
|
+
which: ''
|
|
110
54
|
},
|
|
111
|
-
requester
|
|
112
|
-
sendReportTo: process.env.REPORT_URL || '',
|
|
113
|
-
url: ''
|
|
55
|
+
requester
|
|
114
56
|
};
|
|
115
57
|
// Add properties to the job.
|
|
116
|
-
protoJob.
|
|
58
|
+
protoJob.creationTimeStamp = getNowStamp();
|
|
117
59
|
protoJob.timeStamp = timeStamp;
|
|
118
|
-
protoJob.standard = standard || 'only';
|
|
119
|
-
protoJob.observe = observe || false;
|
|
120
60
|
// If isolation was requested:
|
|
121
|
-
if (isolate) {
|
|
122
|
-
//
|
|
61
|
+
if (script.isolate) {
|
|
62
|
+
// For each act:
|
|
123
63
|
let {acts} = protoJob;
|
|
124
64
|
let lastPlaceholder = {};
|
|
125
65
|
for (const actIndexString in acts) {
|
|
66
|
+
// If it is a placeholder:
|
|
126
67
|
const actIndex = Number.parseInt(actIndexString);
|
|
127
68
|
const act = acts[actIndex];
|
|
128
|
-
const nextAct = acts[actIndex + 1];
|
|
129
69
|
if (act.type === 'placeholder') {
|
|
70
|
+
// Identify it as the current one.
|
|
130
71
|
lastPlaceholder = act;
|
|
131
72
|
}
|
|
73
|
+
// Otherwise, if it is a non-final target-modifying test act:
|
|
132
74
|
else if (
|
|
133
75
|
act.type === 'test'
|
|
134
76
|
&& contaminantNames.has(act.which)
|
|
135
77
|
&& actIndex < acts.length - 1
|
|
136
|
-
&& (nextAct.type === 'test')
|
|
137
78
|
) {
|
|
79
|
+
// Change it to an array of itself and the current placeholder.
|
|
138
80
|
acts[actIndex] = JSON.parse(JSON.stringify([act, lastPlaceholder]));
|
|
139
81
|
}
|
|
140
82
|
};
|
|
83
|
+
// Flatten the acts.
|
|
141
84
|
protoJob.acts = acts.flat();
|
|
142
85
|
}
|
|
86
|
+
// Delete the no-longer-necessary job property.
|
|
87
|
+
delete protoJob.isolate;
|
|
143
88
|
// Initialize an array of jobs.
|
|
144
89
|
const jobs = [];
|
|
90
|
+
// Get an ID for the merger.
|
|
91
|
+
const mergeID = getRandomString(2);
|
|
145
92
|
// For each target in the batch:
|
|
146
93
|
const {targets} = batch;
|
|
147
|
-
|
|
94
|
+
targets.forEach((target, index) => {
|
|
148
95
|
// If the target has the required identifiers:
|
|
149
|
-
const {id,
|
|
150
|
-
if (id &&
|
|
96
|
+
const {id, what, which} = target;
|
|
97
|
+
if (id && what && which) {
|
|
151
98
|
// Initialize a job.
|
|
152
99
|
const job = JSON.parse(JSON.stringify(protoJob));
|
|
153
|
-
//
|
|
154
|
-
|
|
100
|
+
// Make the job ID unique.
|
|
101
|
+
const targetID = alphaNumOf(index);
|
|
102
|
+
job.id = `${timeStamp}-${mergeID}-${targetID}`;
|
|
103
|
+
// Add properties to the job.
|
|
104
|
+
job.mergeID = mergeID;
|
|
105
|
+
job.sendReportTo = process.env.SEND_REPORT_TO || '';
|
|
155
106
|
// Add data to the sources property of the job.
|
|
156
|
-
job.sources.target.id =
|
|
157
|
-
job.sources.target.which = target.which;
|
|
107
|
+
job.sources.target.id = targetID;
|
|
158
108
|
job.sources.target.what = target.what;
|
|
159
|
-
job.sources.
|
|
109
|
+
job.sources.target.which = target.which;
|
|
160
110
|
// Replace each placeholder object in the job with the named replacer array of the target.
|
|
161
111
|
let {acts} = job;
|
|
162
112
|
for (const actIndex in acts) {
|
|
@@ -184,13 +134,14 @@ exports.merge = (
|
|
|
184
134
|
}
|
|
185
135
|
}
|
|
186
136
|
}
|
|
137
|
+
// Flatten the acts.
|
|
187
138
|
job.acts = acts.flat();
|
|
188
139
|
// Append the job to the array of jobs.
|
|
189
140
|
jobs.push(job);
|
|
190
141
|
}
|
|
191
142
|
else {
|
|
192
|
-
console.log('ERROR: Target in batch missing id,
|
|
143
|
+
console.log('ERROR: Target in batch missing id, what, or which property');
|
|
193
144
|
}
|
|
194
|
-
};
|
|
145
|
+
});
|
|
195
146
|
return jobs;
|
|
196
147
|
};
|
package/package.json
CHANGED
package/procs/test.js
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
// Array of 62 alphanumeric characters.
|
|
2
|
+
const alphaNumChars = (() => {
|
|
3
|
+
const digits = Array(10).fill('').map((digit, index) => index.toString());
|
|
4
|
+
const uppers = Array(26).fill('').map((letter, index) => String.fromCodePoint(65 + index));
|
|
5
|
+
const lowers = Array(26).fill('').map((letter, index) => String.fromCodePoint(97 + index));
|
|
6
|
+
return digits.concat(uppers, lowers);
|
|
7
|
+
})();
|
|
8
|
+
|
|
9
|
+
// Returns an alphanumeric representation of an integer.
|
|
10
|
+
const alphaNumOf = num => {
|
|
11
|
+
let resultDigits = [];
|
|
12
|
+
while (num) {
|
|
13
|
+
const remainder = num % 62;
|
|
14
|
+
resultDigits.unshift(alphaNumChars[remainder]);
|
|
15
|
+
num = Math.floor(num / 62);
|
|
16
|
+
}
|
|
17
|
+
return resultDigits.join('');
|
|
18
|
+
};
|
|
19
|
+
|
|
20
|
+
console.log(alphaNumOf(process.argv[2]));
|
package/procs/util.js
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
/*
|
|
2
|
+
util.js
|
|
3
|
+
Utility functions.
|
|
4
|
+
*/
|
|
5
|
+
|
|
6
|
+
// CONSTANTS
|
|
7
|
+
|
|
8
|
+
// Array of 62 alphanumeric characters.
|
|
9
|
+
const alphaNumChars = (() => {
|
|
10
|
+
const digits = Array(10).fill('').map((digit, index) => index.toString());
|
|
11
|
+
const uppers = Array(26).fill('').map((letter, index) => String.fromCodePoint(65 + index));
|
|
12
|
+
const lowers = Array(26).fill('').map((letter, index) => String.fromCodePoint(97 + index));
|
|
13
|
+
return digits.concat(uppers, lowers);
|
|
14
|
+
})();
|
|
15
|
+
|
|
16
|
+
// FUNCTIONS
|
|
17
|
+
|
|
18
|
+
// Returns a string representing a date and time.
|
|
19
|
+
const getTimeString = date => date.toISOString().slice(0, 19);
|
|
20
|
+
// Returns a string representing the date and time.
|
|
21
|
+
exports.getNowString = () => getTimeString(new Date());
|
|
22
|
+
// Returns a time stamp representing a date and time.
|
|
23
|
+
const getTimeStamp = date => getTimeString(date).replace(/[-:]/g, '').slice(2, 13);
|
|
24
|
+
// Returns a time stamp representing the date and time.
|
|
25
|
+
exports.getNowStamp = () => getTimeStamp(new Date());
|
|
26
|
+
// Inserts a character periodically in a string.
|
|
27
|
+
const punctuate = (string, insertion, chunkSize) => {
|
|
28
|
+
const segments = [];
|
|
29
|
+
let startIndex = 0;
|
|
30
|
+
while (startIndex < string.length) {
|
|
31
|
+
segments.push(string.slice(startIndex, startIndex + chunkSize));
|
|
32
|
+
startIndex += chunkSize;
|
|
33
|
+
}
|
|
34
|
+
return segments.join(insertion);
|
|
35
|
+
};
|
|
36
|
+
// Converts a compact timestamp to a date.
|
|
37
|
+
exports.dateOf = timeStamp => {
|
|
38
|
+
if (/^\d{6}T\d{4}$/.test(timeStamp)) {
|
|
39
|
+
const dateString = punctuate(timeStamp.slice(0, 6), '-', 2);
|
|
40
|
+
const timeString = punctuate(timeStamp.slice(7, 11), ':', 2);
|
|
41
|
+
return new Date(`20${dateString}T${timeString}Z`);
|
|
42
|
+
} else {
|
|
43
|
+
return null;
|
|
44
|
+
}
|
|
45
|
+
};
|
|
46
|
+
// Returns a base-62 alphanumeric representation of an integer.
|
|
47
|
+
exports.alphaNumOf = num => {
|
|
48
|
+
let resultDigits = [];
|
|
49
|
+
while (num || ! resultDigits.length) {
|
|
50
|
+
const remainder = num % 62;
|
|
51
|
+
resultDigits.unshift(alphaNumChars[remainder]);
|
|
52
|
+
num = Math.floor(num / 62);
|
|
53
|
+
}
|
|
54
|
+
return resultDigits.join('');
|
|
55
|
+
};
|
|
56
|
+
// Returns a random string.
|
|
57
|
+
exports.getRandomString = length => {
|
|
58
|
+
const chars = [];
|
|
59
|
+
for (let i = 0; i < length; i++) {
|
|
60
|
+
chars.push(alphaNumChars[Math.floor(62 * Math.random())]);
|
|
61
|
+
}
|
|
62
|
+
return chars.join('');
|
|
63
|
+
};
|
package/script.js
CHANGED
|
@@ -73,7 +73,6 @@ exports.script = (id, issues = null, ... issueIDs) => {
|
|
|
73
73
|
timeLimit: 30 + (10 * issueIDs.length || 30 * toolIDs.length),
|
|
74
74
|
standard: 'only',
|
|
75
75
|
observe: true,
|
|
76
|
-
timeStamp: '',
|
|
77
76
|
acts: [
|
|
78
77
|
{
|
|
79
78
|
"type": "placeholder",
|
|
@@ -99,7 +98,7 @@ exports.script = (id, issues = null, ... issueIDs) => {
|
|
|
99
98
|
toolAct.rules.unshift('y');
|
|
100
99
|
}
|
|
101
100
|
}
|
|
102
|
-
// Add any needed option
|
|
101
|
+
// Add any needed option defaults to the act.
|
|
103
102
|
if (toolID === 'axe') {
|
|
104
103
|
toolAct.detailLevel = 2;
|
|
105
104
|
}
|
package/series.js
DELETED
|
@@ -1,72 +0,0 @@
|
|
|
1
|
-
/*
|
|
2
|
-
series.js
|
|
3
|
-
Generates a series of Testaro jobs.
|
|
4
|
-
Arguments:
|
|
5
|
-
0. Initial job.
|
|
6
|
-
1. Job count.
|
|
7
|
-
2. Time interval in minutes.
|
|
8
|
-
*/
|
|
9
|
-
|
|
10
|
-
// ########## FUNCTIONS
|
|
11
|
-
|
|
12
|
-
// Scores the specified raw reports.
|
|
13
|
-
exports.series = (job, count, interval) => {
|
|
14
|
-
// If the arguments are valid:
|
|
15
|
-
if (
|
|
16
|
-
typeof job === 'object'
|
|
17
|
-
&& count
|
|
18
|
-
&& typeof count === 'number'
|
|
19
|
-
&& count === Math.floor(count)
|
|
20
|
-
&& count > 1
|
|
21
|
-
&& interval
|
|
22
|
-
&& typeof interval === 'number'
|
|
23
|
-
&& interval === Math.floor(interval)
|
|
24
|
-
&& interval > 0
|
|
25
|
-
) {
|
|
26
|
-
// Get a copy of the initial job.
|
|
27
|
-
const template = JSON.parse(JSON.stringify(job));
|
|
28
|
-
// If it has an ID:
|
|
29
|
-
const jobID = template.id;
|
|
30
|
-
if (jobID) {
|
|
31
|
-
// If the ID specifies a valid time:
|
|
32
|
-
const s = jobID.slice(0, 11);
|
|
33
|
-
const dateSpec = `20${s[0]}${s[1]}-${s[2]}${s[3]}-${s[4]}${s[5]}`;
|
|
34
|
-
const timeSpec = `${s[7]}${s[8]}:${s[9]}${s[10]}`;
|
|
35
|
-
const dateTimeSpec = `${dateSpec}T${timeSpec}Z`;
|
|
36
|
-
const start = new Date(dateTimeSpec);
|
|
37
|
-
const startNum = start.valueOf();
|
|
38
|
-
if (startNum) {
|
|
39
|
-
// Initialize the series.
|
|
40
|
-
const series = [];
|
|
41
|
-
// For each job required:
|
|
42
|
-
for (let i = 0; i < count; i++) {
|
|
43
|
-
// Create it.
|
|
44
|
-
const nextJob = JSON.parse(JSON.stringify(template));
|
|
45
|
-
nextJob.sources.series = nextJob.id;
|
|
46
|
-
// Revise its ID.
|
|
47
|
-
const nextDate = new Date(startNum + i * interval * 60000);
|
|
48
|
-
const nextTimeStamp = nextDate.toISOString().slice(2, 16).replace(/[-:]/g, '');
|
|
49
|
-
nextJob.id = nextJob.id.replace(/^[^-]+/, nextTimeStamp);
|
|
50
|
-
// Add the job to the series.
|
|
51
|
-
series.push(nextJob);
|
|
52
|
-
}
|
|
53
|
-
return series;
|
|
54
|
-
}
|
|
55
|
-
// Otherwise, i.e. if it does not specify a valid time:
|
|
56
|
-
else {
|
|
57
|
-
// Report this.
|
|
58
|
-
console.log('ERROR: Initial job ID starts with an invalid time specification');
|
|
59
|
-
}
|
|
60
|
-
}
|
|
61
|
-
// Otherwise, i.e. if it has no ID:
|
|
62
|
-
else {
|
|
63
|
-
// Report this.
|
|
64
|
-
console.log('ERROR: Initial job has no ID');
|
|
65
|
-
}
|
|
66
|
-
}
|
|
67
|
-
// Otherwise, i.e. if they are invalid:
|
|
68
|
-
else {
|
|
69
|
-
// Report this.
|
|
70
|
-
console.log('ERROR: Arguments invalid');
|
|
71
|
-
}
|
|
72
|
-
};
|
|
@@ -1,37 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"id": "231120T155027-mon-example",
|
|
3
|
-
"what": "Job for series validation",
|
|
4
|
-
"strict": true,
|
|
5
|
-
"timeLimit": 10,
|
|
6
|
-
"acts": [
|
|
7
|
-
{
|
|
8
|
-
"type": "launch",
|
|
9
|
-
"which": "chromium",
|
|
10
|
-
"url": "https://example.com",
|
|
11
|
-
"what": "Example of web page",
|
|
12
|
-
"startTime": 1662474496075,
|
|
13
|
-
"endTime": 1662474496453
|
|
14
|
-
},
|
|
15
|
-
{
|
|
16
|
-
"type": "test",
|
|
17
|
-
"which": "testaro",
|
|
18
|
-
"url": "https://example.com",
|
|
19
|
-
"withItems": false,
|
|
20
|
-
"rules": [
|
|
21
|
-
"y",
|
|
22
|
-
"bulk"
|
|
23
|
-
]
|
|
24
|
-
}
|
|
25
|
-
],
|
|
26
|
-
"sources": {
|
|
27
|
-
"script": "mon",
|
|
28
|
-
"batch": "target",
|
|
29
|
-
"target": {
|
|
30
|
-
"id": "example",
|
|
31
|
-
"what": "Example of web page"
|
|
32
|
-
},
|
|
33
|
-
"requester": "user@domain.tld"
|
|
34
|
-
},
|
|
35
|
-
"creationTime": "2023-11-20T15:50:27",
|
|
36
|
-
"timeStamp": "231120T155027"
|
|
37
|
-
}
|
|
@@ -1,78 +0,0 @@
|
|
|
1
|
-
/*
|
|
2
|
-
validate.js
|
|
3
|
-
Validates series module.
|
|
4
|
-
*/
|
|
5
|
-
|
|
6
|
-
// ########## IMPORTS
|
|
7
|
-
|
|
8
|
-
// Function to process files.
|
|
9
|
-
const fs = require('fs/promises');
|
|
10
|
-
// Function to generate a job series.
|
|
11
|
-
const {series} = require('../../series');
|
|
12
|
-
|
|
13
|
-
// ########## FUNCTIONS
|
|
14
|
-
|
|
15
|
-
// Validates the series module.
|
|
16
|
-
const validate = async () => {
|
|
17
|
-
// Get the job.
|
|
18
|
-
const jobJSON = await fs.readFile(`${__dirname}/job.json`, 'utf8');
|
|
19
|
-
const job = JSON.parse(jobJSON);
|
|
20
|
-
// Generate the series.
|
|
21
|
-
const jobs = series(job, 3, 5);
|
|
22
|
-
// Validate the series.
|
|
23
|
-
if (Array.isArray(jobs) && jobs.length === 3) {
|
|
24
|
-
console.log('Success: The count of jobs is correct');
|
|
25
|
-
}
|
|
26
|
-
else {
|
|
27
|
-
console.log('ERROR: The jobs are not an array of length 3');
|
|
28
|
-
return;
|
|
29
|
-
}
|
|
30
|
-
const job0 = jobs[0];
|
|
31
|
-
if (
|
|
32
|
-
job.id
|
|
33
|
-
&& job.id === '240223T0815-mon-example'
|
|
34
|
-
&& job0.id === job.id
|
|
35
|
-
&& job0.sources
|
|
36
|
-
&& job0.sources.series
|
|
37
|
-
&& job0.sources.series === job.id
|
|
38
|
-
) {
|
|
39
|
-
console.log('Success: The first job has the correct id and sources.series');
|
|
40
|
-
}
|
|
41
|
-
else {
|
|
42
|
-
console.log('ERROR: The first job has an incorrect id or sources.series');
|
|
43
|
-
return;
|
|
44
|
-
}
|
|
45
|
-
const job1 = jobs[1];
|
|
46
|
-
const job2 = jobs[2];
|
|
47
|
-
if (
|
|
48
|
-
job2.id
|
|
49
|
-
&& job2.id === '240223T0825-mon-example'
|
|
50
|
-
&& job2.sources
|
|
51
|
-
&& job2.sources.series
|
|
52
|
-
&& job2.sources.series === '240223T0815-mon-example'
|
|
53
|
-
) {
|
|
54
|
-
console.log('Success: The third job has the correct id and sources.series');
|
|
55
|
-
}
|
|
56
|
-
else {
|
|
57
|
-
console.log('ERROR: The first job has an incorrect id or sources.series');
|
|
58
|
-
return;
|
|
59
|
-
}
|
|
60
|
-
if (
|
|
61
|
-
job1.acts
|
|
62
|
-
&& job1.acts.length === 3
|
|
63
|
-
&& job1.acts[2].rules
|
|
64
|
-
&& job1.acts[2].rules.length === 2
|
|
65
|
-
&& job2.acts
|
|
66
|
-
&& job2.acts.length === 3
|
|
67
|
-
&& job2.acts[2].rules
|
|
68
|
-
&& job2.acts[2].rules.length === 2
|
|
69
|
-
&& job2.acts[2].rules[1] === job1.acts[2].rules[1]
|
|
70
|
-
) {
|
|
71
|
-
console.log('Success: The second and third jobs invoke the same rule');
|
|
72
|
-
}
|
|
73
|
-
else {
|
|
74
|
-
console.log('ERROR: The second and third job do not invoke the same rule');
|
|
75
|
-
return;
|
|
76
|
-
}
|
|
77
|
-
};
|
|
78
|
-
validate();
|