testilo 22.1.0 → 23.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +137 -123
- package/call.js +0 -31
- package/package.json +1 -1
- package/series.js +0 -72
- package/validation/series/job.json +0 -37
- package/validation/series/validate.js +0 -78
package/README.md
CHANGED
|
@@ -5,7 +5,7 @@ Utilities for Testaro
|
|
|
5
5
|
|
|
6
6
|
The Testilo package contains utilities that facilitate the use of the [Testaro](https://www.npmjs.com/package/testaro) package.
|
|
7
7
|
|
|
8
|
-
Testaro performs
|
|
8
|
+
Testaro performs jobs and creates reports in JSON format. The utilities in Testilo fall into two categories:
|
|
9
9
|
- Job preparation
|
|
10
10
|
- Report enhancement
|
|
11
11
|
|
|
@@ -21,24 +21,34 @@ When Testilo is a dependency of another application, the `.env` file is not impo
|
|
|
21
21
|
|
|
22
22
|
Testilo is written in Node.js. Commands are given to Testilo in a command-line (terminal) interface or programmatically.
|
|
23
23
|
|
|
24
|
-
Shared routines
|
|
24
|
+
Shared routines, called _procs_, are located in the `procs` directory.
|
|
25
25
|
|
|
26
26
|
Testilo can be installed wherever Node.js (version 14 or later) is installed. This can be a server or the same workstation on which Testaro is installed.
|
|
27
27
|
|
|
28
|
-
The reason for Testilo being an independent package, rather than part of Testaro, is that Testilo can be installed on any host, while Testaro can run successfully only on a Windows
|
|
28
|
+
The reason for Testilo being an independent package, rather than part of Testaro, is that Testilo can be installed on any host, while Testaro can run successfully only on a Windows, Macintosh, Ubuntu, or Debian workstation. Testaro runs tests similar to those that a human accessibility tester would run, using whatever browsers, input devices, system settings, simulated and attached devices, and assistive technologies tests may require. Thus, Testaro is limited to functionalities that require workstation attributes. For maximum flexibility in the management of Testaro jobs, all other functionalities are located outside of Testaro. You could have software such as Testilo running on a server, communicating with multiple workstations running Testaro. The workstations could receive jobs from the server and return job reports to the server for further processing.
|
|
29
29
|
|
|
30
30
|
## Configuration
|
|
31
31
|
|
|
32
32
|
Environment variables for Testilo can be specified in a `.env` file. An example:
|
|
33
33
|
|
|
34
34
|
```bash
|
|
35
|
-
FUNCTIONDIR
|
|
35
|
+
FUNCTIONDIR=./procs
|
|
36
36
|
JOBDIR=../testdir/jobs
|
|
37
37
|
REPORTDIR=../testdir/reports
|
|
38
38
|
REQUESTER=a11ymgr@a11yorg.com
|
|
39
39
|
SPECDIR=../testdir/specs
|
|
40
40
|
```
|
|
41
41
|
|
|
42
|
+
The `FUNCTIONDIR` environment variable typically references the `procs` directory, but it could reference a different directory in the filesystem where Testilo resides, if you wanted to customize the procs that Testilo uses.
|
|
43
|
+
|
|
44
|
+
`JOBDIR` references a directory in the filesystem where jobs created by the `merge` proc are to be saved.
|
|
45
|
+
|
|
46
|
+
`REPORTDIR` references a directory in the filesystem where reports are saved.
|
|
47
|
+
|
|
48
|
+
`REQUESTER` is an email address that will be used as a job property if no other email address is specified for the `sources.requester` property of the job.
|
|
49
|
+
|
|
50
|
+
`SPECDIR` references a directory in the filesystem where tanrget lists, batches, and scripts can be found. Those are raw materials from which Testaro creates jobs.
|
|
51
|
+
|
|
42
52
|
## Job preparation
|
|
43
53
|
|
|
44
54
|
### Introduction
|
|
@@ -49,16 +59,27 @@ You can create a job for Testaro directly, without using Testilo.
|
|
|
49
59
|
|
|
50
60
|
Testilo can, however, make job preparation more efficient in these scenarios:
|
|
51
61
|
- You want to perform a battery of tests on multiple targets.
|
|
52
|
-
- You want to test targets for particular issues, using whichever tools happen to have tests for those issues.
|
|
62
|
+
- You want to test targets only for particular issues, using whichever tools happen to have tests for those issues.
|
|
53
63
|
|
|
54
64
|
### Target lists
|
|
55
65
|
|
|
56
|
-
The simplest version of a list of targets is a _target list_. It is an array of arrays defining 1 or more targets. It
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
-
|
|
66
|
+
The simplest version of a list of targets is a _target list_. It is an array of arrays defining 1 or more targets. It can be stored as a tab-delimited text file.
|
|
67
|
+
|
|
68
|
+
A target is defined by 3 items:
|
|
69
|
+
- An ID
|
|
70
|
+
- A description
|
|
71
|
+
- A URL
|
|
60
72
|
|
|
61
|
-
For example, a
|
|
73
|
+
For example, a target list might be:
|
|
74
|
+
|
|
75
|
+
```javaScript
|
|
76
|
+
[
|
|
77
|
+
['w3c', 'World Wide Web Consortium', 'https://www.w3.org/'],
|
|
78
|
+
['moz', 'Mozilla Foundation', 'https://foundation.mozilla.org/en/']
|
|
79
|
+
]
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
If this target list were stored as a file, its content would be this (with “→” representing the Tab character):
|
|
62
83
|
|
|
63
84
|
```text
|
|
64
85
|
w3c→World Wide Web Consortium→https://www.w3.org/
|
|
@@ -128,9 +149,11 @@ Targets can be specified in a more complex way, too. That allows you to create j
|
|
|
128
149
|
|
|
129
150
|
As shown, a batch, unlike a target list, defines named sequences of acts. They can be plugged into jobs, so various complex operations can be performed on each target.
|
|
130
151
|
|
|
152
|
+
A batch is a JavaScript object. It can be converted to JSON and stored in a file.
|
|
153
|
+
|
|
131
154
|
### Scripts
|
|
132
155
|
|
|
133
|
-
The generic, target-independent description of a job is _script_. A script can contain _placeholders_ that Testilo replaces with acts from a batch, creating one job per target. Thus, one script plus
|
|
156
|
+
The generic, target-independent description of a job is _script_. A script can contain _placeholders_ that Testilo replaces with acts from a batch, creating one job per target. Thus, one script plus a batch containing _n_ targets will generate _n_ jobs.
|
|
134
157
|
|
|
135
158
|
Here is a script:
|
|
136
159
|
|
|
@@ -144,6 +167,9 @@ Here is a script:
|
|
|
144
167
|
standard: 'also',
|
|
145
168
|
observe: false,
|
|
146
169
|
timeStamp: '240115T1200',
|
|
170
|
+
requester: 'you@yourdomain.com',
|
|
171
|
+
urlPrefix: '/file/reports',
|
|
172
|
+
urlSuffix: '.json',
|
|
147
173
|
acts: [
|
|
148
174
|
{
|
|
149
175
|
type: 'placeholder',
|
|
@@ -169,13 +195,16 @@ Here is a script:
|
|
|
169
195
|
```
|
|
170
196
|
|
|
171
197
|
A script has several properties that specify facts about the jobs to be created. They include:
|
|
172
|
-
- `id`: an ID
|
|
198
|
+
- `id`: an ID. A script can be converted from a JavaScript object to JSON and saved in a file in the `SPECDIR` directory, where it will be named by its ID (e.g., if the ID is `ts99`, the file name will be `ts99.json`). Thus, each script needs an `id` with a unique value.
|
|
173
199
|
- `what`: a description of the script.
|
|
174
|
-
- `strict`:
|
|
175
|
-
- `isolate`:
|
|
176
|
-
- `standard`:
|
|
177
|
-
- `observe`:
|
|
178
|
-
- `timeStamp`:
|
|
200
|
+
- `strict`: `true` if Testaro is to abort jobs when a target redirects a request to a URL differing substantially from the one specified. If `false` Testaro is to allow redirection. All differences are considered substantial unless the URLs differ only in the presence and absence of a trailing slash.
|
|
201
|
+
- `isolate`: If `true`, Testilo, before creating a job, will isolate test acts, as needed, from effects of previous test acts, by inserting a copy of the latest placeholder after each target-modifying test act other than the final act. If `false`, placeholders will not be duplicated.
|
|
202
|
+
- `standard`: If `also`, jobs will tell Testaro to include in its reports both the original results of the tests of tools and the Testaro-standardized results. If `only`, reports are to include only the standardized test results. If `no`, reports are to include only the original results, without standardization.
|
|
203
|
+
- `observe`: If `true`, jobs will tell Testaro to allow granular observation of job progress. If `false`, jobs will tell Testaro not to permit granular observation, but only to send the report to the server when the report is completed. It is generally user-friendly to allow granular observation, and for user applications to implement it, if they make users wait while jobs are assigned and performed, since that process typically takes about 3 minutes.
|
|
204
|
+
- `timeStamp`: This string specifies a UTC date and time when jobs created with the script are to be permitted to be assigned to agents. Thus, jobs can be created from a script for later performance. The value of `timeStamp` is a compact representation in the format `yymmddThhMM`. If the value is an empty string, Testilo will make the date and time equal to the time when jobs are created.
|
|
205
|
+
- `requester`: the email address that any notices of job completion can be sent to, or an empty string if there is a `REQUESTER` environment variable and it is to be used.
|
|
206
|
+
- `urlPrefix`: the start of a URL that the Testaro reports will be retrievable at.
|
|
207
|
+
- `urlSuffix`: the end of that URL. The job ID will be inserted between the `urlPrefix` and the `urlSuffix`.
|
|
179
208
|
- `acts`: an array of acts.
|
|
180
209
|
|
|
181
210
|
The first act in this example script is a placeholder, whose `which` property is `'private'`. If the above batch were merged with this script, in each job the placeholder would be replaced with the `private` acts of a target. For example, the first act of the first job would launch a Chromium browser, navigate to the Acme login page, complete and submit the login form, wait for the account page to load, run the Axe tests, and then run the QualWeb tests. If the batch contained additional targets, additional jobs would be created, with the login actions for each target specified in the `private` array of the `acts` object of that target.
|
|
@@ -207,22 +236,22 @@ The `batch()` function of the `batch` module generates a batch and returns it as
|
|
|
207
236
|
|
|
208
237
|
A user can invoke `batch` in this way:
|
|
209
238
|
|
|
210
|
-
- Create a target list and save it as a text file (with tab-delimited items in newline-delimited lines) in the `targetLists` subdirectory of the `
|
|
211
|
-
- In the Testilo project directory, execute the statement `node call batch
|
|
239
|
+
- Create a target list and save it as a text file (with tab-delimited items in newline-delimited lines) in the `targetLists` subdirectory of the `SPECDIR` directory. Name the file `x.tsv`, where `x` is the list ID.
|
|
240
|
+
- In the Testilo project directory, execute the statement `node call batch id what`.
|
|
212
241
|
|
|
213
|
-
In this statement, replace `
|
|
242
|
+
In this statement, replace `id` with the list ID and `what` with a string describing the batch.
|
|
214
243
|
|
|
215
244
|
The `call` module will retrieve the named target list.
|
|
216
245
|
The `batch` module will convert the target list to a batch.
|
|
217
|
-
The `call` module will save the batch as a JSON file in the `batches` subdirectory of the `
|
|
246
|
+
The `call` module will save the batch as a JSON file in the `batches` subdirectory of the `SPECDIR` directory.
|
|
218
247
|
|
|
219
248
|
### Issues to script
|
|
220
249
|
|
|
221
|
-
Testilo
|
|
250
|
+
Testilo contains a classification of tool rules into _issues_. It is located in the `procs/score` directory and has a file name starting with `tic` (Testilo issue classification). You can create custom classifications and save them in a `score` subdirectory of the `FUNCTIONDIR` directory.
|
|
222
251
|
|
|
223
252
|
For example, one of the issues in the `tic40.js` file is `mainNot1`. Four rules are classified as belonging to that issue: rule `main_element_only_one` of `aslint` and 3 more rules defined by 3 other tools.
|
|
224
253
|
|
|
225
|
-
If you want Testaro to test targets for only particular issues, you can
|
|
254
|
+
If you want Testaro to test targets for only particular issues, you can use the `script` module to create a script. Jobs created from that script will make Testaro test for only the issues you specify to the `script` module.
|
|
226
255
|
|
|
227
256
|
If you want Testaro to test targets for **all** the rules of all the available tools, you can use the `script` module to create a script that does not impose any issue restrictions.
|
|
228
257
|
|
|
@@ -240,9 +269,9 @@ const scriptObj = script(scriptID, issues, issueID0, issueID1, …);
|
|
|
240
269
|
```
|
|
241
270
|
|
|
242
271
|
This invocation references `scriptID`, `issues`, and `issueID` variables.
|
|
243
|
-
- The `scriptID` variable
|
|
272
|
+
- The `scriptID` variable specifies the ID that the script will have.
|
|
244
273
|
- The `issues` variable (if present) is an object that classifies issues, such as the `issues` object in a `tic` file.
|
|
245
|
-
- The `issueID` variables (if any) are strings, such as `'regionNoText'`, that name properties of the `issues` object.
|
|
274
|
+
- The `issueID` variables (if any) are strings, such as `'regionNoText'`, that name issues, i.e. properties of the `issues` object, that you want jobs from the script to test for.
|
|
246
275
|
|
|
247
276
|
The `script()` function of the `script` module generates a script and returns it as an object. The invoking module can further modify and use the script as needed.
|
|
248
277
|
|
|
@@ -255,41 +284,40 @@ const scriptObj = script(scriptID);
|
|
|
255
284
|
|
|
256
285
|
##### By a user
|
|
257
286
|
|
|
258
|
-
A user can invoke `script` in this way: In the Testilo project directory, execute the statement `node call script
|
|
287
|
+
A user can invoke `script` in this way: In the Testilo project directory, execute the statement `node call script id ticnn issuea issueb …`.
|
|
259
288
|
|
|
260
289
|
In this statement:
|
|
261
|
-
- Replace `
|
|
262
|
-
- Replace `
|
|
263
|
-
- Replace the remaining arguments (`
|
|
290
|
+
- Replace `id` with an ID for the script, such as `headings`.
|
|
291
|
+
- Replace `ticnn` with the base, such as `tic99`, of the name of an issue classification file in the `score` subdirectory of the `FUNCTIONDIR` directory.
|
|
292
|
+
- Replace the remaining arguments (`issuea` etc.) with issue names from that classification file.
|
|
264
293
|
|
|
265
294
|
The `call` module will retrieve the named classification.
|
|
266
295
|
The `script` module will create a script.
|
|
267
|
-
The `call` module will save the script as a JSON file in the `scripts` subdirectory of the `
|
|
296
|
+
The `call` module will save the script as a JSON file in the `scripts` subdirectory of the `SPECDIR` directory.
|
|
268
297
|
|
|
269
|
-
To create a script without any issue restrictions, a user can execute the statement `node call script
|
|
298
|
+
To create a script without any issue restrictions, a user can execute the statement `node call script id`.
|
|
270
299
|
|
|
271
300
|
#### Options
|
|
272
301
|
|
|
273
|
-
When the `script` module creates a script for you, it does not ask you for all of the options that the script may require. Instead, it chooses options. After you invoke `script`, you can edit the script that it creates to revise options.
|
|
302
|
+
When the `script` module creates a script for you, it does not ask you for all of the options that the script may require. Instead, it chooses default options. After you invoke `script`, you can edit the script that it creates to revise options.
|
|
274
303
|
|
|
275
304
|
### Merge
|
|
276
305
|
|
|
277
306
|
Testilo merges batches with scripts, producing jobs, by means of the `merge` module.
|
|
278
307
|
|
|
279
|
-
The `merge` module needs to be given a batch and a script.
|
|
280
|
-
|
|
281
308
|
#### Output
|
|
282
309
|
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
Suppose you ask for a merger of the above batch and script, **without** the isolation option. Then the first job produced by `merge` will look like this:
|
|
310
|
+
Suppose you ask for a merger of the above batch and script. Then the first job produced by `merge` will look like this:
|
|
286
311
|
|
|
287
312
|
```javaScript
|
|
288
313
|
{
|
|
289
|
-
id: '
|
|
290
|
-
what: '
|
|
314
|
+
id: '240115T1200-4Rw-acme',
|
|
315
|
+
what: 'aside mislocation',
|
|
291
316
|
strict: true,
|
|
292
317
|
timeLimit: 60,
|
|
318
|
+
standard: 'also',
|
|
319
|
+
observe: false,
|
|
320
|
+
timeStamp: '240115T1200',
|
|
293
321
|
acts: [
|
|
294
322
|
{
|
|
295
323
|
type: 'launch',
|
|
@@ -323,15 +351,45 @@ Suppose you ask for a merger of the above batch and script, **without** the isol
|
|
|
323
351
|
{
|
|
324
352
|
type: 'test',
|
|
325
353
|
which: 'axe',
|
|
326
|
-
detailLevel:
|
|
327
|
-
rules: [],
|
|
328
|
-
what: 'Axe
|
|
354
|
+
detailLevel: 2,
|
|
355
|
+
rules: ['landmark-complementary-is-top-level'],
|
|
356
|
+
what: 'Axe'
|
|
357
|
+
},
|
|
358
|
+
{
|
|
359
|
+
type: 'launch',
|
|
360
|
+
which: 'chromium'
|
|
361
|
+
},
|
|
362
|
+
{
|
|
363
|
+
type: 'url',
|
|
364
|
+
which: 'https://acmeclothes.com/login.html',
|
|
365
|
+
what: 'Acme Clothes login page'
|
|
366
|
+
},
|
|
367
|
+
{
|
|
368
|
+
type: 'text',
|
|
369
|
+
which: 'User Name',
|
|
370
|
+
what: 'tester34'
|
|
371
|
+
},
|
|
372
|
+
{
|
|
373
|
+
type: 'text',
|
|
374
|
+
which: 'Password',
|
|
375
|
+
what: '34SecretTester'
|
|
376
|
+
},
|
|
377
|
+
{
|
|
378
|
+
type: 'button',
|
|
379
|
+
which: 'Submit',
|
|
380
|
+
what: 'submit the login form'
|
|
381
|
+
},
|
|
382
|
+
{
|
|
383
|
+
type: 'wait',
|
|
384
|
+
which: 'title',
|
|
385
|
+
what: 'account'
|
|
329
386
|
},
|
|
330
387
|
{
|
|
331
388
|
type: 'test',
|
|
332
389
|
which: 'qualWeb',
|
|
333
390
|
withNewContent: false,
|
|
334
|
-
|
|
391
|
+
rules: ['QW-BP25', 'QW-BP26']
|
|
392
|
+
what: 'QualWeb'
|
|
335
393
|
}
|
|
336
394
|
],
|
|
337
395
|
sources: {
|
|
@@ -341,34 +399,22 @@ Suppose you ask for a merger of the above batch and script, **without** the isol
|
|
|
341
399
|
id: 'acme',
|
|
342
400
|
what: 'Acme Clothes'
|
|
343
401
|
},
|
|
344
|
-
requester: 'you@yourdomain.tld'
|
|
402
|
+
requester: 'you@yourdomain.tld',
|
|
403
|
+
url: '/file/reports/240115T1200-4Rw-acme.json'
|
|
345
404
|
},
|
|
346
|
-
creationTime: '2023-11-20T15:50:27'
|
|
347
|
-
timeStamp: '231120T155314'
|
|
405
|
+
creationTime: '2023-11-20T15:50:27'
|
|
348
406
|
}
|
|
349
407
|
```
|
|
350
408
|
|
|
351
409
|
Testilo has substituted the `private` acts from the `acme` target of the batch for the placeholder when creating the job. Testilo also has:
|
|
410
|
+
- inserted a copy of those same acts after the `axe` test act, because `axe` is a target-modifying tool.
|
|
352
411
|
- let the script determine the browser type of the `launch` act.
|
|
353
412
|
- added the creation time to the job.
|
|
354
|
-
-
|
|
355
|
-
-
|
|
356
|
-
- inserted a `sources` property into the job, recording facts about the script, the batch, the target, and the email address given by the user who requested the merger.
|
|
413
|
+
- given the job an ID that combines the time stamp with a differentiator and the batch ID.
|
|
414
|
+
- inserted a `sources` property into the job, recording facts about the script, the batch, the target, the requester, and the report URL.
|
|
357
415
|
|
|
358
416
|
This is a valid Testaro job.
|
|
359
417
|
|
|
360
|
-
##### With isolation
|
|
361
|
-
|
|
362
|
-
If, however, you requested a merger **with** isolation, then Testilo would take cognizance of the fact that an `axe` test act is a target-modifying act. Testilo would therefore act as if another instance of the placeholder had been located in the script after the `axe` test act. So, copies of the same 6 acts that precede the `axe` test act would be inserted **after** the `axe` test act, too.
|
|
363
|
-
|
|
364
|
-
Of the 9 tools providing tests for Testaro, 6 are target-modifying:
|
|
365
|
-
- `alfa`
|
|
366
|
-
- `aslint`
|
|
367
|
-
- `axe`
|
|
368
|
-
- `htmlcs`
|
|
369
|
-
- `ibm`
|
|
370
|
-
- `testaro`
|
|
371
|
-
|
|
372
418
|
#### Invocation
|
|
373
419
|
|
|
374
420
|
There are two ways to use the `merge` module.
|
|
@@ -379,10 +425,25 @@ A module can invoke `merge` in this way:
|
|
|
379
425
|
|
|
380
426
|
```javaScript
|
|
381
427
|
const {merge} = require('testilo/merge');
|
|
382
|
-
const jobs = merge(
|
|
428
|
+
const jobs = merge(
|
|
429
|
+
script, batch, requester, isolate, standard, isGranular, timeStamp, 'file/reports/', '.json'
|
|
430
|
+
);
|
|
383
431
|
```
|
|
384
432
|
|
|
385
|
-
|
|
433
|
+
The `merge` module uses these 9 arguments to create jobs from a script and a batch.
|
|
434
|
+
|
|
435
|
+
The arguments are:
|
|
436
|
+
- `script`: a script.
|
|
437
|
+
- `batch`: a batch.
|
|
438
|
+
- `requester`: an email address, or an empty string if there is a `REQUESTER` environment variable to be used.
|
|
439
|
+
- `isolate`: `true` if test isolation is to be performed, or `false` if not.
|
|
440
|
+
- `standard`: one of the standardization options (`'also'`, `'only'`, or `'no'`).
|
|
441
|
+
- `isGranular`: `true` if Testaro is to allow granular observation of the job when performed under a network watch, or `false` if not.
|
|
442
|
+
- `timeStamp`: a time stamp in the format `240115T1200`.
|
|
443
|
+
- `urlPnrefix`: the start of an absolute or relative report URL.
|
|
444
|
+
- `urlSuffix`: the end of a report URL.
|
|
445
|
+
|
|
446
|
+
In this case, a job with ID `240115T1200-4Rw-acme` will produce a report that can be retrieved at the relative URL `/file/reports/240115T1200-4Rw-acme.json`.
|
|
386
447
|
|
|
387
448
|
The `merge()` function of the `merge` module generates jobs and returns them in an array. The invoking module can further dispose of the jobs as needed.
|
|
388
449
|
|
|
@@ -390,76 +451,29 @@ The `merge()` function of the `merge` module generates jobs and returns them in
|
|
|
390
451
|
|
|
391
452
|
A user can invoke `merge` in this way:
|
|
392
453
|
|
|
393
|
-
- Create a script and save it as a JSON file in the `scripts` subdirectory of the `
|
|
394
|
-
- Create a batch and save it as a JSON file in the `batches` subdirectory of the `
|
|
395
|
-
- In the Testilo project directory, execute
|
|
396
|
-
- `node call merge scriptName batchName email isolate standard granular todoDir pre post`
|
|
454
|
+
- Create a script and save it as a JSON file in the `scripts` subdirectory of the `SPECDIR` directory.
|
|
455
|
+
- Create a batch and save it as a JSON file in the `batches` subdirectory of the `SPECDIR` directory.
|
|
456
|
+
- In the Testilo project directory, execute the statement `node call merge scriptName batchName requester isolate standard observe todoDir urlPrefix urlSuffix`.
|
|
397
457
|
|
|
398
|
-
In
|
|
399
|
-
- `scriptName` with the base name of the script file
|
|
400
|
-
- `batchName` with the base name of the batch file
|
|
401
|
-
- `
|
|
402
|
-
- `isolate` with `true` if you want test isolation or `false` if not
|
|
458
|
+
In this statement, replace:
|
|
459
|
+
- `scriptName` with the base name of the script file.
|
|
460
|
+
- `batchName` with the base name of the batch file.
|
|
461
|
+
- `requester` with an email address, or with an empty string if the environment variable `REQUESTER` exists and you want to use it.
|
|
462
|
+
- `isolate` with `true` if you want test isolation or `false` if not.
|
|
403
463
|
- `standard` with `'also'`, `'only'`, or `'no'` to specify the treatment of standard-format results.
|
|
404
|
-
- `
|
|
405
|
-
- `todoDir` with `true` if the job is to be saved in the `todo` subdirectory or `false` if it is to be saved in the `pending` subdirectory of the `
|
|
406
|
-
- `
|
|
407
|
-
- `
|
|
464
|
+
- `observe` with `true` if granular observation is to be permitted, or `false` if not.
|
|
465
|
+
- `todoDir` with `true` if the job is to be saved in the `todo` subdirectory, or `false` if it is to be saved in the `pending` subdirectory, of the `JOBDIR` directory.
|
|
466
|
+
- `urlPrefix` with the pre-ID part of the report URL.
|
|
467
|
+
- `urlSuffix` with the post-ID part of the report URL.
|
|
408
468
|
|
|
409
469
|
The `call` module will retrieve the named script and batch from their respective directories.
|
|
410
470
|
The `merge` module will create an array of jobs, with or without test isolation.
|
|
411
|
-
The `call` module will save the jobs as JSON files in the `todo` or `pending` subdirectory of the `
|
|
471
|
+
The `call` module will save the jobs as JSON files in the `todo` or `pending` subdirectory of the `JOBDIR` directory.
|
|
412
472
|
|
|
413
473
|
#### Validation
|
|
414
474
|
|
|
415
475
|
To test the `merge` module, in the project directory you can execute the statement `node validation/merge/validate`. If `merge` is valid, all logging statements will begin with “Success” and none will begin with “ERROR”.
|
|
416
476
|
|
|
417
|
-
### Series
|
|
418
|
-
|
|
419
|
-
If you want to monitor a web resource by performing identical jobs repeatedly and comparing the results, you can use the `series` module to create a series of identical jobs.
|
|
420
|
-
|
|
421
|
-
The jobs in a series differ from one another only in the timestamp segments of their `id` properties. For example, if the first job had the `id` value `240528T1316-mon-mozilla` and the events in the series occurred at intervals of 12 hours, then the second job would have the `id` value `240529T0116-mon-mozilla`.
|
|
422
|
-
|
|
423
|
-
The `series` module adds a `sources.series` property to each job in the series. The value of that property is the `id` value of the first job in the series.
|
|
424
|
-
|
|
425
|
-
To support monitoring, a server that receives job requests from testing agents can perform a time check on the first job in the queue. If the time specified by the `id` of the first job is in the future, the server can reply that there is no job to do.
|
|
426
|
-
|
|
427
|
-
#### Invocation
|
|
428
|
-
|
|
429
|
-
There are two ways to use the `series` module.
|
|
430
|
-
|
|
431
|
-
##### By a module
|
|
432
|
-
|
|
433
|
-
A module can invoke `series` in this way:
|
|
434
|
-
|
|
435
|
-
```javaScript
|
|
436
|
-
const {series} = require('testilo/series');
|
|
437
|
-
const jobs = series(job, count, interval);
|
|
438
|
-
```
|
|
439
|
-
|
|
440
|
-
This invocation references a `job` variable, whose value is a job object. The `count` variable is an integer, 2 or greater, specifying how many events the series consists of. The `interval` variable is an integer, 1 or greater, specifying how many minutes are to elapse after each event before the next event. The `series()` function of the `series` module generates an array of job objects and returns the array. The invoking module can further dispose of the jobs as needed.
|
|
441
|
-
|
|
442
|
-
##### By a user
|
|
443
|
-
|
|
444
|
-
A user can invoke `series` in this way:
|
|
445
|
-
|
|
446
|
-
- Create a job and save it as a JSON file in the `todo` subdirectory of the `process.env.JOBDIR` directory.
|
|
447
|
-
- In the Testilo project directory, execute this statement:
|
|
448
|
-
- `node call series j c i`
|
|
449
|
-
|
|
450
|
-
In this statement, replace:
|
|
451
|
-
- `j` with a string that the filename of the starting job begins with
|
|
452
|
-
- `c` with a count
|
|
453
|
-
- `i` with an interval in minutes
|
|
454
|
-
|
|
455
|
-
The `call` module will retrieve the first job that matches `j` from the `pending` subdirectory of the `process.env.JOBDIR` directory.
|
|
456
|
-
The `series` module will create an array of jobs.
|
|
457
|
-
The `call` module will save the jobs as JSON files in the `todo` subdirectory of the `process.env.JOBDIR` directory.
|
|
458
|
-
|
|
459
|
-
#### Validation
|
|
460
|
-
|
|
461
|
-
To test the `series` module, in the project directory you can execute the statement `node validation/series/validate`. If `series` is valid, all logging statements will begin with “Success” and none will begin with “ERROR”.
|
|
462
|
-
|
|
463
477
|
## Report scoring
|
|
464
478
|
|
|
465
479
|
### Introduction
|
package/call.js
CHANGED
|
@@ -24,8 +24,6 @@ const {batch} = require('./batch');
|
|
|
24
24
|
const {script} = require('./script');
|
|
25
25
|
// Function to process a merger.
|
|
26
26
|
const {merge} = require('./merge');
|
|
27
|
-
// Function to generate a job series.
|
|
28
|
-
const {series} = require('./series');
|
|
29
27
|
// Function to score reports.
|
|
30
28
|
const {score} = require('./score');
|
|
31
29
|
// Function to digest reports.
|
|
@@ -99,29 +97,6 @@ const callMerge = async (
|
|
|
99
97
|
`Script ${scriptID} and batch ${batchID} merged as ${timeStamp}-… in ${jobDir}/${destination}`
|
|
100
98
|
);
|
|
101
99
|
};
|
|
102
|
-
// Fulfills a series request.
|
|
103
|
-
const callSeries = async (idStart, count, interval) => {
|
|
104
|
-
// Get the initial job.
|
|
105
|
-
const jobNames = await fs.readdir(`${jobDir}/pending`);
|
|
106
|
-
const seriesJobName = jobNames.find(jobName => jobName.startsWith(idStart));
|
|
107
|
-
// If it exists:
|
|
108
|
-
if (seriesJobName) {
|
|
109
|
-
// Generate a job series.
|
|
110
|
-
const jobJSON = await fs.readFile(`${jobDir}/todo/${seriesJobName}`, 'utf8');
|
|
111
|
-
const job = JSON.parse(jobJSON);
|
|
112
|
-
const jobSeries = series(job, Number.parseInt(count), Number.parseInt(interval));
|
|
113
|
-
// Save the jobs.
|
|
114
|
-
for (const item of jobSeries) {
|
|
115
|
-
await fs.writeFile(`${jobDir}/todo/${item.id}.json`, `${JSON.stringify(item, null, 2)}\n`);
|
|
116
|
-
}
|
|
117
|
-
console.log(`Series of ${jobSeries.length} jobs generated and saved in ${jobDir}/todo`);
|
|
118
|
-
}
|
|
119
|
-
// Otherwise, i.e. if it does not exist:
|
|
120
|
-
else {
|
|
121
|
-
// Report this.
|
|
122
|
-
console.log('ERROR: No matching to-do job found');
|
|
123
|
-
}
|
|
124
|
-
};
|
|
125
100
|
// Gets selected reports.
|
|
126
101
|
const getReports = async (type, selector = '') => {
|
|
127
102
|
const allFileNames = await fs.readdir(`${reportDir}/${type}`);
|
|
@@ -253,12 +228,6 @@ else if (fn === 'merge' && fnArgs.length === 9) {
|
|
|
253
228
|
console.log('Execution completed');
|
|
254
229
|
});
|
|
255
230
|
}
|
|
256
|
-
else if (fn === 'series' && fnArgs.length === 3) {
|
|
257
|
-
callSeries(... fnArgs)
|
|
258
|
-
.then(() => {
|
|
259
|
-
console.log('Execution completed');
|
|
260
|
-
});
|
|
261
|
-
}
|
|
262
231
|
else if (fn === 'score' && fnArgs.length > 0 && fnArgs.length < 3) {
|
|
263
232
|
callScore(... fnArgs)
|
|
264
233
|
.then(() => {
|
package/package.json
CHANGED
package/series.js
DELETED
|
@@ -1,72 +0,0 @@
|
|
|
1
|
-
/*
|
|
2
|
-
series.js
|
|
3
|
-
Generates a series of Testaro jobs.
|
|
4
|
-
Arguments:
|
|
5
|
-
0. Initial job.
|
|
6
|
-
1. Job count.
|
|
7
|
-
2. Time interval in minutes.
|
|
8
|
-
*/
|
|
9
|
-
|
|
10
|
-
// ########## FUNCTIONS
|
|
11
|
-
|
|
12
|
-
// Scores the specified raw reports.
|
|
13
|
-
exports.series = (job, count, interval) => {
|
|
14
|
-
// If the arguments are valid:
|
|
15
|
-
if (
|
|
16
|
-
typeof job === 'object'
|
|
17
|
-
&& count
|
|
18
|
-
&& typeof count === 'number'
|
|
19
|
-
&& count === Math.floor(count)
|
|
20
|
-
&& count > 1
|
|
21
|
-
&& interval
|
|
22
|
-
&& typeof interval === 'number'
|
|
23
|
-
&& interval === Math.floor(interval)
|
|
24
|
-
&& interval > 0
|
|
25
|
-
) {
|
|
26
|
-
// Get a copy of the initial job.
|
|
27
|
-
const template = JSON.parse(JSON.stringify(job));
|
|
28
|
-
// If it has an ID:
|
|
29
|
-
const jobID = template.id;
|
|
30
|
-
if (jobID) {
|
|
31
|
-
// If the ID specifies a valid time:
|
|
32
|
-
const s = jobID.slice(0, 11);
|
|
33
|
-
const dateSpec = `20${s[0]}${s[1]}-${s[2]}${s[3]}-${s[4]}${s[5]}`;
|
|
34
|
-
const timeSpec = `${s[7]}${s[8]}:${s[9]}${s[10]}`;
|
|
35
|
-
const dateTimeSpec = `${dateSpec}T${timeSpec}Z`;
|
|
36
|
-
const start = new Date(dateTimeSpec);
|
|
37
|
-
const startNum = start.valueOf();
|
|
38
|
-
if (startNum) {
|
|
39
|
-
// Initialize the series.
|
|
40
|
-
const series = [];
|
|
41
|
-
// For each job required:
|
|
42
|
-
for (let i = 0; i < count; i++) {
|
|
43
|
-
// Create it.
|
|
44
|
-
const nextJob = JSON.parse(JSON.stringify(template));
|
|
45
|
-
nextJob.sources.series = nextJob.id;
|
|
46
|
-
// Revise its ID.
|
|
47
|
-
const nextDate = new Date(startNum + i * interval * 60000);
|
|
48
|
-
const nextTimeStamp = nextDate.toISOString().slice(2, 16).replace(/[-:]/g, '');
|
|
49
|
-
nextJob.id = nextJob.id.replace(/^[^-]+/, nextTimeStamp);
|
|
50
|
-
// Add the job to the series.
|
|
51
|
-
series.push(nextJob);
|
|
52
|
-
}
|
|
53
|
-
return series;
|
|
54
|
-
}
|
|
55
|
-
// Otherwise, i.e. if it does not specify a valid time:
|
|
56
|
-
else {
|
|
57
|
-
// Report this.
|
|
58
|
-
console.log('ERROR: Initial job ID starts with an invalid time specification');
|
|
59
|
-
}
|
|
60
|
-
}
|
|
61
|
-
// Otherwise, i.e. if it has no ID:
|
|
62
|
-
else {
|
|
63
|
-
// Report this.
|
|
64
|
-
console.log('ERROR: Initial job has no ID');
|
|
65
|
-
}
|
|
66
|
-
}
|
|
67
|
-
// Otherwise, i.e. if they are invalid:
|
|
68
|
-
else {
|
|
69
|
-
// Report this.
|
|
70
|
-
console.log('ERROR: Arguments invalid');
|
|
71
|
-
}
|
|
72
|
-
};
|
|
@@ -1,37 +0,0 @@
|
|
|
1
|
-
{
|
|
2
|
-
"id": "231120T155027-mon-example",
|
|
3
|
-
"what": "Job for series validation",
|
|
4
|
-
"strict": true,
|
|
5
|
-
"timeLimit": 10,
|
|
6
|
-
"acts": [
|
|
7
|
-
{
|
|
8
|
-
"type": "launch",
|
|
9
|
-
"which": "chromium",
|
|
10
|
-
"url": "https://example.com",
|
|
11
|
-
"what": "Example of web page",
|
|
12
|
-
"startTime": 1662474496075,
|
|
13
|
-
"endTime": 1662474496453
|
|
14
|
-
},
|
|
15
|
-
{
|
|
16
|
-
"type": "test",
|
|
17
|
-
"which": "testaro",
|
|
18
|
-
"url": "https://example.com",
|
|
19
|
-
"withItems": false,
|
|
20
|
-
"rules": [
|
|
21
|
-
"y",
|
|
22
|
-
"bulk"
|
|
23
|
-
]
|
|
24
|
-
}
|
|
25
|
-
],
|
|
26
|
-
"sources": {
|
|
27
|
-
"script": "mon",
|
|
28
|
-
"batch": "target",
|
|
29
|
-
"target": {
|
|
30
|
-
"id": "example",
|
|
31
|
-
"what": "Example of web page"
|
|
32
|
-
},
|
|
33
|
-
"requester": "user@domain.tld"
|
|
34
|
-
},
|
|
35
|
-
"creationTime": "2023-11-20T15:50:27",
|
|
36
|
-
"timeStamp": "231120T155027"
|
|
37
|
-
}
|
|
@@ -1,78 +0,0 @@
|
|
|
1
|
-
/*
|
|
2
|
-
validate.js
|
|
3
|
-
Validates series module.
|
|
4
|
-
*/
|
|
5
|
-
|
|
6
|
-
// ########## IMPORTS
|
|
7
|
-
|
|
8
|
-
// Function to process files.
|
|
9
|
-
const fs = require('fs/promises');
|
|
10
|
-
// Function to generate a job series.
|
|
11
|
-
const {series} = require('../../series');
|
|
12
|
-
|
|
13
|
-
// ########## FUNCTIONS
|
|
14
|
-
|
|
15
|
-
// Validates the series module.
|
|
16
|
-
const validate = async () => {
|
|
17
|
-
// Get the job.
|
|
18
|
-
const jobJSON = await fs.readFile(`${__dirname}/job.json`, 'utf8');
|
|
19
|
-
const job = JSON.parse(jobJSON);
|
|
20
|
-
// Generate the series.
|
|
21
|
-
const jobs = series(job, 3, 5);
|
|
22
|
-
// Validate the series.
|
|
23
|
-
if (Array.isArray(jobs) && jobs.length === 3) {
|
|
24
|
-
console.log('Success: The count of jobs is correct');
|
|
25
|
-
}
|
|
26
|
-
else {
|
|
27
|
-
console.log('ERROR: The jobs are not an array of length 3');
|
|
28
|
-
return;
|
|
29
|
-
}
|
|
30
|
-
const job0 = jobs[0];
|
|
31
|
-
if (
|
|
32
|
-
job.id
|
|
33
|
-
&& job.id === '240223T0815-mon-example'
|
|
34
|
-
&& job0.id === job.id
|
|
35
|
-
&& job0.sources
|
|
36
|
-
&& job0.sources.series
|
|
37
|
-
&& job0.sources.series === job.id
|
|
38
|
-
) {
|
|
39
|
-
console.log('Success: The first job has the correct id and sources.series');
|
|
40
|
-
}
|
|
41
|
-
else {
|
|
42
|
-
console.log('ERROR: The first job has an incorrect id or sources.series');
|
|
43
|
-
return;
|
|
44
|
-
}
|
|
45
|
-
const job1 = jobs[1];
|
|
46
|
-
const job2 = jobs[2];
|
|
47
|
-
if (
|
|
48
|
-
job2.id
|
|
49
|
-
&& job2.id === '240223T0825-mon-example'
|
|
50
|
-
&& job2.sources
|
|
51
|
-
&& job2.sources.series
|
|
52
|
-
&& job2.sources.series === '240223T0815-mon-example'
|
|
53
|
-
) {
|
|
54
|
-
console.log('Success: The third job has the correct id and sources.series');
|
|
55
|
-
}
|
|
56
|
-
else {
|
|
57
|
-
console.log('ERROR: The first job has an incorrect id or sources.series');
|
|
58
|
-
return;
|
|
59
|
-
}
|
|
60
|
-
if (
|
|
61
|
-
job1.acts
|
|
62
|
-
&& job1.acts.length === 3
|
|
63
|
-
&& job1.acts[2].rules
|
|
64
|
-
&& job1.acts[2].rules.length === 2
|
|
65
|
-
&& job2.acts
|
|
66
|
-
&& job2.acts.length === 3
|
|
67
|
-
&& job2.acts[2].rules
|
|
68
|
-
&& job2.acts[2].rules.length === 2
|
|
69
|
-
&& job2.acts[2].rules[1] === job1.acts[2].rules[1]
|
|
70
|
-
) {
|
|
71
|
-
console.log('Success: The second and third jobs invoke the same rule');
|
|
72
|
-
}
|
|
73
|
-
else {
|
|
74
|
-
console.log('ERROR: The second and third job do not invoke the same rule');
|
|
75
|
-
return;
|
|
76
|
-
}
|
|
77
|
-
};
|
|
78
|
-
validate();
|