@jrpool/kilotest 50.0.0 → 52.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/workflows/ci.yml +2 -2
- package/.github/workflows/periodic-smoke-tests.yml +2 -2
- package/.github/workflows/publish-mcp.yml +1 -1
- package/README.md +5 -5
- package/api/requestFeature.test.ts +81 -3
- package/api/requestFeature.ts +46 -10
- package/api/requestRetest.test.ts +7 -0
- package/api/requestTest.test.ts +23 -0
- package/api/requestTest.ts +7 -1
- package/api/util.test.ts +9 -0
- package/apiIntegration.test.ts +17 -0
- package/docs/SERVICE.md +27 -6
- package/env.example +8 -0
- package/index.test.ts +137 -0
- package/index.ts +52 -5
- package/linkValidation.test.ts +1 -0
- package/package.json +1 -1
- package/robots.txt +1 -0
- package/smokeTest.ts +10 -5
- package/test/setup.ts +5 -0
- package/util.test.ts +171 -13
- package/util.ts +127 -16
- package/web/deleteNotesForm/deleteNotesForm.test.ts +237 -0
- package/web/deleteNotesForm/index.html +29 -0
- package/web/deleteNotesForm/index.ts +167 -0
- package/web/manage/index.html +1 -0
- package/web/tutorialAI/index.ts +6 -11
- package/web/tutorialAI/tutorialAI.test.ts +9 -3
- package/web/tutorialWeb/tutorialWeb.test.ts +8 -2
- package/.claude/scheduled_tasks.lock +0 -1
package/.github/workflows/ci.yml
CHANGED
package/README.md
CHANGED
|
@@ -10,7 +10,7 @@ An ensemble testing and reporting service for front-end web quality
|
|
|
10
10
|
|
|
11
11
|
## Features
|
|
12
12
|
|
|
13
|
-
This application uses an ensemble of 12 rule engines to test
|
|
13
|
+
This application uses an ensemble of 12 rule engines to test web pages for front-end quality (i.e. accessibility, usability, and standards conformity).
|
|
14
14
|
|
|
15
15
|
The testing paradigm employed by Kilotest is discussed in these papers:
|
|
16
16
|
|
|
@@ -20,11 +20,11 @@ The testing paradigm employed by Kilotest is discussed in these papers:
|
|
|
20
20
|
|
|
21
21
|
Kilotest acts as a web server for human users, an MCP server for AI platforms, and an API for programmatic access.
|
|
22
22
|
|
|
23
|
-
Testing for Kilotest is performed by one or more testing agents that obtain jobs from Kilotest, perform them, and send reports of the results back to Kilotest. Those agents are instances of the [Testaro](https://www.npmjs.com/package/testaro) package.
|
|
23
|
+
Testing for Kilotest is performed by one or more workers (testing agents) that obtain jobs from Kilotest, perform them, and send reports of the results back to Kilotest. Those agents are instances of the [Testaro](https://www.npmjs.com/package/testaro) package. Thus, the pages that Kilotest tests need not be on the public Internet. They need only be reachable by the Testaro workers that do the testing for Kilotest. So, one or more Testaro workers running inside a private intranet can test internal pages, as well as any external ones reachable from inside the intranet.
|
|
24
24
|
|
|
25
|
-
An active production instance of Kilotest may require multiple
|
|
25
|
+
An active production instance of Kilotest may require multiple Testaro workers to handle the load, because testing one web page typically takes about 3 minutes and workers test only one page at a time.
|
|
26
26
|
|
|
27
|
-
## Getting started locally with 1
|
|
27
|
+
## Getting started locally with 1 worker
|
|
28
28
|
|
|
29
29
|
### Installation
|
|
30
30
|
|
|
@@ -36,7 +36,7 @@ In the steps below, hosts `T` and `K` may be the same host or two different host
|
|
|
36
36
|
1. Clone the Kilotest repository into a new directory on host `K`.
|
|
37
37
|
1. In that directory, install the Kilotest dependencies: `npm install`.
|
|
38
38
|
1. Copy the `env.testaro` file from the `kilotest` directory to `.env` in the `testaro` directory and replace the `__placeholder__` values in `.env` with actual values.
|
|
39
|
-
1. Copy the `env.example` file in the `kilotest` directory to a new `.env` file in the same directory and replace the `__placeholder__` values in `.env` with actual values. `TESTARO_WORKERS` is a JSON object mapping each Testaro worker's ID to an object with its `secret` and its `name`; the ID and secret for a worker must match the `NETWATCH_URL_WORKER_ID` and `NETWATCH_URL_AUTH` values in that worker's `.env`. A worker authenticates each request with an HTTP Basic `Authorization` header (`id:secret`, base64-encoded); as of this writing that header is not yet sent by the published Testaro package, so this scheme requires a matching update to Testaro's `netWatch.js` request code (only to send the header; no change to Testaro's handling of jobs or reports is needed). The ID is a credential and is never published; the `name` is written by Kilotest into a job's and report's `sources.worker` property, which Testaro does not read or alter, so it is a distinct property from `sources.agent` (Testaro's own, untrusted, self-reported label, which Kilotest ignores). Choose a `name` that is safe to appear in every report Kilotest serves.
|
|
39
|
+
1. Copy the `env.example` file in the `kilotest` directory to a new `.env` file in the same directory and replace the `__placeholder__` values in `.env` with actual values. `TESTARO_WORKERS` is a JSON object mapping each Testaro worker's ID to an object with its `secret` and its `name`; the ID and secret for a worker must match the `NETWATCH_URL_WORKER_ID` and `NETWATCH_URL_AUTH` values in that worker's `.env`. A worker authenticates each request with an HTTP Basic `Authorization` header (`id:secret`, base64-encoded); as of this writing that header is not yet sent by the published Testaro package, so this scheme requires a matching update to Testaro's `netWatch.js` request code (only to send the header; no change to Testaro's handling of jobs or reports is needed). The ID is a credential and is never published; the `name` is written by Kilotest into a job's and report's `sources.worker` property, which Testaro does not read or alter, so it is a distinct property from `sources.agent` (Testaro's own, untrusted, self-reported label, which Kilotest ignores). Choose a `name` that is safe to appear in every report Kilotest serves. `ALLOW_INTERNAL_TARGETS` controls whether this deployment can be induced to test pages at private, loopback, or link-local addresses; see [`SERVICE.md`](docs/SERVICE.md#testing-target-restrictions) before changing it from its default.
|
|
40
40
|
|
|
41
41
|
### Usage
|
|
42
42
|
|
|
@@ -7,6 +7,20 @@
|
|
|
7
7
|
|
|
8
8
|
import {test, beforeEach, after} from 'node:test';
|
|
9
9
|
import assert from 'node:assert/strict';
|
|
10
|
+
import fs from 'node:fs/promises';
|
|
11
|
+
import path from 'node:path';
|
|
12
|
+
|
|
13
|
+
// Blank the alert configuration unconditionally, before ./requestFeature.ts (which
|
|
14
|
+
// calls sendAlert) is ever imported below, so this file sends no real alert emails
|
|
15
|
+
// even when run directly rather than via `npm test`.
|
|
16
|
+
for (const key of ['MANAGER_EMAIL', 'ALERT_API_HOST', 'ALERT_API_PATH', 'ALERT_API_KEY', 'ALERT_FROM']) {
|
|
17
|
+
process.env[key] = '';
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
// ENVIRONMENT
|
|
21
|
+
|
|
22
|
+
const testDir = path.join(import.meta.dirname, '../test/fixtures/featureRequests');
|
|
23
|
+
process.env.FEATURE_REQUESTS_PATH = path.join(testDir, 'featureRequests.json');
|
|
10
24
|
|
|
11
25
|
// SETUP AND TEARDOWN
|
|
12
26
|
|
|
@@ -14,16 +28,19 @@ let logged: any[] = [];
|
|
|
14
28
|
const originalLog = console.log;
|
|
15
29
|
|
|
16
30
|
// Capture console.log so the alert sendAlert emits can be observed. The alert
|
|
17
|
-
// configuration is
|
|
18
|
-
beforeEach(() => {
|
|
31
|
+
// configuration is blanked above, so sendAlert logs a WARNING instead of sending.
|
|
32
|
+
beforeEach(async () => {
|
|
19
33
|
logged = [];
|
|
20
34
|
console.log = (...args) => logged.push(args.join(' '));
|
|
35
|
+
await fs.mkdir(testDir, {recursive: true});
|
|
36
|
+
await fs.writeFile(process.env.FEATURE_REQUESTS_PATH!, '[]\n');
|
|
21
37
|
});
|
|
22
38
|
|
|
23
39
|
import {response} from './requestFeature.ts';
|
|
24
40
|
|
|
25
|
-
after(() => {
|
|
41
|
+
after(async () => {
|
|
26
42
|
console.log = originalLog;
|
|
43
|
+
await fs.rm(testDir, {recursive: true, force: true});
|
|
27
44
|
});
|
|
28
45
|
|
|
29
46
|
// TESTS
|
|
@@ -72,3 +89,64 @@ test('requestFeature includes tool name and metadata', async () => {
|
|
|
72
89
|
assert.equal(body['tool name'], 'requestFeature');
|
|
73
90
|
assert.ok(body['response metadata'].identifier);
|
|
74
91
|
});
|
|
92
|
+
|
|
93
|
+
test('requestFeature records a valid feature request to the feature-requests file', async () => {
|
|
94
|
+
await response(['Add a dark mode toggle to the reports page']);
|
|
95
|
+
const featureRequests = JSON.parse(await fs.readFile(process.env.FEATURE_REQUESTS_PATH!, 'utf8'));
|
|
96
|
+
assert.equal(featureRequests.length, 1);
|
|
97
|
+
assert.equal(featureRequests[0].content, 'Add a dark mode toggle to the reports page');
|
|
98
|
+
assert.ok(featureRequests[0].timeStamp);
|
|
99
|
+
});
|
|
100
|
+
|
|
101
|
+
test('requestFeature appends to, rather than overwrites, existing feature requests', async () => {
|
|
102
|
+
await response(['First feature request of at least 20 characters']);
|
|
103
|
+
await response(['Second feature request of at least 20 characters']);
|
|
104
|
+
const featureRequests = JSON.parse(await fs.readFile(process.env.FEATURE_REQUESTS_PATH!, 'utf8'));
|
|
105
|
+
assert.equal(featureRequests.length, 2);
|
|
106
|
+
});
|
|
107
|
+
|
|
108
|
+
test('requestFeature does not record an invalid feature request', async () => {
|
|
109
|
+
await response(['short']);
|
|
110
|
+
const featureRequests = JSON.parse(await fs.readFile(process.env.FEATURE_REQUESTS_PATH!, 'utf8'));
|
|
111
|
+
assert.equal(featureRequests.length, 0);
|
|
112
|
+
});
|
|
113
|
+
|
|
114
|
+
test('requestFeature creates the feature-requests file when it does not exist', async () => {
|
|
115
|
+
await fs.unlink(process.env.FEATURE_REQUESTS_PATH!);
|
|
116
|
+
await response(['A feature request when no file yet exists']);
|
|
117
|
+
const featureRequests = JSON.parse(await fs.readFile(process.env.FEATURE_REQUESTS_PATH!, 'utf8'));
|
|
118
|
+
assert.equal(featureRequests.length, 1);
|
|
119
|
+
assert.equal(featureRequests[0].content, 'A feature request when no file yet exists');
|
|
120
|
+
});
|
|
121
|
+
|
|
122
|
+
test('requestFeature rejects a feature request identical to one already on file', async () => {
|
|
123
|
+
const feature = 'Add a dark mode toggle to the reports page';
|
|
124
|
+
const firstBody = await response([feature]);
|
|
125
|
+
const firstDetails = firstBody['response content']['details about your request'] as any;
|
|
126
|
+
assert.equal(firstDetails.error, undefined);
|
|
127
|
+
const secondBody = await response([feature]);
|
|
128
|
+
const secondDetails = secondBody['response content']['details about your request'] as any;
|
|
129
|
+
assert.ok(secondDetails.error.includes('identical to one already submitted'));
|
|
130
|
+
const featureRequests = JSON.parse(await fs.readFile(process.env.FEATURE_REQUESTS_PATH!, 'utf8'));
|
|
131
|
+
// The duplicate was not recorded a second time.
|
|
132
|
+
assert.equal(featureRequests.length, 1);
|
|
133
|
+
});
|
|
134
|
+
|
|
135
|
+
test('requestFeature does not notify the manager of a duplicate feature request', async () => {
|
|
136
|
+
const feature = 'Add a dark mode toggle to the settings page';
|
|
137
|
+
await response([feature]);
|
|
138
|
+
logged = [];
|
|
139
|
+
await response([feature]);
|
|
140
|
+
assert.ok(
|
|
141
|
+
!logged.some(line => line.startsWith('WARNING (Kilotest: MCP feature request received)'))
|
|
142
|
+
);
|
|
143
|
+
});
|
|
144
|
+
|
|
145
|
+
test('requestFeature accepts a feature request that differs from an existing one', async () => {
|
|
146
|
+
await response(['An existing feature request on file']);
|
|
147
|
+
const body = await response(['A different feature request entirely']);
|
|
148
|
+
const details = body['response content']['details about your request'] as any;
|
|
149
|
+
assert.equal(details.error, undefined);
|
|
150
|
+
const featureRequests = JSON.parse(await fs.readFile(process.env.FEATURE_REQUESTS_PATH!, 'utf8'));
|
|
151
|
+
assert.equal(featureRequests.length, 2);
|
|
152
|
+
});
|
package/api/requestFeature.ts
CHANGED
|
@@ -8,14 +8,20 @@
|
|
|
8
8
|
import {z} from 'zod';
|
|
9
9
|
import {getResponseMetadata, getThisHost, getToolsFacts} from './util.ts';
|
|
10
10
|
import {sendAlert} from '../alerts.ts';
|
|
11
|
-
import {checkLength} from '../util.ts';
|
|
11
|
+
import {checkCommentDuplicate, checkLength, getJSON, getNowStamp} from '../util.ts';
|
|
12
12
|
import {requestFeatureResponseSchema} from './schemas.ts';
|
|
13
|
+
import fs from 'node:fs/promises';
|
|
14
|
+
import path from 'node:path';
|
|
13
15
|
|
|
14
16
|
// TYPES
|
|
15
17
|
|
|
16
18
|
// The response content defined by the response schema.
|
|
17
19
|
type ResponseContent = z.infer<typeof requestFeatureResponseSchema>['response content'];
|
|
18
20
|
|
|
21
|
+
// FUNCTIONS (helpers)
|
|
22
|
+
|
|
23
|
+
const getFeatureRequestsPath = () => process.env.FEATURE_REQUESTS_PATH || path.join(import.meta.dirname, '../db/featureRequests.json');
|
|
24
|
+
|
|
19
25
|
// FUNCTIONS
|
|
20
26
|
|
|
21
27
|
// Returns the response body.
|
|
@@ -35,15 +41,45 @@ export const response = async (args: string[]) => {
|
|
|
35
41
|
}
|
|
36
42
|
// Otherwise, i.e. if it is valid:
|
|
37
43
|
else {
|
|
38
|
-
//
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
44
|
+
// Get the feature requests already on file.
|
|
45
|
+
const featureRequestsPath = getFeatureRequestsPath();
|
|
46
|
+
let featureRequests: {timeStamp: string; content: string}[] = [];
|
|
47
|
+
try {
|
|
48
|
+
const existing = await fs.readFile(featureRequestsPath, 'utf8');
|
|
49
|
+
featureRequests = JSON.parse(existing);
|
|
50
|
+
}
|
|
51
|
+
catch {
|
|
52
|
+
// Initialize an empty feature-requests array.
|
|
53
|
+
}
|
|
54
|
+
// If the request is identical to one already on file:
|
|
55
|
+
const duplicateCheck = checkCommentDuplicate(featureRequests, feature);
|
|
56
|
+
if (duplicateCheck.status === 'error') {
|
|
57
|
+
responseContent = {
|
|
58
|
+
'details about your request': {
|
|
59
|
+
error: 'request invalid: your feature request is identical to one already submitted, ' +
|
|
60
|
+
'but you are welcome to submit a different one'
|
|
61
|
+
}
|
|
62
|
+
};
|
|
63
|
+
}
|
|
64
|
+
// Otherwise, i.e. if it is not a duplicate:
|
|
65
|
+
else {
|
|
66
|
+
// Record the feature request alongside the existing ones.
|
|
67
|
+
featureRequests.push({
|
|
68
|
+
timeStamp: getNowStamp(),
|
|
69
|
+
content: feature
|
|
70
|
+
});
|
|
71
|
+
await fs.mkdir(path.dirname(featureRequestsPath), {recursive: true});
|
|
72
|
+
await fs.writeFile(featureRequestsPath, getJSON(featureRequests));
|
|
73
|
+
// Notify the manager.
|
|
74
|
+
await sendAlert('Kilotest: MCP feature request received', feature);
|
|
75
|
+
// Add the disposition to the response content.
|
|
76
|
+
responseContent = {
|
|
77
|
+
'details about your request': {
|
|
78
|
+
'date and time received': new Date().toISOString(),
|
|
79
|
+
disposition: 'received and logged; manager notified'
|
|
80
|
+
}
|
|
81
|
+
};
|
|
82
|
+
}
|
|
47
83
|
}
|
|
48
84
|
// Create a response body.
|
|
49
85
|
const body = {
|
|
@@ -11,6 +11,13 @@ import fs from 'node:fs/promises';
|
|
|
11
11
|
import path from 'node:path';
|
|
12
12
|
import {fixtureDBDir} from '../test/dbFixture.ts';
|
|
13
13
|
|
|
14
|
+
// Blank the alert configuration unconditionally, before ./requestRetest.ts (whose
|
|
15
|
+
// processTestRequest calls sendAlert) is ever imported below, so this file sends no
|
|
16
|
+
// real alert emails even when run directly rather than via `npm test`.
|
|
17
|
+
for (const key of ['MANAGER_EMAIL', 'ALERT_API_HOST', 'ALERT_API_PATH', 'ALERT_API_KEY', 'ALERT_FROM']) {
|
|
18
|
+
process.env[key] = '';
|
|
19
|
+
}
|
|
20
|
+
|
|
14
21
|
// SETUP AND TEARDOWN
|
|
15
22
|
|
|
16
23
|
const savedDBDir = process.env.DB_DIR;
|
package/api/requestTest.test.ts
CHANGED
|
@@ -11,6 +11,17 @@ import fs from 'node:fs/promises';
|
|
|
11
11
|
import path from 'node:path';
|
|
12
12
|
import {fixtureDBDir} from '../test/dbFixture.ts';
|
|
13
13
|
|
|
14
|
+
// Blank the alert configuration unconditionally, before ./requestTest.ts (whose
|
|
15
|
+
// processTestRequest calls sendAlert) is ever imported below, so this file sends no
|
|
16
|
+
// real alert emails even when run directly rather than via `npm test`.
|
|
17
|
+
for (const key of ['MANAGER_EMAIL', 'ALERT_API_HOST', 'ALERT_API_PATH', 'ALERT_API_KEY', 'ALERT_FROM']) {
|
|
18
|
+
process.env[key] = '';
|
|
19
|
+
}
|
|
20
|
+
// Allow internal targets by default, so tests that submit an ordinary https://example.com/...
|
|
21
|
+
// URL do not depend on real DNS/network access to pass the resolution check that
|
|
22
|
+
// isAllowedTarget performs. The test of the check itself (below) overrides this.
|
|
23
|
+
process.env.ALLOW_INTERNAL_TARGETS = 'true';
|
|
24
|
+
|
|
14
25
|
// SETUP AND TEARDOWN
|
|
15
26
|
|
|
16
27
|
const savedDBDir = process.env.DB_DIR;
|
|
@@ -70,6 +81,18 @@ test('requestTest rejects a syntactically invalid URL with the correct length',
|
|
|
70
81
|
assert.ok(!logged.some(line => line.includes('new test request awaits approval')));
|
|
71
82
|
});
|
|
72
83
|
|
|
84
|
+
test('requestTest rejects a private-address URL unless internal targets are allowed', async () => {
|
|
85
|
+
delete process.env.ALLOW_INTERNAL_TARGETS;
|
|
86
|
+
try {
|
|
87
|
+
const body = await response(['Internal Page', 'https://192.168.1.1/page', 'A reason that is long enough.']);
|
|
88
|
+
const details = body['response content']['details about your request'] as any;
|
|
89
|
+
assert.ok(details.error.includes('does not resolve to a page that this deployment can test'));
|
|
90
|
+
assert.ok(!logged.some(line => line.includes('new test request awaits approval')));
|
|
91
|
+
} finally {
|
|
92
|
+
process.env.ALLOW_INTERNAL_TARGETS = 'true';
|
|
93
|
+
}
|
|
94
|
+
});
|
|
95
|
+
|
|
73
96
|
test('requestTest rejects an already-tested page', async () => {
|
|
74
97
|
const body = await response(['Mixed Outcomes Page', 'https://example.com/mixed', 'A reason that is long enough.']);
|
|
75
98
|
const details = body['response content']['details about your request'] as any;
|
package/api/requestTest.ts
CHANGED
|
@@ -7,7 +7,7 @@
|
|
|
7
7
|
|
|
8
8
|
import {z} from 'zod';
|
|
9
9
|
import {getResponseMetadata, getThisHost, getToolsFacts} from './util.ts';
|
|
10
|
-
import {checkLength, isURL, processTestRequest} from '../util.ts';
|
|
10
|
+
import {checkLength, isAllowedTarget, isURL, processTestRequest} from '../util.ts';
|
|
11
11
|
import {requestTestResponseSchema} from './schemas.ts';
|
|
12
12
|
|
|
13
13
|
// TYPES
|
|
@@ -43,6 +43,12 @@ export const response = async (args: string[]) => {
|
|
|
43
43
|
error: 'request invalid: you specified an invalid URL for the page'
|
|
44
44
|
};
|
|
45
45
|
}
|
|
46
|
+
// Otherwise, i.e. if the URL does not resolve to an allowed target:
|
|
47
|
+
else if (!(await isAllowedTarget(url))) {
|
|
48
|
+
requestDetails = {
|
|
49
|
+
error: 'request invalid: the URL you specified does not resolve to a page that this deployment can test'
|
|
50
|
+
};
|
|
51
|
+
}
|
|
46
52
|
// Otherwise, i.e. if the description and URL are valid:
|
|
47
53
|
else {
|
|
48
54
|
// Process the request.
|
package/api/util.test.ts
CHANGED
|
@@ -7,6 +7,15 @@
|
|
|
7
7
|
|
|
8
8
|
import {test} from 'node:test';
|
|
9
9
|
import assert from 'node:assert/strict';
|
|
10
|
+
|
|
11
|
+
// Blank the alert configuration unconditionally, before ../util.ts (which
|
|
12
|
+
// processTestRequest calls sendAlert from) is ever imported below, so this file sends
|
|
13
|
+
// no real alert emails even when run directly (e.g. `npx tsx --test api/util.test.ts`)
|
|
14
|
+
// rather than via `npm test`, which normally guards against this via test/setup.ts.
|
|
15
|
+
for (const key of ['MANAGER_EMAIL', 'ALERT_API_HOST', 'ALERT_API_PATH', 'ALERT_API_KEY', 'ALERT_FROM']) {
|
|
16
|
+
process.env[key] = '';
|
|
17
|
+
}
|
|
18
|
+
|
|
10
19
|
import {getReportBasics, getResponseMetadata, getRuleEngineFacts, getRuleEnginesFacts, getIssueSpec, getToolsFacts} from './util.ts';
|
|
11
20
|
import {processTestRequest} from '../util.ts';
|
|
12
21
|
|
package/apiIntegration.test.ts
CHANGED
|
@@ -8,19 +8,29 @@
|
|
|
8
8
|
import {test, before, after} from 'node:test';
|
|
9
9
|
import assert from 'node:assert/strict';
|
|
10
10
|
import http from 'node:http';
|
|
11
|
+
import fs from 'node:fs/promises';
|
|
12
|
+
import path from 'node:path';
|
|
11
13
|
import {requestHandler} from './index.ts';
|
|
12
14
|
|
|
13
15
|
// CONSTANTS
|
|
14
16
|
|
|
15
17
|
const port = 3998;
|
|
18
|
+
const featureRequestsPath = path.join(import.meta.dirname, 'test/fixtures/apiIntegration/featureRequests.json');
|
|
16
19
|
|
|
17
20
|
// SETUP AND TEARDOWN
|
|
18
21
|
|
|
19
22
|
const savedDBDir = process.env.DB_DIR;
|
|
23
|
+
const savedFeatureRequestsPath = process.env.FEATURE_REQUESTS_PATH;
|
|
20
24
|
let server: http.Server;
|
|
21
25
|
|
|
22
26
|
before(async () => {
|
|
23
27
|
process.env.DB_DIR = (await import('./test/dbFixture.ts')).fixtureDBDir;
|
|
28
|
+
// Use a fixture path, rather than the real default, for feature requests, so this
|
|
29
|
+
// test's submission neither pollutes real data nor is rejected as a duplicate of
|
|
30
|
+
// whatever a prior run of this same test already left on disk.
|
|
31
|
+
process.env.FEATURE_REQUESTS_PATH = featureRequestsPath;
|
|
32
|
+
await fs.mkdir(path.dirname(featureRequestsPath), {recursive: true});
|
|
33
|
+
await fs.writeFile(featureRequestsPath, '[]\n');
|
|
24
34
|
server = http.createServer(requestHandler);
|
|
25
35
|
await new Promise<void>(resolve => server.listen(port, () => resolve()));
|
|
26
36
|
});
|
|
@@ -32,6 +42,13 @@ after(async () => {
|
|
|
32
42
|
else {
|
|
33
43
|
delete process.env.DB_DIR;
|
|
34
44
|
}
|
|
45
|
+
if (savedFeatureRequestsPath !== undefined) {
|
|
46
|
+
process.env.FEATURE_REQUESTS_PATH = savedFeatureRequestsPath;
|
|
47
|
+
}
|
|
48
|
+
else {
|
|
49
|
+
delete process.env.FEATURE_REQUESTS_PATH;
|
|
50
|
+
}
|
|
51
|
+
await fs.rm(path.dirname(featureRequestsPath), {recursive: true, force: true});
|
|
35
52
|
server.closeAllConnections?.();
|
|
36
53
|
await new Promise<void>(resolve => {
|
|
37
54
|
const timer = setTimeout(() => {
|
package/docs/SERVICE.md
CHANGED
|
@@ -256,14 +256,21 @@ kilotest.com {
|
|
|
256
256
|
}
|
|
257
257
|
# Enable Zstandard and Gzip compression of responses.
|
|
258
258
|
encode zstd gzip
|
|
259
|
-
# Specify the only paths of forwardable requests.
|
|
259
|
+
# Specify the only paths of forwardable requests.
|
|
260
260
|
@allowedGET {
|
|
261
|
-
method GET
|
|
261
|
+
method GET
|
|
262
262
|
path /mcp / /index.html /robots.txt /openapi.yaml /openapi.json /swagger.yaml /swagger.json /api-docs /llms.txt /llms-full.txt /capability.md *.html* /fullReport.json/* /api/* /tutorialWeb/images/* /tutorialAI/images/* /qai /qai/comments /favicon.* /style.css /sitemap.xml
|
|
263
263
|
}
|
|
264
|
+
# UptimeRobot's uptime check is the only known HEAD caller, and it targets only the site
|
|
265
|
+
# root (see "Health monitoring" below), so HEAD is forwarded only for that path, not the
|
|
266
|
+
# full GET allowlist above. index.ts's HEAD branch matches this same scope.
|
|
267
|
+
@allowedHEAD {
|
|
268
|
+
method HEAD
|
|
269
|
+
path / /index.html
|
|
270
|
+
}
|
|
264
271
|
@allowedPOST {
|
|
265
272
|
method POST
|
|
266
|
-
path /mcp /requestRetest.html/* /requestTest.html /requestAction.html /reannotate.html /renewWCAG.html /ai0BalanceForm.html /expungeReportsForm.html /hideReportForm.html /metrics.html /pruneReportsForm.html /rewindReportsForm.html /showHiddenReportsForm.html /unhideReportForm.html /api/* /tutorialWebComment.html /tutorialAIComment.html /worker/job /worker/report
|
|
273
|
+
path /mcp /requestRetest.html/* /requestTest.html /requestAction.html /reannotate.html /renewWCAG.html /ai0BalanceForm.html /deleteNotesForm.html /expungeReportsForm.html /hideReportForm.html /metrics.html /pruneReportsForm.html /rewindReportsForm.html /showHiddenReportsForm.html /unhideReportForm.html /api/* /tutorialWebComment.html /tutorialAIComment.html /worker/job /worker/report
|
|
267
274
|
}
|
|
268
275
|
# Respond to OPTIONS requests.
|
|
269
276
|
@allowedOPTIONS method OPTIONS
|
|
@@ -275,13 +282,19 @@ kilotest.com {
|
|
|
275
282
|
}
|
|
276
283
|
respond 204
|
|
277
284
|
}
|
|
278
|
-
# Forward any other GET,
|
|
285
|
+
# Forward any other GET, HEAD, or POST request, if allowed, to port 3000.
|
|
279
286
|
handle @allowedGET {
|
|
280
287
|
reverse_proxy localhost:3000 {
|
|
281
288
|
# Improve SSE latency.
|
|
282
289
|
flush_interval -1
|
|
283
290
|
}
|
|
284
291
|
}
|
|
292
|
+
handle @allowedHEAD {
|
|
293
|
+
reverse_proxy localhost:3000 {
|
|
294
|
+
# Improve SSE latency.
|
|
295
|
+
flush_interval -1
|
|
296
|
+
}
|
|
297
|
+
}
|
|
285
298
|
handle @allowedPOST {
|
|
286
299
|
reverse_proxy localhost:3000 {
|
|
287
300
|
# Improve SSE latency.
|
|
@@ -362,7 +375,7 @@ The deployment is monitored for external health and availability using [UptimeRo
|
|
|
362
375
|
|
|
363
376
|
UptimeRobot also sends a recovery message when the service recovers and a subsequent check succeeds after a prior failure.
|
|
364
377
|
|
|
365
|
-
This monitoring service requires `Caddyfile` to permit `HEAD` requests, not only `GET` requests, to the root path.
|
|
378
|
+
This monitoring service requires both `Caddyfile` and Kilotest itself to permit `HEAD` requests, not only `GET` requests, to the root path: `Caddyfile`’s `@allowedHEAD` matcher forwards such a request to port 3000, and `index.ts`’s own dedicated `HEAD` branch in `handleRequest` answers it. Before that branch existed, Kilotest treated `HEAD` as an invalid method for every path (its dispatch recognized only `GET` and `POST`), so UptimeRobot’s hourly check received a `400` response and reported the service as down even while it was actually up.
|
|
366
379
|
|
|
367
380
|
## Alerting
|
|
368
381
|
|
|
@@ -386,7 +399,7 @@ Alerts are sent to the maintainer in the following conditions:
|
|
|
386
399
|
|
|
387
400
|
7. **Test request received via API**: When a test request is submitted via the API, an alert is sent with subject `"Kilotest: new test request in the API"` (via the `requestTest` API operation) or `"Kilotest: new retest request in the API"` (via the `requestRetest` API operation), including the target URL, description, and the user's stated reason. A request is treated as a duplicate, and no alert is sent for it, only if a request already pending in `db/jobs/testRequests.json`, i.e., submitted but not yet manually approved by the maintainer into an actual job, for the same URL has the identical description; this check ignores the stated reason and the test-versus-retest distinction, and is not limited to a recent time window, it compares against every request still pending approval for that URL. Once the maintainer approves a pending request (turning it into a job via `/enqueue.html`), all pending requests for that URL are cleared, so this check cannot detect a duplicate against a request that has already been approved, is currently running, or has already completed; an identical request submitted after approval alerts again. The check and its dedup key are identical for the UI and API paths, since both funnel through the same underlying request-queuing function.
|
|
388
401
|
|
|
389
|
-
8. **MCP feature request received**: When a user submits a feature request via the MCP server's `requestFeature` tool, an alert is sent with subject `"MCP feature request received"`, including the text of the feature request.
|
|
402
|
+
8. **MCP feature request received**: When a user submits a feature request via the MCP server's `requestFeature` tool, an alert is sent with subject `"MCP feature request received"`, including the text of the feature request, and the request is also appended to `db/featureRequests.json`, the same way tutorial comments are appended to their own files (see item 1), so that a manager can review or delete it later via `/deleteNotesForm.html`.
|
|
390
403
|
|
|
391
404
|
9. **WAVE balance low**: When Kilotest detects that the account balance for the WAVE API service (used for accessibility testing) has fallen below a threshold, an alert is sent with subject `"Kilotest: WAVE balance low"`, indicating the number of credits remaining.
|
|
392
405
|
|
|
@@ -582,6 +595,14 @@ If restoration becomes necessary, the files can be copied in the opposite direct
|
|
|
582
595
|
|
|
583
596
|
After that the maintainer can inspect the original and restored directories and delete or move report files as needed. After that the `db/reportsExtract.json` file should be deleted, so it will be regenerated when next needed.
|
|
584
597
|
|
|
598
|
+
### Testing target restrictions
|
|
599
|
+
|
|
600
|
+
A page submitted for testing need not be publicly accessible; it need only be reachable from the Testaro worker assigned the job. Without a restriction on which addresses a worker may be sent to, an arbitrary requester could induce a worker to fetch a target it has no legitimate reason to reach, such as [a private or link-local address](https://developer.mozilla.org/en-US/docs/Glossary/Private_network) on the network local to that worker, including a metadata service run by a cloud host. This class of attack is known as [server-side request forgery (SSRF)](https://developer.mozilla.org/en-US/docs/Web/Security/Attacks/SSRF).
|
|
601
|
+
|
|
602
|
+
Kilotest guards against this with the `ALLOW_INTERNAL_TARGETS` environment variable (documented in `env.example`). Left unset, a deployment restricts testing to addresses that resolve to the public address space, checked both when a URL is submitted and again against the actually visited URL of each report before that report is stored (since a redirect could otherwise route a worker to a disallowed target after submission-time validation passed). Setting it to `true` opts a deployment in to testing private, loopback, and link-local addresses, and removes this protection entirely for that Kilotest instance.
|
|
603
|
+
|
|
604
|
+
Setting `ALLOW_INTERNAL_TARGETS` to `true` is not safe merely because every Testaro worker able to poll that Kilotest instance is confined to a private network. A report on an internal page is stored and served by Kilotest itself, so if that Kilotest instance is reachable from the public Internet, anyone able to reach it, not only someone with access to the private network, can read the content of that report, including page text and, depending on job configuration, a screenshot or the HTML source of the page. Setting this variable to `true` is appropriate only for a Kilotest instance that is itself kept off the public Internet, or otherwise restricts who can request and retrieve reports, in addition to confining every one of its Testaro workers to the private network being tested.
|
|
605
|
+
|
|
585
606
|
### Possible future Testaro integration
|
|
586
607
|
|
|
587
608
|
Kilotest uses Testaro to run jobs. In previous versions of Kilotest, Testaro was a dependency. It is currently not a dependency. Instead, Testaro instances are installed on one or more other hosts, and each instance polls Kilotest to ask for jobs to run.
|
package/env.example
CHANGED
|
@@ -15,6 +15,14 @@ DEPLOYED_KILOTEST_HOST=https://kilotest.com
|
|
|
15
15
|
# Path of the data directory (defaults to the db subdirectory of the project root). Tests set
|
|
16
16
|
# this to a fixture directory so they never read or write the real db.
|
|
17
17
|
DB_DIR=__placeholder__
|
|
18
|
+
# Whether this deployment allows testing pages at private, loopback, or link-local addresses
|
|
19
|
+
# (true) or restricts testing to publicly routable addresses (any other value or omission).
|
|
20
|
+
# Set this to true only when every Testaro worker that can poll this Kilotest instance is
|
|
21
|
+
# intentionally deployed on a private network to test that network’s own internal pages;
|
|
22
|
+
# leave it unset for a deployment whose Testaro workers accept jobs from arbitrary requesters,
|
|
23
|
+
# since such a worker could otherwise be induced to fetch a target it has no legitimate reason
|
|
24
|
+
# to reach, such as its own host’s cloud-metadata service.
|
|
25
|
+
ALLOW_INTERNAL_TARGETS=__placeholder__
|
|
18
26
|
|
|
19
27
|
# Alert configuration:
|
|
20
28
|
|