@jrpool/kilotest 51.1.0 → 55.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.devin/config.local.json +4 -1
- package/.github/workflows/ci.yml +5 -2
- package/.github/workflows/periodic-smoke-tests.yml +2 -2
- package/.github/workflows/publish-mcp.yml +1 -1
- package/README.md +5 -5
- package/api/requestFeature.test.ts +36 -0
- package/api/requestFeature.ts +26 -4
- package/api/requestRetest.test.ts +18 -0
- package/api/requestRetest.ts +4 -0
- package/api/requestTest.test.ts +50 -0
- package/api/requestTest.ts +25 -1
- package/capability.md +2 -2
- package/docs/AI-TOOLS.md +3 -3
- package/docs/SERVICE.md +23 -2
- package/env.example +32 -0
- package/index.test.ts +126 -0
- package/index.ts +74 -23
- package/llms-full.txt +1 -1
- package/llms.txt +1 -1
- package/mcp.test.ts +41 -4
- package/mcp.ts +67 -17
- package/openapi.yaml +5 -5
- package/package.json +2 -1
- package/scripts/generate-openapi.ts +1 -1
- package/test/setup.ts +5 -0
- package/util.test.ts +429 -0
- package/util.ts +182 -3
- package/web/requestRetest/index.ts +9 -0
- package/web/requestRetest/requestRetest.test.ts +14 -0
- package/web/requestTest/index.ts +9 -0
- package/web/requestTest/requestTest.test.ts +16 -0
- package/web/tutorialAI/index.ts +25 -3
- package/web/tutorialAI/tutorialAI.test.ts +32 -0
- package/web/tutorialWeb/index.ts +25 -3
- package/web/tutorialWeb/tutorialWeb.test.ts +32 -0
package/.devin/config.local.json
CHANGED
package/.github/workflows/ci.yml
CHANGED
|
@@ -12,16 +12,19 @@ jobs:
|
|
|
12
12
|
|
|
13
13
|
steps:
|
|
14
14
|
- name: Checkout code
|
|
15
|
-
uses: actions/checkout@
|
|
15
|
+
uses: actions/checkout@v5
|
|
16
16
|
|
|
17
17
|
- name: Set Node.js version
|
|
18
|
-
uses: actions/setup-node@
|
|
18
|
+
uses: actions/setup-node@v5
|
|
19
19
|
with:
|
|
20
20
|
node-version: '>=24.16.0'
|
|
21
21
|
|
|
22
22
|
- name: Install dependencies
|
|
23
23
|
run: npm ci
|
|
24
24
|
|
|
25
|
+
- name: Run typecheck
|
|
26
|
+
run: npm run typecheck
|
|
27
|
+
|
|
25
28
|
- name: Run lint
|
|
26
29
|
run: npm run lint
|
|
27
30
|
|
package/README.md
CHANGED
|
@@ -10,7 +10,7 @@ An ensemble testing and reporting service for front-end web quality
|
|
|
10
10
|
|
|
11
11
|
## Features
|
|
12
12
|
|
|
13
|
-
This application uses an ensemble of 12 rule engines to test
|
|
13
|
+
This application uses an ensemble of 12 rule engines to test web pages for front-end quality (i.e. accessibility, usability, and standards conformity).
|
|
14
14
|
|
|
15
15
|
The testing paradigm employed by Kilotest is discussed in these papers:
|
|
16
16
|
|
|
@@ -20,11 +20,11 @@ The testing paradigm employed by Kilotest is discussed in these papers:
|
|
|
20
20
|
|
|
21
21
|
Kilotest acts as a web server for human users, an MCP server for AI platforms, and an API for programmatic access.
|
|
22
22
|
|
|
23
|
-
Testing for Kilotest is performed by one or more testing agents that obtain jobs from Kilotest, perform them, and send reports of the results back to Kilotest. Those agents are instances of the [Testaro](https://www.npmjs.com/package/testaro) package.
|
|
23
|
+
Testing for Kilotest is performed by one or more workers (testing agents) that obtain jobs from Kilotest, perform them, and send reports of the results back to Kilotest. Those agents are instances of the [Testaro](https://www.npmjs.com/package/testaro) package. Thus, the pages that Kilotest tests need not be on the public Internet. They need only be reachable by the Testaro workers that do the testing for Kilotest. So, one or more Testaro workers running inside a private intranet can test internal pages, as well as any external ones reachable from inside the intranet.
|
|
24
24
|
|
|
25
|
-
An active production instance of Kilotest may require multiple
|
|
25
|
+
An active production instance of Kilotest may require multiple Testaro workers to handle the load, because testing one web page typically takes about 3 minutes and workers test only one page at a time.
|
|
26
26
|
|
|
27
|
-
## Getting started locally with 1
|
|
27
|
+
## Getting started locally with 1 worker
|
|
28
28
|
|
|
29
29
|
### Installation
|
|
30
30
|
|
|
@@ -36,7 +36,7 @@ In the steps below, hosts `T` and `K` may be the same host or two different host
|
|
|
36
36
|
1. Clone the Kilotest repository into a new directory on host `K`.
|
|
37
37
|
1. In that directory, install the Kilotest dependencies: `npm install`.
|
|
38
38
|
1. Copy the `env.testaro` file from the `kilotest` directory to `.env` in the `testaro` directory and replace the `__placeholder__` values in `.env` with actual values.
|
|
39
|
-
1. Copy the `env.example` file in the `kilotest` directory to a new `.env` file in the same directory and replace the `__placeholder__` values in `.env` with actual values. `TESTARO_WORKERS` is a JSON object mapping each Testaro worker's ID to an object with its `secret` and its `name`; the ID and secret for a worker must match the `NETWATCH_URL_WORKER_ID` and `NETWATCH_URL_AUTH` values in that worker's `.env`. A worker authenticates each request with an HTTP Basic `Authorization` header (`id:secret`, base64-encoded); as of this writing that header is not yet sent by the published Testaro package, so this scheme requires a matching update to Testaro's `netWatch.js` request code (only to send the header; no change to Testaro's handling of jobs or reports is needed). The ID is a credential and is never published; the `name` is written by Kilotest into a job's and report's `sources.worker` property, which Testaro does not read or alter, so it is a distinct property from `sources.agent` (Testaro's own, untrusted, self-reported label, which Kilotest ignores). Choose a `name` that is safe to appear in every report Kilotest serves.
|
|
39
|
+
1. Copy the `env.example` file in the `kilotest` directory to a new `.env` file in the same directory and replace the `__placeholder__` values in `.env` with actual values. `TESTARO_WORKERS` is a JSON object mapping each Testaro worker's ID to an object with its `secret` and its `name`; the ID and secret for a worker must match the `NETWATCH_URL_WORKER_ID` and `NETWATCH_URL_AUTH` values in that worker's `.env`. A worker authenticates each request with an HTTP Basic `Authorization` header (`id:secret`, base64-encoded); as of this writing that header is not yet sent by the published Testaro package, so this scheme requires a matching update to Testaro's `netWatch.js` request code (only to send the header; no change to Testaro's handling of jobs or reports is needed). The ID is a credential and is never published; the `name` is written by Kilotest into a job's and report's `sources.worker` property, which Testaro does not read or alter, so it is a distinct property from `sources.agent` (Testaro's own, untrusted, self-reported label, which Kilotest ignores). Choose a `name` that is safe to appear in every report Kilotest serves. `ALLOW_INTERNAL_TARGETS` controls whether this deployment can be induced to test pages at private, loopback, or link-local addresses; see [`SERVICE.md`](docs/SERVICE.md#testing-target-restrictions) before changing it from its default.
|
|
40
40
|
|
|
41
41
|
### Usage
|
|
42
42
|
|
|
@@ -81,9 +81,45 @@ test('requestFeature accepts a non-empty feature request and notifies the manage
|
|
|
81
81
|
assert.ok(logged.some(line =>
|
|
82
82
|
line.startsWith('WARNING (Kilotest: MCP feature request received)')
|
|
83
83
|
&& line.includes('Add a dark mode toggle')
|
|
84
|
+
// The alert reports the resulting queue size, so a maintainer who has been away can
|
|
85
|
+
// see at a glance how close the queue is to full.
|
|
86
|
+
&& line.includes('Feature requests now awaiting review: 1 of 20')
|
|
84
87
|
));
|
|
85
88
|
});
|
|
86
89
|
|
|
90
|
+
test('requestFeature rejects a request with a fallback channel when the queue is full', async () => {
|
|
91
|
+
// Fill featureRequests.json with 20 distinct requests, reaching the cap.
|
|
92
|
+
const filler = Array.from({length: 20}, (_, i) => ({
|
|
93
|
+
timeStamp: '260101T0000', content: `Filler feature request number ${i}, long enough to pass the length check.`
|
|
94
|
+
}));
|
|
95
|
+
await fs.writeFile(process.env.FEATURE_REQUESTS_PATH!, JSON.stringify(filler));
|
|
96
|
+
const body = await response(['One feature request too many, long enough to pass the length check.']);
|
|
97
|
+
const details = body['response content']['details about your request'] as any;
|
|
98
|
+
assert.ok(details.error.includes('too many feature requests are awaiting review right now.'));
|
|
99
|
+
assert.ok(details.error.includes('https://github.com/jrpool/kilotest/issues'));
|
|
100
|
+
assert.ok(details.error.includes('info@kilotest.com'));
|
|
101
|
+
const featureRequests = JSON.parse(await fs.readFile(process.env.FEATURE_REQUESTS_PATH!, 'utf8'));
|
|
102
|
+
// The rejected request was not recorded.
|
|
103
|
+
assert.equal(featureRequests.length, 20);
|
|
104
|
+
});
|
|
105
|
+
|
|
106
|
+
test('requestFeature never rejects for a full inbox when FEATURE_REQUESTS_MAX is 0 (no limit)', async () => {
|
|
107
|
+
process.env.FEATURE_REQUESTS_MAX = '0';
|
|
108
|
+
try {
|
|
109
|
+
// Fill featureRequests.json past the default cap.
|
|
110
|
+
const filler = Array.from({length: 25}, (_, i) => ({
|
|
111
|
+
timeStamp: '260101T0000', content: `Filler feature request number ${i}, long enough to pass the length check.`
|
|
112
|
+
}));
|
|
113
|
+
await fs.writeFile(process.env.FEATURE_REQUESTS_PATH!, JSON.stringify(filler));
|
|
114
|
+
const body = await response(['One more feature request, long enough to pass the length check.']);
|
|
115
|
+
const details = body['response content']['details about your request'] as any;
|
|
116
|
+
assert.equal(details.error, undefined);
|
|
117
|
+
}
|
|
118
|
+
finally {
|
|
119
|
+
delete process.env.FEATURE_REQUESTS_MAX;
|
|
120
|
+
}
|
|
121
|
+
});
|
|
122
|
+
|
|
87
123
|
test('requestFeature includes tool name and metadata', async () => {
|
|
88
124
|
const body = await response(['A sufficiently long feature description']);
|
|
89
125
|
assert.equal(body['tool name'], 'requestFeature');
|
package/api/requestFeature.ts
CHANGED
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
import {z} from 'zod';
|
|
9
9
|
import {getResponseMetadata, getThisHost, getToolsFacts} from './util.ts';
|
|
10
10
|
import {sendAlert} from '../alerts.ts';
|
|
11
|
-
import {checkCommentDuplicate, checkLength, getJSON, getNowStamp} from '../util.ts';
|
|
11
|
+
import {checkCommentDuplicate, checkLength, describeMax, getEnvMax, getJSON, getNowStamp} from '../util.ts';
|
|
12
12
|
import {requestFeatureResponseSchema} from './schemas.ts';
|
|
13
13
|
import fs from 'node:fs/promises';
|
|
14
14
|
import path from 'node:path';
|
|
@@ -21,6 +21,13 @@ type ResponseContent = z.infer<typeof requestFeatureResponseSchema>['response co
|
|
|
21
21
|
// FUNCTIONS (helpers)
|
|
22
22
|
|
|
23
23
|
const getFeatureRequestsPath = () => process.env.FEATURE_REQUESTS_PATH || path.join(import.meta.dirname, '../db/featureRequests.json');
|
|
24
|
+
// Maximum number of feature requests held at once. Once reached, a new, distinct request
|
|
25
|
+
// is rejected rather than stored, so the feature-requests file and the manager alert
|
|
26
|
+
// emails it generates cannot grow without bound (modeled on a full mailbox rejecting new
|
|
27
|
+
// mail; see GitHub issue #3, abuse type 3). Configurable via FEATURE_REQUESTS_MAX, since
|
|
28
|
+
// different Kilotest deployments' maintainers may want a different size; a value of 0
|
|
29
|
+
// means no limit.
|
|
30
|
+
const getFeatureRequestsMax = () => getEnvMax('FEATURE_REQUESTS_MAX', 20);
|
|
24
31
|
|
|
25
32
|
// FUNCTIONS
|
|
26
33
|
|
|
@@ -61,7 +68,18 @@ export const response = async (args: string[]) => {
|
|
|
61
68
|
}
|
|
62
69
|
};
|
|
63
70
|
}
|
|
64
|
-
// Otherwise,
|
|
71
|
+
// Otherwise, if the feature requests already on file have reached the cap (a cap of 0
|
|
72
|
+
// means no limit, so the queue is never full):
|
|
73
|
+
else if (getFeatureRequestsMax() > 0 && featureRequests.length >= getFeatureRequestsMax()) {
|
|
74
|
+
responseContent = {
|
|
75
|
+
'details about your request': {
|
|
76
|
+
error: 'request invalid: too many feature requests are awaiting review right now. ' +
|
|
77
|
+
'Please try again later, or post your request at https://github.com/jrpool/kilotest/issues ' +
|
|
78
|
+
'or email info@kilotest.com.'
|
|
79
|
+
}
|
|
80
|
+
};
|
|
81
|
+
}
|
|
82
|
+
// Otherwise, i.e. if it is not a duplicate and the queue is not full:
|
|
65
83
|
else {
|
|
66
84
|
// Record the feature request alongside the existing ones.
|
|
67
85
|
featureRequests.push({
|
|
@@ -70,8 +88,12 @@ export const response = async (args: string[]) => {
|
|
|
70
88
|
});
|
|
71
89
|
await fs.mkdir(path.dirname(featureRequestsPath), {recursive: true});
|
|
72
90
|
await fs.writeFile(featureRequestsPath, getJSON(featureRequests));
|
|
73
|
-
// Notify the manager
|
|
74
|
-
|
|
91
|
+
// Notify the manager, including the resulting count, so a maintainer who has been
|
|
92
|
+
// away sees at a glance how urgently the requests need review.
|
|
93
|
+
await sendAlert(
|
|
94
|
+
'Kilotest: MCP feature request received',
|
|
95
|
+
`${feature}\nFeature requests now awaiting review: ${featureRequests.length} of ${describeMax(getFeatureRequestsMax())}`
|
|
96
|
+
);
|
|
75
97
|
// Add the disposition to the response content.
|
|
76
98
|
responseContent = {
|
|
77
99
|
'details about your request': {
|
|
@@ -173,5 +173,23 @@ test('requestRetest accepts a valid retest request for the latest report of a pa
|
|
|
173
173
|
line.startsWith('WARNING (Kilotest: new retest request awaits approval)')
|
|
174
174
|
&& line.includes('Mixed Outcomes Page')
|
|
175
175
|
&& line.includes('https://example.com/mixed')
|
|
176
|
+
// The alert reports the resulting queue size, so a maintainer who has been away can
|
|
177
|
+
// see at a glance how close the queue is to full.
|
|
178
|
+
&& line.includes('Requests now awaiting approval: 1 of 20')
|
|
176
179
|
));
|
|
177
180
|
});
|
|
181
|
+
|
|
182
|
+
test('requestRetest rejects a request with a fallback channel when the queue is full', async () => {
|
|
183
|
+
// Fill testRequests.json with 20 pending requests, all for one URL, to reach the cap.
|
|
184
|
+
const filler = Array.from({length: 20}, (_, i) => ({
|
|
185
|
+
timeStamp: '260101T0000', description: `Filler Page ${i}`, reason: 'Filler reason.'
|
|
186
|
+
}));
|
|
187
|
+
await fs.writeFile(testRequestsPath, JSON.stringify({'https://example.com/filler': filler}));
|
|
188
|
+
const body = await response(['260101T0003', 'ret', 'A reason that is long enough.']);
|
|
189
|
+
const details = body['response content']['details about your request'] as any;
|
|
190
|
+
assert.equal(details.error, undefined);
|
|
191
|
+
const disposition = body['response content']['disposition of your request'] as any;
|
|
192
|
+
assert.ok(disposition['what happens next'].includes('too many requests are awaiting approval right now.'));
|
|
193
|
+
assert.ok(disposition['what happens next'].includes('https://github.com/jrpool/kilotest/issues'));
|
|
194
|
+
assert.ok(disposition['what happens next'].includes('info@kilotest.com'));
|
|
195
|
+
});
|
package/api/requestRetest.ts
CHANGED
|
@@ -79,6 +79,10 @@ export const response = async (args: string[]) => {
|
|
|
79
79
|
else if (requestResult === 'superseded') {
|
|
80
80
|
failureReason = 'a later report about a page with the same description is available.'
|
|
81
81
|
}
|
|
82
|
+
else if (requestResult === 'queueFull') {
|
|
83
|
+
failureReason = 'too many requests are awaiting approval right now. Please try again later, ' +
|
|
84
|
+
'or post your request at https://github.com/jrpool/kilotest/issues or email info@kilotest.com.'
|
|
85
|
+
}
|
|
82
86
|
else {
|
|
83
87
|
failureReason = 'an identical request is already awaiting approval.'
|
|
84
88
|
}
|
package/api/requestTest.test.ts
CHANGED
|
@@ -17,6 +17,10 @@ import {fixtureDBDir} from '../test/dbFixture.ts';
|
|
|
17
17
|
for (const key of ['MANAGER_EMAIL', 'ALERT_API_HOST', 'ALERT_API_PATH', 'ALERT_API_KEY', 'ALERT_FROM']) {
|
|
18
18
|
process.env[key] = '';
|
|
19
19
|
}
|
|
20
|
+
// Allow internal targets by default, so tests that submit an ordinary https://example.com/...
|
|
21
|
+
// URL do not depend on real DNS/network access to pass the resolution check that
|
|
22
|
+
// isAllowedTarget performs. The test of the check itself (below) overrides this.
|
|
23
|
+
process.env.ALLOW_INTERNAL_TARGETS = 'true';
|
|
20
24
|
|
|
21
25
|
// SETUP AND TEARDOWN
|
|
22
26
|
|
|
@@ -77,6 +81,34 @@ test('requestTest rejects a syntactically invalid URL with the correct length',
|
|
|
77
81
|
assert.ok(!logged.some(line => line.includes('new test request awaits approval')));
|
|
78
82
|
});
|
|
79
83
|
|
|
84
|
+
test('requestTest rejects a private-address URL unless internal targets are allowed', async () => {
|
|
85
|
+
delete process.env.ALLOW_INTERNAL_TARGETS;
|
|
86
|
+
try {
|
|
87
|
+
const body = await response(['Internal Page', 'https://192.168.1.1/page', 'A reason that is long enough.']);
|
|
88
|
+
const details = body['response content']['details about your request'] as any;
|
|
89
|
+
assert.ok(details.error.includes('does not resolve to a page that this deployment can test'));
|
|
90
|
+
assert.ok(!logged.some(line => line.includes('new test request awaits approval')));
|
|
91
|
+
} finally {
|
|
92
|
+
process.env.ALLOW_INTERNAL_TARGETS = 'true';
|
|
93
|
+
}
|
|
94
|
+
});
|
|
95
|
+
|
|
96
|
+
test('requestTest rejects a URL that redirects to a disallowed target, and alerts a manager', async (t) => {
|
|
97
|
+
delete process.env.ALLOW_INTERNAL_TARGETS;
|
|
98
|
+
t.mock.method(globalThis, 'fetch', async () => ({url: 'https://10.0.0.5/page'}) as Response);
|
|
99
|
+
try {
|
|
100
|
+
const body = await response(['Redirecting Page', 'https://example.com/redirector', 'A reason that is long enough.']);
|
|
101
|
+
const details = body['response content']['details about your request'] as any;
|
|
102
|
+
assert.ok(details.error.includes('does not resolve to a page that this deployment can test'));
|
|
103
|
+
assert.ok(logged.some(line =>
|
|
104
|
+
line.includes('WARNING (Kilotest: request rejected for redirecting to a disallowed target)')
|
|
105
|
+
&& line.includes('https://example.com/redirector')
|
|
106
|
+
));
|
|
107
|
+
} finally {
|
|
108
|
+
process.env.ALLOW_INTERNAL_TARGETS = 'true';
|
|
109
|
+
}
|
|
110
|
+
});
|
|
111
|
+
|
|
80
112
|
test('requestTest rejects an already-tested page', async () => {
|
|
81
113
|
const body = await response(['Mixed Outcomes Page', 'https://example.com/mixed', 'A reason that is long enough.']);
|
|
82
114
|
const details = body['response content']['details about your request'] as any;
|
|
@@ -149,6 +181,9 @@ test('requestTest accepts a valid new page request', async () => {
|
|
|
149
181
|
line.startsWith('WARNING (Kilotest: new test request awaits approval)')
|
|
150
182
|
&& line.includes('Brand New Page')
|
|
151
183
|
&& line.includes('https://example.com/brandnew')
|
|
184
|
+
// The alert reports the resulting queue size, so a maintainer who has been away can
|
|
185
|
+
// see at a glance how close the queue is to full.
|
|
186
|
+
&& line.includes('Requests now awaiting approval: 1 of 20')
|
|
152
187
|
));
|
|
153
188
|
});
|
|
154
189
|
|
|
@@ -158,3 +193,18 @@ test('requestTest includes disposition information for a valid request', async (
|
|
|
158
193
|
assert.ok(disposition);
|
|
159
194
|
assert.ok(disposition['what happens next']);
|
|
160
195
|
});
|
|
196
|
+
|
|
197
|
+
test('requestTest rejects a request with a fallback channel when the queue is full', async () => {
|
|
198
|
+
// Fill testRequests.json with 20 pending requests, all for one URL, to reach the cap.
|
|
199
|
+
const filler = Array.from({length: 20}, (_, i) => ({
|
|
200
|
+
timeStamp: '260101T0000', description: `Filler Page ${i}`, reason: 'Filler reason.'
|
|
201
|
+
}));
|
|
202
|
+
await fs.writeFile(testRequestsPath, JSON.stringify({'https://example.com/filler': filler}));
|
|
203
|
+
const body = await response(['One Too Many Page', 'https://example.com/one-too-many', 'A reason that is long enough.']);
|
|
204
|
+
const details = body['response content']['details about your request'] as any;
|
|
205
|
+
assert.equal(details.error, undefined);
|
|
206
|
+
const disposition = body['response content']['disposition of your request'] as any;
|
|
207
|
+
assert.ok(disposition['what happens next'].includes('too many requests are awaiting approval right now.'));
|
|
208
|
+
assert.ok(disposition['what happens next'].includes('https://github.com/jrpool/kilotest/issues'));
|
|
209
|
+
assert.ok(disposition['what happens next'].includes('info@kilotest.com'));
|
|
210
|
+
});
|
package/api/requestTest.ts
CHANGED
|
@@ -7,8 +7,9 @@
|
|
|
7
7
|
|
|
8
8
|
import {z} from 'zod';
|
|
9
9
|
import {getResponseMetadata, getThisHost, getToolsFacts} from './util.ts';
|
|
10
|
-
import {checkLength, isURL, processTestRequest} from '../util.ts';
|
|
10
|
+
import {checkLength, isAllowedRedirectTarget, isAllowedTarget, isURL, processTestRequest} from '../util.ts';
|
|
11
11
|
import {requestTestResponseSchema} from './schemas.ts';
|
|
12
|
+
import {sendAlert} from '../alerts.ts';
|
|
12
13
|
|
|
13
14
|
// TYPES
|
|
14
15
|
|
|
@@ -43,6 +44,25 @@ export const response = async (args: string[]) => {
|
|
|
43
44
|
error: 'request invalid: you specified an invalid URL for the page'
|
|
44
45
|
};
|
|
45
46
|
}
|
|
47
|
+
// Otherwise, i.e. if the URL does not resolve to an allowed target:
|
|
48
|
+
else if (!(await isAllowedTarget(url))) {
|
|
49
|
+
requestDetails = {
|
|
50
|
+
error: 'request invalid: the URL you specified does not resolve to a page that this deployment can test'
|
|
51
|
+
};
|
|
52
|
+
}
|
|
53
|
+
// Otherwise, i.e. if the URL redirects to a target this deployment does not allow:
|
|
54
|
+
else if (!(await isAllowedRedirectTarget(url))) {
|
|
55
|
+
requestDetails = {
|
|
56
|
+
error: 'request invalid: the URL you specified does not resolve to a page that this deployment can test'
|
|
57
|
+
};
|
|
58
|
+
// Alert a manager, since a redirect to a disallowed target on a request that
|
|
59
|
+
// itself named an allowed hostname suggests an attempted or accidental SSRF,
|
|
60
|
+
// rather than an ordinary mistake by a legitimate submitter.
|
|
61
|
+
await sendAlert(
|
|
62
|
+
'Kilotest: request rejected for redirecting to a disallowed target',
|
|
63
|
+
`A request to test ${url} (described as "${description}") was rejected because it redirects to a target this deployment does not allow (e.g. a private, loopback, or link-local address).`
|
|
64
|
+
);
|
|
65
|
+
}
|
|
46
66
|
// Otherwise, i.e. if the description and URL are valid:
|
|
47
67
|
else {
|
|
48
68
|
// Process the request.
|
|
@@ -76,6 +96,10 @@ export const response = async (args: string[]) => {
|
|
|
76
96
|
else if (requestResult === 'retest') {
|
|
77
97
|
failureReason = 'a report about a page with the same description and URL is available.'
|
|
78
98
|
}
|
|
99
|
+
else if (requestResult === 'queueFull') {
|
|
100
|
+
failureReason = 'too many requests are awaiting approval right now. Please try again later, ' +
|
|
101
|
+
'or post your request at https://github.com/jrpool/kilotest/issues or email info@kilotest.com.'
|
|
102
|
+
}
|
|
79
103
|
else {
|
|
80
104
|
failureReason = 'an identical request is already awaiting approval.'
|
|
81
105
|
}
|
package/capability.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
## Capability
|
|
4
4
|
|
|
5
|
-
Web-page front-end quality audit. Kilotest tests
|
|
5
|
+
Web-page front-end quality audit. Kilotest tests web pages for front-end quality, namely accessibility, usability, and standards conformity, and makes structured findings available for retrieval.
|
|
6
6
|
|
|
7
7
|
## Purpose
|
|
8
8
|
|
|
@@ -36,7 +36,7 @@ A list of available reports.
|
|
|
36
36
|
|
|
37
37
|
## Side effects
|
|
38
38
|
|
|
39
|
-
Kilotest reads the specified
|
|
39
|
+
Kilotest reads the specified web page during testing. It does not modify the target site. Testing is asynchronous and may take up to approximately one day before reports are available for newly requested pages.
|
|
40
40
|
|
|
41
41
|
## Cost
|
|
42
42
|
|
package/docs/AI-TOOLS.md
CHANGED
|
@@ -42,7 +42,7 @@ The external features that support the use of Kilotest as a collection of AI too
|
|
|
42
42
|
- A [pull request](https://github.com/public-apis/public-apis/pull/6346/changes) to add Kilotest to the list of public APIs in the `public-apis` repository.
|
|
43
43
|
- An [issue](https://github.com/APIs-guru/openapi-directory/issues/2677) to add Kilotest to `openapi-directory`.
|
|
44
44
|
- A [pull request](https://github.com/w3c/wai-evaluation-tools-list/pull/1153) to add Kilotest to the WAI evaluation tools list.
|
|
45
|
-
- Registration of Kilotest as an active server in the [official MCP Registry](https://registry.modelcontextprotocol.io
|
|
45
|
+
- Registration of Kilotest as an active server in the [official MCP Registry](https://registry.modelcontextprotocol.io) (`io.github.jrpool/kilotest`), maintained by the [Model Context Protocol](https://modelcontextprotocol.io/) project. The [PulseMCP](https://www.pulsemcp.com/servers) directory and the VS Code MCP gallery both crawl this registry, so Kilotest is discoverable through them with no separate submission.
|
|
46
46
|
- Registration of Kilotest with the [Smithery](https://smithery.ai/servers/pool/kilotest) MCP server registry.
|
|
47
47
|
- Registration of Kilotest with the [Glama](https://glama.ai/mcp/connectors/com.kilotest/kilotest) MCP server registry, including a claim of ownership of the MCP connector by the maintainer via a `TXT` DNS record for `_glama-claim.kilotest.com`, with the value `glama_claim_gVV4HrPdcXr9TEfj1p3icEUBJ3_OVJGX`.
|
|
48
48
|
- Registration of Kilotest with [Awesome MCP Servers](https://www.tensorblock.co/mcp/servers/github-jrpool-kilotest-f518ba36).
|
|
@@ -107,7 +107,7 @@ Other future work:
|
|
|
107
107
|
Capability: web-page-quality-audit
|
|
108
108
|
|
|
109
109
|
Purpose:
|
|
110
|
-
Test a
|
|
110
|
+
Test a web page for accessibility,
|
|
111
111
|
usability, HTML and web-standards problems.
|
|
112
112
|
|
|
113
113
|
Interfaces:
|
|
@@ -123,7 +123,7 @@ Other future work:
|
|
|
123
123
|
rule-engine diagnoses, WCAG mappings
|
|
124
124
|
|
|
125
125
|
Side effects:
|
|
126
|
-
Reads the specified
|
|
126
|
+
Reads the specified web page.
|
|
127
127
|
Does not modify the target site.
|
|
128
128
|
|
|
129
129
|
Cost:
|
package/docs/SERVICE.md
CHANGED
|
@@ -270,7 +270,7 @@ kilotest.com {
|
|
|
270
270
|
}
|
|
271
271
|
@allowedPOST {
|
|
272
272
|
method POST
|
|
273
|
-
path /mcp /requestRetest.html/* /requestTest.html /requestAction.html /reannotate.html /renewWCAG.html /ai0BalanceForm.html /expungeReportsForm.html /hideReportForm.html /metrics.html /pruneReportsForm.html /rewindReportsForm.html /showHiddenReportsForm.html /unhideReportForm.html /api/* /tutorialWebComment.html /tutorialAIComment.html /worker/job /worker/report
|
|
273
|
+
path /mcp /requestRetest.html/* /requestTest.html /requestAction.html /reannotate.html /renewWCAG.html /ai0BalanceForm.html /deleteNotesForm.html /expungeReportsForm.html /hideReportForm.html /metrics.html /pruneReportsForm.html /rewindReportsForm.html /showHiddenReportsForm.html /unhideReportForm.html /api/* /tutorialWebComment.html /tutorialAIComment.html /worker/job /worker/report
|
|
274
274
|
}
|
|
275
275
|
# Respond to OPTIONS requests.
|
|
276
276
|
@allowedOPTIONS method OPTIONS
|
|
@@ -417,6 +417,19 @@ Alerts are sent via the `sendAlert()` function in [alerts.ts](../alerts.ts), whi
|
|
|
417
417
|
|
|
418
418
|
If any of these five variables is missing or empty at alert time, the alert is not sent. Instead, a warning message is logged to the console, and the application continues normal operation. The warning includes the alert subject and body so the maintainer can manually review what would have been sent.
|
|
419
419
|
|
|
420
|
+
### Inbox size limits
|
|
421
|
+
|
|
422
|
+
Four submission types (test requests and retest requests together, web tutorial comments, AI tutorial comments, and MCP feature requests) are each held in a pending "inbox" (`db/jobs/testRequests.json`, `db/comments/tutorialWeb.json`, `db/comments/tutorialAI.json`, and `db/featureRequests.json` respectively) awaiting manual review by the maintainer. Each inbox has a maximum size; once reached, a new, distinct submission is rejected (with a message suggesting the requester instead post to the [GitHub issues page](https://github.com/jrpool/kilotest/issues) or email `info@kilotest.com`) rather than added, so neither the inbox file nor the manager alert emails it generates can grow without bound (see GitHub issue #3, abuse type 3, repeated/automated submissions). This is modeled on a full mailbox rejecting new mail, rather than on a submission rate limit: it is the backlog size itself, not how quickly it grows, that costs the maintainer review time and risks unbounded storage. Every alert sent for an accepted submission reports the resulting inbox count against its cap, so a maintainer returning after an absence can immediately see how close an inbox is to full.
|
|
423
|
+
|
|
424
|
+
Each cap defaults to 20 and can be set independently via an environment variable:
|
|
425
|
+
|
|
426
|
+
1. **`TEST_REQUEST_QUEUE_MAX`**: The maximum number of test and retest requests awaiting approval at once, summed across all requested URLs.
|
|
427
|
+
2. **`TUTORIAL_WEB_COMMENTS_MAX`**: The maximum number of comments held on the web tutorial page (`/tutorialWeb.html`).
|
|
428
|
+
3. **`TUTORIAL_AI_COMMENTS_MAX`**: The maximum number of comments held on the AI agent tutorial page (`/tutorialAI.html`).
|
|
429
|
+
4. **`FEATURE_REQUESTS_MAX`**: The maximum number of feature requests held via the MCP server's `requestFeature` tool.
|
|
430
|
+
|
|
431
|
+
Each variable, if set, must be a non-negative integer; if unset, empty, or not a non-negative integer, the default of 20 is used instead. A value of `0` means no limit: that inbox is never treated as full, regardless of size. Unlike the alert-configuration variables above, these are not required: an unset value never disables the limit, it only selects the default size.
|
|
432
|
+
|
|
420
433
|
### Resend integration
|
|
421
434
|
|
|
422
435
|
Kilotest uses [Resend](https://resend.com/) as its transactional email delivery service. Resend is configured through the environment variables above.
|
|
@@ -489,7 +502,7 @@ Kilotest records basic counts of how it is used, so the maintainer can answer qu
|
|
|
489
502
|
|
|
490
503
|
- The generic `.html` GET dispatch in `index.ts`, once per successfully served web page, recorded under the `pageViews` category by page name (for example `tutorialWeb`, `listReports`), unless the page is a manager-only page (see "Manager page activity" below), in which case it is recorded there instead.
|
|
491
504
|
- The home page (`/` and `/index.html`), served by its own branch in `index.ts` rather than through the generic `.html` dispatch, recorded under `pageViews` as `index`.
|
|
492
|
-
- Each of the
|
|
505
|
+
- Each of the 9 MCP tool handlers in `mcp.ts`, once per successful tool call, recorded under the `mcpToolCalls` category by tool name (for example `listReports`, `requestTest`).
|
|
493
506
|
- The `/api/*` GET and POST service branches in `index.ts`, once per call, recorded under the `apiOperations` category by operation name.
|
|
494
507
|
|
|
495
508
|
`db/metrics.json` also stores a `since` time stamp, set when the file is first created, so the counts can be read as "since this date" rather than assumed to cover Kilotest's entire history. `getMetrics()` backfills any category absent from an existing `db/metrics.json` (for example one written before a category such as `managerActivity` existed), so an older file remains readable rather than causing every subsequent request to error.
|
|
@@ -595,6 +608,14 @@ If restoration becomes necessary, the files can be copied in the opposite direct
|
|
|
595
608
|
|
|
596
609
|
After that the maintainer can inspect the original and restored directories and delete or move report files as needed. After that the `db/reportsExtract.json` file should be deleted, so it will be regenerated when next needed.
|
|
597
610
|
|
|
611
|
+
### Testing target restrictions
|
|
612
|
+
|
|
613
|
+
A page submitted for testing need not be publicly accessible; it need only be reachable from the Testaro worker assigned the job. Without a restriction on which addresses a worker may be sent to, an arbitrary requester could induce a worker to fetch a target it has no legitimate reason to reach, such as [a private or link-local address](https://developer.mozilla.org/en-US/docs/Glossary/Private_network) on the network local to that worker, including a metadata service run by a cloud host. This class of attack is known as [server-side request forgery (SSRF)](https://developer.mozilla.org/en-US/docs/Web/Security/Attacks/SSRF).
|
|
614
|
+
|
|
615
|
+
Kilotest guards against this with the `ALLOW_INTERNAL_TARGETS` environment variable (documented in `env.example`). Left unset, a deployment restricts testing to addresses that resolve to the public address space, checked both when a URL is submitted and again against the actually visited URL of each report before that report is stored (since a redirect could otherwise route a worker to a disallowed target after submission-time validation passed). Setting it to `true` opts a deployment in to testing private, loopback, and link-local addresses, and removes this protection entirely for that Kilotest instance.
|
|
616
|
+
|
|
617
|
+
Setting `ALLOW_INTERNAL_TARGETS` to `true` is not safe merely because every Testaro worker able to poll that Kilotest instance is confined to a private network. A report on an internal page is stored and served by Kilotest itself, so if that Kilotest instance is reachable from the public Internet, anyone able to reach it, not only someone with access to the private network, can read the content of that report, including page text and, depending on job configuration, a screenshot or the HTML source of the page. Setting this variable to `true` is appropriate only for a Kilotest instance that is itself kept off the public Internet, or otherwise restricts who can request and retrieve reports, in addition to confining every one of its Testaro workers to the private network being tested.
|
|
618
|
+
|
|
598
619
|
### Possible future Testaro integration
|
|
599
620
|
|
|
600
621
|
Kilotest uses Testaro to run jobs. In previous versions of Kilotest, Testaro was a dependency. It is currently not a dependency. Instead, Testaro instances are installed on one or more other hosts, and each instance polls Kilotest to ask for jobs to run.
|
package/env.example
CHANGED
|
@@ -2,6 +2,12 @@
|
|
|
2
2
|
ENVIRONMENT=__placeholder__
|
|
3
3
|
# http (local or reverse-proxied server) or https.
|
|
4
4
|
PROTOCOL=__placeholder__
|
|
5
|
+
# Port the server listens on (defaults to 3000).
|
|
6
|
+
PORT=__placeholder__
|
|
7
|
+
# Path of the private key file. Required only when PROTOCOL is https.
|
|
8
|
+
KEY=__placeholder__
|
|
9
|
+
# Path of the certificate file. Required only when PROTOCOL is https.
|
|
10
|
+
CERT=__placeholder__
|
|
5
11
|
# 3-character password for manager operations.
|
|
6
12
|
AUTH_CODE=__placeholder__
|
|
7
13
|
# ID of each Testaro worker and, for each, its secret and published name.
|
|
@@ -15,6 +21,32 @@ DEPLOYED_KILOTEST_HOST=https://kilotest.com
|
|
|
15
21
|
# Path of the data directory (defaults to the db subdirectory of the project root). Tests set
|
|
16
22
|
# this to a fixture directory so they never read or write the real db.
|
|
17
23
|
DB_DIR=__placeholder__
|
|
24
|
+
# Whether this deployment allows testing pages at private, loopback, or link-local addresses
|
|
25
|
+
# (true) or restricts testing to publicly routable addresses (any other value or omission).
|
|
26
|
+
# Set this to true only when every Testaro worker that can poll this Kilotest instance is
|
|
27
|
+
# intentionally deployed on a private network to test that network’s own internal pages;
|
|
28
|
+
# leave it unset for a deployment whose Testaro workers accept jobs from arbitrary requesters,
|
|
29
|
+
# since such a worker could otherwise be induced to fetch a target it has no legitimate reason
|
|
30
|
+
# to reach, such as its own host’s cloud-metadata service.
|
|
31
|
+
ALLOW_INTERNAL_TARGETS=__placeholder__
|
|
32
|
+
# Path of the web tutorial comments file (defaults to db/comments/tutorialWeb.json).
|
|
33
|
+
TUTORIAL_WEB_COMMENTS_PATH=__placeholder__
|
|
34
|
+
# Path of the AI tutorial comments file (defaults to db/comments/tutorialAI.json).
|
|
35
|
+
TUTORIAL_AI_COMMENTS_PATH=__placeholder__
|
|
36
|
+
# Path of the feature requests file (defaults to db/featureRequests.json).
|
|
37
|
+
FEATURE_REQUESTS_PATH=__placeholder__
|
|
38
|
+
|
|
39
|
+
# Inbox size limits: each defaults to 20 if unset, empty, or not a non-negative integer.
|
|
40
|
+
# A value of 0 means no limit.
|
|
41
|
+
|
|
42
|
+
# Maximum number of test/retest requests awaiting approval at once, across all URLs.
|
|
43
|
+
TEST_REQUEST_QUEUE_MAX=__placeholder__
|
|
44
|
+
# Maximum number of comments held on the web tutorial page.
|
|
45
|
+
TUTORIAL_WEB_COMMENTS_MAX=__placeholder__
|
|
46
|
+
# Maximum number of comments held on the AI agent tutorial page.
|
|
47
|
+
TUTORIAL_AI_COMMENTS_MAX=__placeholder__
|
|
48
|
+
# Maximum number of feature requests held via the MCP server's requestFeature tool.
|
|
49
|
+
FEATURE_REQUESTS_MAX=__placeholder__
|
|
18
50
|
|
|
19
51
|
# Alert configuration:
|
|
20
52
|
|