@jrpool/kilotest 31.2.2 → 33.0.0

package/llms-full.txt

@@ -2,19 +2,34 @@
 
 ## What Kilotest is
 
-Kilotest is an application that performs ensemble testing of web pages for accessibility, usability, and standard conformance and reports the test results. For brevity, hereafter in this document those three attributes are referred to as “front-end quality”.
+Kilotest is an application that performs ensemble testing of web pages for front-end quality (i.e. accessibility, usability, and standard conformance) and reports the test results.
 
-## What Kilotest does for AI agents
+## What Kilotest does for language models
 
-Kilotest is deployed as a service with a public URL. Kilotest can test only web pages that can be accessed from the public Internet and that are not protected by authentication or other access controls. Kilotest has not implemented any mechanism for testing private, internal, password-protected, or otherwise restricted pages.
+Kilotest is deployed on the public Internet as a service at `kilotest.com`. Kilotest can test only web pages that can be accessed from the public Internet and that are not protected by authentication or other access controls. Kilotest has not implemented any mechanism for testing private, internal, password-protected, or otherwise restricted pages.
 
-An LLM cannot produce for an AI agent thorough, accurate, and inexpensive information about the front-end quality of a web page. To get such information, the agent needs to select and use specialized tools. However, such selection and use, too, require specialized skills that LLMs lack. Kilotest assumes responsibility for these functions. Kilotest:
+An LLM can, under some conditions, load web pages and use its knowledge to assess successes and failures of front-end quality. But such an assessment is almost always:
 
-- selects, as tools, an ensemble of 10 rule engines that implement tests for, in total, more than a thousand rules for front-end quality
+- expensive, substituting inference for deterministic rule application
+- fragmentary, because of capability limitations, such as inability to run browsers and interact with web pages
+- inaccurate, because of the tendency to replace missing information with hallucinations
+
+To provide a more inexpensive, thorough, and accurate assessment, the agent could ask an ensemble of specialized rule engines to test the page. However, selecting and running such rule engines and consolidating their results is itself expensive and difficult. Kilotest assumes responsibility for these functions. Kilotest:
+
+- defines a tool that tests web pages for front-end quality
+- defines tools that report the test results
+- defines connectors to those tools, compatible with major AI platforms
+
+The tool that tests web pages for front-end quality has attributes that no other such tool has. It:
+
+- selects an ensemble of 10 rule engines that implement tests for, in total, more than a thousand rules for front-end quality
 - runs the tests of the rule engines
 - combines the reports of the rule engines into a single integrated report
 - consolidates the 1000+ rules having 10 different naming systems into 300+ “issues” with a uniform naming system
-- extracts from the report statistics and details at the level of granularity required by an agent to fulfill any particular request
+
+The tools that report test results extract from reports statistics and details at various levels of granularity, as required by an agent to fulfill particular requests.
+
+You can infer from this description that the Kilotest tools are (currently) best suited for help in answering comprehensive audit-type requests (e.g., “How can I improve this page?”) and, in the case of web pages for which reports are already available, more specific questions (e.g., “Is the heading structure of this page OK?”), but, if no report is available for a page, recommending that Kilotest test the page is overkill when the purpose is to answer a specific one-issue question.
 
 ## How Kilotest integrates methods
 
@@ -25,17 +40,13 @@ Testing for front-end quality requires a mixture of deterministic and inferentia
 
 The rule engines in the Kilotest ensemble take responsibility for determining whether each rule requires inferential testing and, if so, obtaining inference from an appropriate model.
 
-## Kilotest versus pure inference
-
-LLM platforms asked to evaluate front-end quality with no delegation are not all able to load pages of web applications for inspection. When they can do so, they typically cannot operate on the pages with browser automation, as Kilotest does. As a result, LLMs typically give less comprehensive answers than they would if they called Kilotest as a tool. When even browsing to a page is impossible, an LLM is likely to give speculative evaluations with hallucinated examples, based on statistical frequency, with no mention of issues that are rarely tested for, instead of providing the truthful and comprehensive assessments that users require.
+## Kilotest versus other MCP servers
 
-## Kilotest versus other tools
+In the marketplace for MCP servers that test for front-end quality, all servers other than that of Kilotest leverage at most 1 rule engine, most often `axe-core`. The Kilotest testing tool leverages `axe-core` plus 9 other rule engines. This has these main effects:
 
-In the marketplace for MCP servers that test for front-end quality, all tools other than Kilotest leverage at most 1 rule engine, most often `axe-core`. Kilotest leverages `axe-core` plus 9 other rule engines. This has these main effects:
-
-- For pages that have not yet been tested, a single-tool MCP can provide faster results than Kilotest. A tool running only the `axe-core` tests can complete its work in about 5 seconds or less. Kilotest usually completes its work in 2 to 3 minutes, because some of the tests involve navigation and interaction with the page and LLM inference. Moreover, the Kilotest API feature suite is currently in an alpha phase, allowing agents, like human users, to **recommend** new pages for testing but allowing only managers to act on such recommendations. The wait for manager action can take up to a day. A feature permitting immediate testing ordered by AI agents is planned, but, until it is implemented, Kilotest will be useful for not-yet-tested pages only in long-running workflows.
-- For pages that have already been tested by Kilotest, Kilotest can provide faster results than a single-tool MCP server, because Kilotest stores test results for subsequent retrieval. A retrieval from Kilotest can be completed in less than 2 seconds.
-- Kilotest results are more comprehensive than single-tool MCP server results. Every rule engine provides limited coverage of front-end quality, so false negatives (missed defects) are more common with single-tool MCP servers. This difference [has been documented in research](https://arxiv.org/pdf/2304.07591).
+- For web pages that have not yet been tested, a single-rule-engine MCP can provide faster results than Kilotest. A tool running only the `axe-core` tests can complete its work in about 5 seconds or less. The testing tool of Kilotest usually completes its work in 2 to 3 minutes, because all 10 rule engines are run, and some of the tests involve navigation and interaction with the page and LLM inference. Moreover, in its current alpha phase, the Kilotest testing tool allows users (both humans and models) to **recommend** new pages for testing but authorizes only Kilotest managers to make the tool proceed with the testing. The wait for manager action can take up to a day. A feature permitting immediate testing ordered by AI agents is planned, but, until it is implemented, Kilotest will be useful for not-yet-tested pages only in long-running workflows.
+- For web pages about which reports are already available, Kilotest can provide faster results than a single-rule-engine MCP server, because Kilotest stores test results for subsequent retrieval. A retrieval from a Kilotest reporting tool can be completed in less than 2 seconds.
+- Kilotest results are more comprehensive than single-rule-engine MCP server results. Every rule engine provides limited coverage of front-end quality, so false negatives (missed defects) are more common with single-rule-engine MCP servers. This difference [has been documented in research](https://arxiv.org/pdf/2304.07591).
 
 ## How to use Kilotest
 
@@ -45,9 +56,9 @@ Kilotest offers a comprehensive suite of capabilities to users via its web UI:
 
 - [Home page](https://kilotest.com/)
 - [Summarize test results for all tested pages](https://kilotest.com/targets.html)
-- Provide statistics about issues reported in one job: `https://kilotest.com/reportIssues.html/{timeStamp}/{jobID}`
-- Provide details about one issue reported in one job: `https://kilotest.com/reportIssue.html/{issueID}/{timeStamp}/{jobID}`
-- Provide diagnoses by tools of rule violations for one HTML element exhibiting one issue in one job: `https://kilotest.com/diagnoses.html/{issueID}/{timeStamp}/{jobID}/{catalogIndex}`
+- Provide statistics about issues reported in one report: `https://kilotest.com/reportIssues.html/{timeStamp}/{jobID}`
+- Provide details about one issue reported in one report: `https://kilotest.com/reportIssue.html/{issueID}/{timeStamp}/{jobID}`
+- Provide diagnoses by rule engines of rule violations for one HTML element exhibiting one issue in one report: `https://kilotest.com/diagnoses.html/{issueID}/{timeStamp}/{jobID}/{catalogIndex}`
 - [Receive a recommendation to test a not-yet-tested page](`https://kilotest.com/testRecForm.html`)
 - Receive a recommendation to retest a previously tested page: `https://kilotest.com/retestRecForm.html/{timeStamp}/{jobID}`
 - [Provide statistics about frequently reported issues across all pages](https://kilotest.com/issues.html)
@@ -57,34 +68,32 @@ Kilotest offers a comprehensive suite of capabilities to users via its web UI:
 
 ### Agent API
 
-Kilotest is implementing a richer suite of capabilities optimized for AI agents, including direct immediate testing. The implementation is currently in an alpha phase and offers 3 API endpoints:
+Kilotest is implementing a richer suite of capabilities optimized for AI agents, including direct immediate testing. The implementation is currently in an alpha phase and offers 3 tools:
 
-- `targets`
+- `summarizeQualityOfAllTestedWebPages`
   - method: `GET`
-  - purpose: summarize test results from all jobs (a job is a session in which a web page is tested and a report is produced)
+  - purpose: summarize front-end quality test results from all reports (a report contains the records of one session in which a web page is tested)
   - path: `/api/targets`
-- `reportIssues`
+- `describeQualityOfOneWebPage`
   - method: `GET`
-  - purpose: provide statistics about issues reported in one job report
+  - purpose: provide statistics about front-end quality issues reported in one report
   - path: `/api/reportIssues/{timeStamp}/{jobID}`
   - parameters
-    - `timeStamp`: initial segment of job identifier
-    - `jobID`: final segment of job identifier
+    - `timeStamp`: initial segment of report identifier
+    - `jobID`: final segment of report identifier
   - source of parameters: response to a `targets` request
-- `testRecForm`
+- `recommendQualityTestingOfOneWebPage`
   - method: `POST`
-  - purpose: receive a recommendation to test a not-yet-tested page
+  - purpose: recommend quality testing of one web page
   - path: `/api/testRecForm`
   - payload properties
     - `what`: description of the page to be tested
     - `url`: URL of the page to be tested
     - `why`: reason for testing the page
-  - how to verify disposition of the recommendation: submit a `targets` request and inspect the response to determine whether a report on the page is now available
+  - how to verify disposition of the recommendation: submit a `summarizeQualityOfAllTestedWebPages` request and inspect the response to determine whether a report on the page is now available
 
 An [OpenAPI specification for Kilotest](https://kilotest.com/openapi.yaml) is available.
 
-Until direct immediate testing is available, an agent can recommend testing of a web page with a `testRecForm` request.
-
 ### More information
 
 More information about Kilotest features and internals:

package/mcp.js

@@ -0,0 +1,96 @@
+/*
+  mcp.js
+  Handles MCP (Model Context Protocol) requests for Kilotest tools.
+*/
+
+// IMPORTS
+
+const {McpServer} = require('@modelcontextprotocol/sdk/server/mcp.js');
+const {StreamableHTTPServerTransport} = require('@modelcontextprotocol/sdk/server/streamableHttp.js');
+const {z} = require('zod');
+const {isReportAvailable, isURL} = require('./util');
+const targetsAPI = require('./targets/api');
+const reportIssuesAPI = require('./reportIssues/api');
+const testRecFormAPI = require('./testRecForm/api');
+
+// FUNCTIONS
+
+// Creates and returns an McpServer with Kilotest tools registered.
+const createMCPServer = () => {
+  const server = new McpServer({name: 'Kilotest', version: '1.0.0'});
+  server.registerTool(
+    'summarizeQualityOfAllTestedWebPages',
+    {
+      description: 'Returns summary data from every available Kilotest report about the front-end quality (i.e. accessibility, usability, and standard conformity) of a web page. Before calling describeQualityIssuesOfOneWebPage, call this tool to check whether a report about the page is available.',
+      inputSchema: {},
+      annotations: {
+        title: 'Summarize quality of all tested web pages',
+        readOnlyHint: true,
+        idempotentHint: true,
+        destructiveHint: false,
+        openWorldHint: false
+      }
+    },
+    async () => {
+      const result = await targetsAPI.response();
+      return {content: [{type: 'text', text: JSON.stringify(result)}]};
+    }
+  );
+  server.registerTool(
+    'describeQualityOfOneWebPage',
+    {
+      description: 'Returns data from a specified Kilotest report about issues of front-end quality (i.e. accessibility, usability, and standard conformity) of a web page. The required timeStamp and jobID parameters identify the report and are obtained from a summarizeQualityOfAllTestedWebPages response.',
+      inputSchema: {
+        timeStamp: z.string().describe('Report timestamp in YYMMDDTHHMM format, e.g. 260503T0432'),
+        jobID: z.string().describe('Job identifier, e.g. x9z')
+      },
+      annotations: {
+        title: 'Describe the quality of one web page',
+        readOnlyHint: true,
+        idempotentHint: true,
+        destructiveHint: false,
+        openWorldHint: false
+      }
+    },
+    async ({timeStamp, jobID}) => {
+      const result = await reportIssuesAPI.response([timeStamp, jobID]);
+      return {content: [{type: 'text', text: JSON.stringify(result)}]};
+    }
+  );
+  server.registerTool(
+    'recommendQualityTestingOfOneWebPage',
+    {
+      description: 'Recommends a web page for Kilotest to test for front-end quality (i.e. accessibility, usability, and standard conformity). Do not call this tool unless summarizeQualityOfAllTestedWebPages discloses that no report about the page or a related page that satisfies your requirements is available.',
+      inputSchema: {
+        what: z.string().describe('Short description of the page, following the naming conventions visible in the summarizeQualityOfAllTestedWebPages response'),
+        url: z.string().describe('Full HTTPS URL of the page to test'),
+        why: z.string().describe('Reason for recommending this page for testing')
+      },
+      annotations: {
+        title: 'Recommend quality testing of one web page',
+        readOnlyHint: false,
+        idempotentHint: false,
+        destructiveHint: false,
+        openWorldHint: false
+      }
+    },
+    async ({what, url, why}) => {
+      if (!isURL(url)) {
+        return {content: [{type: 'text', text: JSON.stringify({error: 'Invalid URL'})}], isError: true};
+      }
+      if (await isReportAvailable(what, url)) {
+        return {content: [{type: 'text', text: JSON.stringify({error: 'A report about the page is already available'})}], isError: true};
+      }
+      const result = await testRecFormAPI.response(what, url, why);
+      return {content: [{type: 'text', text: JSON.stringify(result)}]};
+    }
+  );
+  return server;
+};
+// Handles an MCP request.
+exports.handleMCP = async (request, response) => {
+  const transport = new StreamableHTTPServerTransport({sessionIdGenerator: undefined});
+  const server = createMCPServer();
+  await server.connect(transport);
+  await transport.handleRequest(request, response);
+};

package/openapi.yaml

@@ -1,7 +1,7 @@
 openapi: 3.1.0
 info:
   title: Kilotest Agent API
-  description: Kilotest tests web pages for accessibility, usability, and standard conformity using an ensemble of ten independent tools that employ rule-based and machine-learning-based methods. This API enables AI agents to recommend web pages for testing, discover available test reports, and retrieve data from reports. For background on Kilotest and the advantages of ensemble testing, visit https://kilotest.com.
+  description: Kilotest tests web pages for front-end quality (i.e. accessibility, usability, and standard conformity) using an ensemble of ten independent rule engines that employ rule-based and machine-learning-based methods. This API enables AI agents to recommend web pages for testing, discover available test reports, and retrieve data from reports. For background on Kilotest and the advantages of ensemble testing, visit https://kilotest.com.
   version: 1.0.0
   contact:
     name: Kilotest
@@ -15,9 +15,9 @@ servers:
 paths:
   /api/targets:
     get:
-      operationId: summarizeAccessibilityOfAllTestedWebPages
-      summary: Summarizes all available reports
-      description: Returns summary data about every non-hidden report available from Kilotest, including the name and URL of the tested web page, when the testing was performed, how many accessibility, usability, and standard-conformity issues were reported, and URLs for retrieving more detailed data from the report. This is the first endpoint to call if you want data about a particular web page. The result will tell you whether a report on that page already exists. If so, you can retrieve data from it. If not, you can use the submitWebAccessibilityTestRequest endpoint to recommend the page for testing.
+      operationId: summarizeQualityOfAllTestedWebPages
+      summary: Summarize quality of all tested web pages
+      description: Returns summary data about every non-hidden report available from Kilotest, including the name and URL of the tested web page, when the testing was performed, how many front-end quality (i.e. accessibility, usability, and standard-conformity) issues were reported, and URLs for retrieving more detailed data from the report. This is the first endpoint to call if you want data about a particular web page. The result will tell you whether a report on that page already exists. If so, you can retrieve data from it. If not, you can use the submitWebAccessibilityTestRequest endpoint to recommend the page for testing.
       responses:
         '200':
           description: Summaries of available reports
@@ -28,9 +28,9 @@ paths:
 
   /api/testRecForm:
     post:
-      operationId: submitWebAccessibilityTestRequest
-      summary: Receives a new testing recommendation
-      description: Receives a recommendation for Kilotest to test, for the first time, a particular web page for accessibility, usability, and standard conformity. Recommendations are typically approved and the testing completed within a day, whereupon the results can be found with the summarizeAccessibilityOfAllTestedWebPages operation. Before submitting a recommendation, use the summarizeAccessibilityOfAllTestedWebPages operation to ensure that the page has not yet been tested, and also to see the stylistic rules for the naming of pages. An attempt to recommend an already tested page for testing will fail.
+      operationId: recommendQualityTestingOfOneWebPage
+      summary: Recommend quality testing of one web page
+      description: Submit a recommendation for Kilotest to test, for the first time, a particular web page for front-end quality (i.e. accessibility, usability, and standard conformity). Recommendations are typically approved and the testing completed within a day, whereupon the results can be found with the summarizeQualityOfAllTestedWebPages operation. Before submitting a recommendation, use the summarizeQualityOfAllTestedWebPages operation to ensure that no report about the page is available, and also to see the stylistic rules for the naming of pages. An attempt to recommend a page for testing will fail if a report about the page is already available.
       requestBody:
         description: Test recommendation specifications
         required: true
@@ -48,9 +48,9 @@ paths:
 
   /api/reportIssues/{timeStamp}/{jobID}:
     get:
-      operationId: listAccessibilityIssuesOnOneWebPage
-      summary: Gets data on issues from a specific report
-      description: Returns data about the issues reported in a specific Kilotest report, grouped by priority. The data on each issue include the tools that reported it, the number of HTML elements exhibiting it, and URLs for retrieving element-level detail. The timeStamp and jobID components identify the report and are available in the response from the summarizeAccessibilityOfAllTestedWebPages operation.
+      operationId: describeQualityOfOneWebPage
+      summary: Describe quality of one web page
+      description: Get data about the quality of a specified Kilotest report. The data about each issue include its priority, the rule engines that reported it, the number of HTML elements exhibiting it, and URLs for retrieving element-level detail. The timeStamp and jobID parameters identify the report and are available in the response from the summarizeQualityOfAllTestedWebPages operation.
       parameters:
         - name: timeStamp
           in: path
@@ -81,14 +81,14 @@ paths:
 
   /api/reportIssue/{issueID}/{timeStamp}/{jobID}:
     get:
-      operationId: listHTMLElementsHavingOneAccessibilityIssue
-      summary: Gets details about a specific issue in a specific report (NOT YET IMPLEMENTED)
-      description: Returns details about a single issue within a specific report, including which HTML elements exhibit the issue and, for each such element, URLs for retrieving tool-by-tool diagnoses of the issue on the element. NOT YET IMPLEMENTED.
+      operationId: describeHTMLElementsHavingOneQualityIssue
+      summary: Describe HTML elements having one quality issue (NOT YET IMPLEMENTED)
+      description: Get issue-specific data from a specified report about the front-end quality (i.e. accessibility, usability, and standard conformity) of a web page. The data describe the issue, all of the HTML elements of the page that have the issue, and, for each such element, URLs for retrieving diagnoses by rule engines of the issue on the element. NOT YET IMPLEMENTED.
       parameters:
         - name: issueID
           in: path
           required: true
-          description: Issue identifier (e.g., imageNoText). Available under "issues reported" > priority level > "identifier" in the listAccessibilityIssuesOnOneWebPage response.
+          description: Issue identifier (e.g., imageNoText). Available under "issues reported" > priority level > "identifier" in the describeQualityOfOneWebPage response.
           schema:
             type: string
             examples:
@@ -191,29 +191,29 @@ components:
     TestRecFormResponse:
       $ref: '#/components/schemas/CommonResponseFields'
 
-    ToolInfo:
+    RuleEngineInfo:
       type: object
-      description: An accessibility testing tool in the Kilotest ensemble.
+      description: A rule engine in the Kilotest ensemble.
       properties:
         identifier:
           type: string
-          description: Short programmatic identifier for the tool.
+          description: Short programmatic identifier for the rule engine.
           examples:
             - alfa
         name:
           type: string
-          description: Display name of the tool.
+          description: Display name of the rule engine.
           examples:
             - Alfa
         sponsor:
           type: string
-          description: Organization that created, initially sponsored, or now sponsors the tool.
+          description: Organization that created, initially sponsored, or now sponsors the rule engine.
           examples:
             - Siteimprove
 
-    ToolFailure:
+    RuleEngineFailure:
       type: object
-      description: A tool that was unable to complete testing of the page.
+      description: A rule engine that was unable to complete testing of the page.
       properties:
         name:
           type: string
@@ -224,9 +224,9 @@ components:
           examples:
             - Not enough credits.
 
-    ToolsSummary:
+    RuleEnginesSummary:
       type: object
-      description: Count and names of a set of tools.
+      description: Count and names of a set of rule engines.
       properties:
         number:
           type: integer
@@ -276,12 +276,12 @@ components:
           type: integer
         number of HTML elements reported as exhibiting issues:
           type: integer
-        tools that tried to test the page:
-          $ref: '#/components/schemas/ToolsSummary'
-        tools that were unable to test the page:
-          $ref: '#/components/schemas/ToolsSummary'
-        tools that reported issues:
-          $ref: '#/components/schemas/ToolsSummary'
+        rule engines that tried to test the page:
+          $ref: '#/components/schemas/RuleEnginesSummary'
+        rule engines that were unable to test the page:
+          $ref: '#/components/schemas/RuleEnginesSummary'
+        rule engines that reported issues:
+          $ref: '#/components/schemas/RuleEnginesSummary'
         URLs for getting data on the reported issues:
           $ref: '#/components/schemas/NextTierURLs'
         URL for getting the full technical report as JSON:
@@ -301,7 +301,7 @@ components:
 
     IssueEntry:
       type: object
-      description: Details about a specific accessibility issue found on a page.
+      description: Details about a specific front-end quality issue found on a page.
       properties:
         identifier:
           type: string
@@ -329,8 +329,8 @@ components:
         impact on a user:
           type: string
           description: How this issue is likely to affect users.
-        tools reporting the issue:
-          $ref: '#/components/schemas/ToolsSummary'
+        rule engines reporting the issue:
+          $ref: '#/components/schemas/RuleEnginesSummary'
         number of HTML elements reported as exhibiting the issue:
           type: integer
         URLs for details about the issue on the page:
@@ -379,16 +379,16 @@ components:
                 URL:
                   type: string
                   format: uri
-            tools that tried to test the page:
+            rule engines that tried to test the page:
               type: array
               items:
-                $ref: '#/components/schemas/ToolInfo'
-            tools that were unable to test the page:
+                $ref: '#/components/schemas/RuleEngineInfo'
+            rule engines that were unable to test the page:
               type: array
               items:
-                $ref: '#/components/schemas/ToolFailure'
-            tools that reported issues:
-              $ref: '#/components/schemas/ToolsSummary'
+                $ref: '#/components/schemas/RuleEngineFailure'
+            rule engines that reported issues:
+              $ref: '#/components/schemas/RuleEnginesSummary'
             number of issues reported:
               type: object
               properties:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jrpool/kilotest",
-  "version": "31.2.2",
+  "version": "33.0.0",
   "description": "An ensemble testing service with a focus on accessibility",
   "main": "index.js",
   "scripts": {
@@ -24,8 +24,10 @@
   },
   "homepage": "https://github.com/jrpool/kilotest",
   "dependencies": {
+    "@modelcontextprotocol/sdk": "*",
     "dotenv": "*",
-    "testilo": "*"
+    "testilo": "*",
+    "zod": "*"
   },
   "devDependencies": {
     "@eslint/css": "^1.0.0",

package/pm2.config.js

@@ -1,15 +1,17 @@
 module.exports = {
-  apps: [{
-    name: 'kilotest',
-    script: 'index.js',
-    instances: 1,
-    autorestart: true,
-    watch: false,
-    max_memory_restart: '500M',
-    env: {
-      NODE_ENV: 'production',
-      BASE_PATH: '/',
-      DEMO_SSE_DELAY_MS: '100'
+  apps: [
+    {
+      name: 'kilotest',
+      script: 'index.js',
+      instances: 1,
+      autorestart: true,
+      watch: false,
+      max_memory_restart: '500M',
+      env: {
+        NODE_ENV: 'production',
+        BASE_PATH: '/',
+        DEMO_SSE_DELAY_MS: '100'
+      }
     }
-  }]
+  ]
 };

package/reportIssue/index.js

@@ -82,7 +82,7 @@ const populateQuery = async (issueID, timeStamp, jobID, query) => {
     violatorData.reporters = getToolNamesString(violatorData.reporters);
   });
   const reporterCount = query.reporters.size;
-  query.reporterCount = reporterCount === 1 ? '1 tool' : `${reporterCount} tools`;
+  query.reporterCount = reporterCount === 1 ? '1 rule engine' : `${reporterCount} rule engines`;
   // Convert the set of issue reporters to a string.
   query.reporters = getToolNamesString(query.reporters);
   // Convert the violator data to an array.

package/reportIssues/api.js

@@ -29,7 +29,7 @@ const getIssueFacts = (thisHost, timeStamp, jobID, issue) => {
       'numeric identifier': wcag
     },
     'impact on a user': why,
-    'tools reporting the issue': {
+    'rule engines reporting the issue': {
       'number': reporterCount,
       'names': reporters.map(tool => tool.toolName)
     },
@@ -40,7 +40,7 @@ const getIssueFacts = (thisHost, timeStamp, jobID, issue) => {
     }
   };
 };
-// Returns a response to a target-issues request.
+// Returns a response to a report-issues request.
 exports.response = async args => {
   const [timeStamp, jobID] = args;
   const reportIsHidden = await isHidden(timeStamp, jobID);
@@ -63,12 +63,13 @@ exports.response = async args => {
   const thisHost = process.env.THIS_KILOTEST_HOST;
   // Get a response.
   const content = {
-    summary: `This document fulfills a request made by an agent to the Kilotest service. The agent requested data from a Kilotest report about the accessibility, usability, and standard-conformity of a web page. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten testing tools, performs tests on web pages, using a combination of rule- and machine-learning-based methods, and produces reports. Kilotest exposes several API endpoints for agents and several web UI URLs for humans to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of tools, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
-    'tool name': 'Kilotest',
+    summary: `This document fulfills a request made by a language model to a Kilotest tool. The model requested data from a Kilotest report about the front-end quality (i.e. accessibility, usability, and standard-conformity) of a web page. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten rule engines, performs tests on web pages, using a combination of rule- and machine-learning-based methods, and produces reports. Kilotest exposes several API endpoints recommend web pages for testing and to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
+    'tool collection name': 'Kilotest',
+    'tool name': 'describeQualityOfOneWebPage',
     request: {
       'type of request': {
         identifier: 'reportIssues',
-        description: 'What issues does the specified report describe?'
+        description: 'Describe the quality of one web page.'
       },
       method: 'GET',
       URLs: {
@@ -76,9 +77,10 @@ exports.response = async args => {
         'equivalent URL for humans': `${thisHost}/reportIssues.html/${timeStamp}/${jobID}`
       },
       'closest ancestor request': {
-        description: 'Which web pages are reports available about, and what are the statistics about the issues reported for each page?',
+        identifier: 'summarizeQualityOfAllTestedWebPages',
+        description: 'Summarize the quality of all tested web pages.',
         URLs: {
-          'for you': `${thisHost}/api/targets.html`,
+          'for you': `${thisHost}/api/targets`,
           'for humans': `${thisHost}/targets.html`
         }
       }
@@ -96,9 +98,9 @@ exports.response = async args => {
       description: what,
       URL: url
     },
-    'tools that tried to test the page': getToolsFacts(Object.keys(tools)),
-    'tools that were unable to test the page': preventedTools,
-    'tools that reported issues': {
+    'rule engines that tried to test the page': getToolsFacts(Object.keys(tools)),
+    'rule engines that were unable to test the page': preventedTools,
+    'rule engines that reported issues': {
       number: reporterCount,
       names: reporters.map(tool => tool.toolName)
     },

package/reportIssues/index.js

@@ -65,7 +65,7 @@ const populateQuery = async (timeStamp, jobID, query) => {
   query.timeStamp = timeStamp;
   query.jobID = jobID;
   // Add reporter information to the query.
-  query.reporterCount = reporterCount === 1 ? '1 tool' : `${reporterCount} tools`;
+  query.reporterCount = reporterCount === 1 ? '1 rule engine' : `${reporterCount} rule engines`;
   query.reporters = reporterList;
   // Add a summary of the issues to the query.
   query.issueCount = issueCount === 1 ? '1 issue was' : `${issueCount} issues were`;
@@ -107,7 +107,7 @@ const populateQuery = async (timeStamp, jobID, query) => {
         // Add the issue facts to the lines.
         detailsLines.push(`${margin}      <li>Why it matters: ${why}`);
         detailsLines.push(`${margin}      <li>Related WCAG standard: ${wcagLink}`);
-        const reporterCountString = reporterCount === 1 ? '1 tool' : `${reporterCount} tools`;
+        const reporterCountString = reporterCount === 1 ? '1 rule engine' : `${reporterCount} rule engines`;
         detailsLines.push(
           `${margin}      <li>Reported by ${reporterCountString} (${reporterList})</li>`
         );

package/targets/api.js

@@ -51,15 +51,15 @@ exports.response = async () => {
         URL: url
       },
       'whether a later report about the same page exists': !! superseded,
-      'tools that tried to test the page': {
+      'rule engines that tried to test the page': {
         number: toolCount,
         names: toolNames
       },
-      'tools that were unable to test the page': {
+      'rule engines that were unable to test the page': {
         number: preventedToolCount,
         names: preventedToolNames
       },
-      'tools that reported issues': {
+      'rule engines that reported issues': {
         number: reporterCount,
         names: reporterNames
       },
@@ -74,12 +74,13 @@ exports.response = async () => {
   }
   // Get a response.
   const content = {
-    summary: `This document fulfills a request made by an agent to the Kilotest service. The agent requested data about the web pages that Kilotest had tested for accessibility, usability, and standard-conformity and, for each page, statistics about the results of the tests. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten testing tools, performs tests on web pages, using a combination of rule- and machine-learning-based methods, and produces reports. Kilotest exposes API endpoints for agents and web UI URLs for humans to recommend web pages for testing and to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of tools, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), whose home page contains an introduction and a link to a tutorial.`,
-    'tool name': 'Kilotest',
+    summary: `This document fulfills a request made by an agent to the Kilotest service. The agent requested data about the web pages that Kilotest had tested for accessibility, usability, and standard-conformity and, for each page, statistics about the results of the tests. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten rule engines, performs tests on web pages, using a combination of rule- and machine-learning-based methods, and produces reports. Kilotest exposes API endpoints for agents and web UI URLs for humans to recommend web pages for testing and to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), whose home page contains an introduction and a link to a tutorial.`,
+    'tool collection name': 'Kilotest',
+    'tool name': 'summarizeQualityOfAllTestedWebPages',
     request: {
       'type of request': {
         identifier: 'targets',
-        description: 'Give me summary data about each available report.'
+        description: 'Summarize the quality of all tested web pages.'
       },
       method: 'GET',
       URLs: {

package/testRecForm/api.js

@@ -20,12 +20,13 @@ exports.response = async (what, url, why) => {
   await updateRecs(what, url, why);
   // Get a response.
   const content = {
-    summary: `This response acknowledges a request made by an agent to the Kilotest service. The agent recommended that Kilotest test, for the first time, the ${what} web page at ${url} for accessibility, usability, and standard-conformity. A Kilotest manager usually approves a recommendation within a day. When the recommendation is approved, the testing will be performed and results will become available. You can check for the availability of the results at ${thisHost}/api/targets. Kilotest performs its testing with the help of Testaro, Testilo, and an ensemble of ten testing tools, using a combination of rule- and machine-learning-based methods. Kilotest exposes several API endpoints for agents and several web UI URLs for humans to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of tools, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
-    'tool name': 'Kilotest',
+    summary: `This response acknowledges a request made by an agent to the Kilotest service. The agent recommended that Kilotest test, for the first time, the ${what} web page at ${url} for accessibility, usability, and standard-conformity. A Kilotest manager usually approves a recommendation within a day. When the recommendation is approved, the testing will be performed and results will become available. You can check for the availability of the results at ${thisHost}/api/targets. Kilotest performs its testing with the help of Testaro, Testilo, and an ensemble of ten rule engines, using a combination of rule- and machine-learning-based methods. Kilotest exposes several API endpoints for agents and several web UI URLs for humans to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
+    'tool collection name': 'Kilotest',
+    'tool name': 'recommendQualityTestingOfOneWebPage',
     request: {
       'type of request': {
         identifier: 'testRecForm',
-        description: 'I recommend that Kilotest test a particular web page.'
+        description: 'Recommend quality testing of one web page.'
       },
       method: 'POST',
       payload: {

package/tutorial/index.html

@@ -433,7 +433,7 @@
         <h2>Further reading</h2>
         <ul>
           <li><a href="https://www.w3.org/WAI/WCAG22/Understanding/">Understanding WCAG 2.2</a> — W3C explanations of each success criterion</li>
-          <li><a href="https://www.w3.org/WAI/test-evaluate/tools/list/">Web Accessibility Evaluation Tools List</a> — W3C registry of rule-engine tools</li>
+          <li><a href="https://www.w3.org/WAI/test-evaluate/tools/list/">Web Accessibility Evaluation Tools List</a> — W3C registry of software that performs accessibility testing</li>
           <li><a href="https://www.w3.org/WAI/WCAG22/Understanding/identify-input-purpose">Understanding SC 1.3.5: Identify Input Purpose</a> — detailed guidance on <code>autocomplete</code> requirements</li>
           <li><a href="https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#autofill">HTML Living Standard: Autofill</a> — the definitive list of valid <code>autocomplete</code> tokens</li>
           <li><a href="https://arxiv.org/abs/2304.07591">Accessibility Metatesting: Comparing Nine Testing Tools</a> — research on rule-engine coverage variation</li>