@jrpool/kilotest 31.0.1 → 31.2.0

package/AI-TOOL.md

@@ -0,0 +1,66 @@
+# Kilotest as an AI Tool
+
+## Introduction
+
+Until 2026 Kilotest was intended, and implemented, as a web application for human users.
+
+Beginning in May 2026, it [became evident](https://github.com/jrpool/kilotest/issues/2) that Kilotest could also act as a tool for use by language models. Language models are asked for help in all domains, including the domain of software quality. When asked about the front-end quality (accessibility, usability, and standard conformity) of specific web pages, models gave answers without the use of tools, and the answers were inferior: simplistic, speculative, and fabricated. If a model were to use Kilotest as a tool, the model could give comprehensive, authoritative, grounded, and factual answers. Every reported defect could be documented and ascribed to one or more specific tools in the Kilotest ensemble.
+
+Given the potential of Kilotest as an AI tool and the expected continued growth in the share of questions that are directed to language models, a decision was made to make Kilotest discoverable and usable as a tool for language models.
+
+## Internal additions
+
+The internal changes that have been made in the Kilotest codebase to support the use of Kilotest as an AI tool are:
+
+- An API, consisting of:
+  - Additions to `index.js`.
+  - Addition to `util.js`.
+  - Within directories providing API functionality:
+    - `api.js` modules.
+    - `util.js` modules, if needed, containing resources shared by the `index.js` and `api.js` modules in those directories.
+- Additions to the `env.example` file.
+- An `llms.txt` file and an `llms-full.txt` file, documenting the use of Kilotest by language models, conforming to the [llms-txt](https://llmstxt.org/) specification.
+- An `openapi.yaml` file, documenting the Kilotest API, conforming to the [OpenAPI specification](https://spec.openapis.org/oas/v3.1.0).
+- A `sitemap.xml` file.
+- A `researchAgent.js` file, testing the API.
+
+## External actions
+
+The external actions that have been taken to support the use of Kilotest as an AI tool are:
+
+- A [pull request](https://github.com/public-apis/public-apis/pull/6346/changes) to add Kilotest to the list of public APIs in the `public-apis` repository.
+- A [request](https://rapidapi.com/studio/api_91f2ce07-2572-48bd-a34d-ff01ed6cd039/publish/general) to add Kilotest to the Rapid API Hub.
+- An [issue](https://github.com/APIs-guru/openapi-directory/issues/2677) to add Kilotest to `openapi-directory`.
+- A [pull request](https://github.com/w3c/wai-evaluation-tools-list/pull/1153) to add Kilotest to the WAI evaluation tools list.
+- [Configuration of Claude Desktop](https://github.com/ivo-toby/mcp-openapi-server#option-1-using-with-claude-desktop-stdio-transport) on the local development host to let Claude models find Kilotest.
+
+## Use cases
+
+The rationale for Kilotest as a tool for language models is set forth in the `llms-full.txt` file and is not repeated here.
+
+Some common anticipated use cases for this role are:
+
+- A user of a web-builder platform with responsibility for a website asks an AI platform for help in creating or improving the website.
+- A prospective customer of a web development service asks an AI platform to evaluate the quality of websites in the portfolios of candidate vendors.
+- A professional web developer within an organization asks an AI platform for a code review.
+- A risk-management professional within an organization asks an AI platform to report on any web accessibility defects that could expose the organization to prosecution or civil litigation.
+- An person who depends on web accessibility because of disabilities asks an AI platform to provide documentary support for a complaint to the owner of a website about accessibility defects.
+- A disability-rights advocate or attorney concerned with inaccessibility in a particular industry asks an AI platform to perform a front-end-quality comparison of some websites in that industry.
+
+## Strategy
+
+### Ease of use
+
+Any use of Kilotest as a tool of language models will fail for some of the anticipated users if they are obligated to be aware of Kilotest and to translate that awareness into instructions or documentation. Therefore, for success in all the use cases, users should be able to tell models what is wanted and rely on models to figure out whether they need tools and, if so, which ones and how to use them.
+
+### Increments
+
+The standardization of tool discovery and utilization by and for language models and AI platforms is partial. Major differences in protocols exist among model and platform families. Therefore, small testable increments of improvement in the tool functionality of Kilotest can best be defined by model and platform family. For example, working on discoverability first and then on usability would not be effective, because both are complex and testability would be postponed until both were complete. Instead, a coherent model and platform family should be identified and any and all improvements to make use cases successful within that family should be made and tested, before development proceeds to the next family.
+
+One benefit of this type of incrementalism is that, after the first increment succeeds, it becomes possible to make and test external changes publicizing the fact that a particular platform-model combination delivers unprecedentedly comprehensive, truthful, and low-cost answers to questions about front-end web quality.
+
+Another benefit is that subsequent increments can be defined incrementally rather than in advance. Lessons learned from the work on each increment can inform the choice of what to work on next.
+
+#### Increment 1
+
+In the first increment, the objective is to make Kilotest automatically discovered and used by Anthropic Claude models when used within the Claude Desktop application. That investigation is under way. Results will be summarized here.

package/eslint.config.mjs

@@ -8,7 +8,7 @@ import { defineConfig } from "eslint/config";
 export default defineConfig([
   {
     ignores: [
-      "DEVELOPMENT.md",
+      "IDEAS.md",
       "package-lock.json"
     ]
   },

package/index.js

@@ -171,6 +171,17 @@ const checkBalancesForAlerts = async report => {
     }
   }
 };
+// Minifies a URL.
+const minifyURL = url => url.replace(/www\.|\/$/g, '');
+// Returns whether a report on a page is available.
+const isReportAvailable = async (what, url) => {
+  const logs = await getLogs();
+  const whats = logs.map(log => log.what);
+  const urls = logs.map(log => log.url);
+  const miniURLs = urls.map(url => minifyURL(url));
+  const miniURL = minifyURL(url);
+  return whats.includes(what) || miniURLs.includes(miniURL);
+};
 // Handles a request.
 const requestHandler = async (request, response) => {
   // Sets response headers.
@@ -214,13 +225,6 @@ const requestHandler = async (request, response) => {
       setHeaders('text/html', '/index.html', 'medium');
       response.end(homePage);
     }
-    // Otherwise, if it is for the AI plugin specification:
-    else if (pathname === '/.well-known/ai-plugin.json') {
-      const aiPlugin = await fs.readFile('.well-known/ai-plugin.json', 'utf8');
-      // Serve it.
-      setHeaders('application/json', '/.well-known/ai-plugin.json', 'low');
-      response.end(aiPlugin);
-    }
     // Otherwise, if it is for the crawler specification:
     else if (pageName === 'robots.txt') {
       const robots = await fs.readFile('robots.txt', 'utf8');
@@ -457,20 +461,28 @@ const requestHandler = async (request, response) => {
       const {what, url, why} = postData;
       // If the request is valid:
       if (what && url.startsWith('https://') && why) {
-        // Serve headers for a response.
-        setHeaders('text/html', `${pathname}${search}`, 'high');
-        // Get the answer data.
-        const answerData = await require(path.join(__dirname, 'testRec', 'index'))
-        .answer(what, url, why);
-        // If they are valid:
-        if (answerData.status === 'ok') {
-          // Serve the answer page.
-          response.end(answerData.answerPage);
+        // If a report on the page is already available:
+        if (await isReportAvailable(what, url)) {
+          // Report the error.
+          await serveError({message: 'ERROR: Page has already been tested'}, response, true);
         }
-        // Otherwise, i.e. if they are invalid:
+        // Otherwise, i.e. if no report on the page is available:
         else {
-          // Report the error.
-          await serveError({message: answerData.error}, response, true);
+          // Serve headers for a response.
+          setHeaders('text/html', `${pathname}${search}`, 'high');
+          // Get the answer data.
+          const answerData = await require(path.join(__dirname, 'testRec', 'index'))
+          .answer(what, url, why);
+          // If they are valid:
+          if (answerData.status === 'ok') {
+            // Serve the answer page.
+            response.end(answerData.answerPage);
+          }
+          // Otherwise, i.e. if they are invalid:
+          else {
+            // Report the error.
+            await serveError({message: answerData.error}, response, true);
+          }
         }
       }
       // Otherwise, i.e. if the request is invalid:
@@ -683,19 +695,12 @@ const requestHandler = async (request, response) => {
       // Otherwise, if the first segment is the test recommendation service:
       else if (segments[0] === 'testRecForm') {
         const {what, url, why} = postData;
-        const logs = await getLogs();
-        const whats = logs.map(log => log.what);
-        const urls = logs.map(log => log.url);
         // If the payload is a valid test recommendation:
         if (what && isURL(url) && why) {
-          // If an available report has the same URL or the same page description:
-          if (whats.includes(what) || urls.includes(url)) {
+          // If a report on the page is already available:
+          if (await isReportAvailable(what, url)) {
             // Report this.
-            await serveError(
-              'ERROR: A report with the same page description or URL is already available',
-              response,
-              false
-            );
+            await serveError({message: 'ERROR: A report on the page is already available'}, response, false);
           }
           // Otherwise, i.e. if no report on the page is available:
           else {
@@ -707,6 +712,11 @@ const requestHandler = async (request, response) => {
             response.end(JSON.stringify(responseData));
           }
         }
+        // Otherwise, i.e. if it is not a valid test recommendation:
+        else {
+          // Report this.
+          await serveError({message: 'ERROR: Invalid test recommendation'}, response, false);
+        }
       }
       // Otherwise, i.e. if the request is invalid:
       else {

package/llms-full.txt

@@ -27,7 +27,7 @@ The rule engines in the Kilotest ensemble take responsibility for determining wh
 
 ## Kilotest versus pure inference
 
-LLMs that attempt to evaluate front-end quality with no delegation are likely to be unable to load and operate pages of web applications with browser automation, as Kilotest does. As a result, LLMs typically create hallucinated evaluations based on statistical frequency, with no mention of issues that are rarely tested for, instead of providing the truthful and comprehensive assessments that users require.
+LLM platforms asked to evaluate front-end quality with no delegation are not all able to load pages of web applications for inspection. When they can do so, they typically cannot operate on the pages with browser automation, as Kilotest does. As a result, LLMs typically give less comprehensive answers than they would if they called Kilotest as a tool. When even browsing to a page is impossible, an LLM is likely to give speculative evaluations with hallucinated examples, based on statistical frequency, with no mention of issues that are rarely tested for, instead of providing the truthful and comprehensive assessments that users require.
 
 ## Kilotest versus other tools
 

package/openapi.yaml

@@ -13,11 +13,24 @@ servers:
     description: Kilotest production server
 
 paths:
+  /api/targets:
+    get:
+      operationId: summarizeAccessibilityOfAllTestedWebPages
+      summary: Summarizes all available reports
+      description: Returns summary data about every non-hidden report available from Kilotest, including the name and URL of the tested web page, when the testing was performed, how many accessibility, usability, and standard-conformity issues were reported, and URLs for retrieving more detailed data from the report. This is the first endpoint to call if you want data about a particular web page. The result will tell you whether a report on that page already exists. If so, you can retrieve data from it. If not, you can use the submitWebAccessibilityTestRequest endpoint to recommend the page for testing.
+      responses:
+        '200':
+          description: Summaries of available reports
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/TargetsResponse'
+
   /api/testRecForm:
     post:
-      operationId: testRecForm
-      summary: Receives a testing recommendation
-      description: Receives a recommendation for Kilotest to test a particular web page.
+      operationId: submitWebAccessibilityTestRequest
+      summary: Receives a new testing recommendation
+      description: Receives a recommendation for Kilotest to test, for the first time, a particular web page for accessibility, usability, and standard conformity. Recommendations are typically approved and the testing completed within a day, whereupon the results can be found with the summarizeAccessibilityOfAllTestedWebPages operation. Before submitting a recommendation, use the summarizeAccessibilityOfAllTestedWebPages operation to ensure that the page has not yet been tested, and also to see the stylistic rules for the naming of pages. An attempt to recommend an already tested page for testing will fail.
       requestBody:
         description: Test recommendation specifications
         required: true
@@ -33,24 +46,11 @@ paths:
               schema:
                 $ref: '#/components/schemas/TestRecFormResponse'
 
-  /api/targets:
-    get:
-      operationId: targets
-      summary: Summarizes all available reports
-      description: Returns summary data about every non-hidden report available from Kilotest, including the name and URL of the tested web page, when the testing was performed, how many issues were reported, and URLs for retrieving more detailed data from the report.
-      responses:
-        '200':
-          description: Summaries of available reports
-          content:
-            application/json:
-              schema:
-                $ref: '#/components/schemas/TargetsResponse'
-
   /api/reportIssues/{timeStamp}/{jobID}:
     get:
-      operationId: getReportIssues
+      operationId: listAccessibilityIssuesOnOneWebPage
       summary: Gets data on issues from a specific report
-      description: Returns data about the issues reported in a specific Kilotest report, grouped by priority. The data on each issue include the tools that reported it, the number of HTML elements exhibiting it, and URLs for retrieving element-level detail. The timeStamp and jobID components identify the report and are available in the targets response.
+      description: Returns data about the issues reported in a specific Kilotest report, grouped by priority. The data on each issue include the tools that reported it, the number of HTML elements exhibiting it, and URLs for retrieving element-level detail. The timeStamp and jobID components identify the report and are available in the response from the summarizeAccessibilityOfAllTestedWebPages operation.
       parameters:
         - name: timeStamp
           in: path
@@ -81,14 +81,14 @@ paths:
 
   /api/reportIssue/{issueID}/{timeStamp}/{jobID}:
     get:
-      operationId: getReportIssue
+      operationId: listHTMLElementsHavingOneAccessibilityIssue
       summary: Gets details about a specific issue in a specific report (NOT YET IMPLEMENTED)
       description: Returns details about a single issue within a specific report, including which HTML elements exhibit the issue and, for each such element, URLs for retrieving tool-by-tool diagnoses of the issue on the element. NOT YET IMPLEMENTED.
       parameters:
         - name: issueID
           in: path
           required: true
-          description: Issue identifier (e.g., imageNoText). Available under "issues reported" > priority level > "identifier" in the getReportIssues response.
+          description: Issue identifier (e.g., imageNoText). Available under "issues reported" > priority level > "identifier" in the listAccessibilityIssuesOnOneWebPage response.
           schema:
             type: string
             examples:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jrpool/kilotest",
-  "version": "31.0.1",
+  "version": "31.2.0",
   "description": "An ensemble testing service with a focus on accessibility",
   "main": "index.js",
   "scripts": {

package/researchAgent.js

@@ -65,15 +65,16 @@ const requestService = async () => {
       const content = chunks.join('');
       try {
         // Output it.
-        const contentObj = JSON.parse(content);
-        console.log(JSON.stringify(contentObj, null, 2));
+        const targetsObj = JSON.parse(content);
+        console.log(JSON.stringify(targetsObj, null, 2));
+        const reports = targetsObj['available reports'];
         // Get the IDs of the available reports.
-        const reportIDs = contentObj['available reports'].map(report => report.identifier);
+        const reportIDs = reports.map(report => report.identifier);
         // Choose one at random.
         const [timeStamp, jobID] = reportIDs[Math.floor(Math.random() * reportIDs.length)]
         .split('-');
         const path = `/api/reportIssues/${timeStamp}/${jobID}`;
-  console.log('======================');
+        console.log('======================');
         console.log(`About to submit ${scheme} request as JSON on port ${port} to ${host}${path}`);
         const requestOptions = getRequestOptions(path);
         // Submit an issues request for it.
@@ -98,13 +99,13 @@ const requestService = async () => {
               // Output it.
               const contentObj = JSON.parse(content);
               console.log(JSON.stringify(contentObj, null, 2));
-              const testRecPath = `/api/testRecForm`;
+              const testRecGoodPath = `/api/testRecForm`;
               console.log('======================');
               console.log(
-                `About to submit ${scheme} POST request as JSON on port ${port} to ${host}${testRecPath}`
+                `About to submit good ${scheme} POST request as JSON on port ${port} to ${host}${testRecGoodPath}`
               );
-              const testRecOptions = getRequestOptions(testRecPath, 'POST');
-              // Submit a test recommendation.
+              const testRecOptions = getRequestOptions(testRecGoodPath, 'POST');
+              // Submit a good test recommendation.
               client.request(testRecOptions, response => {
                 // Initialize a collection of data from the response.
                 const chunks = [];
@@ -126,14 +127,59 @@ const requestService = async () => {
                     // Output it.
                     const contentObj = JSON.parse(content);
                     console.log(JSON.stringify(contentObj, null, 2));
+                    const testRecBadPath = `/api/testRecForm`;
+                    console.log('======================');
+                    console.log(
+                      `About to submit bad ${scheme} POST request as JSON on port ${port} to ${host}${testRecBadPath}`
+                    );
+                    const testRecOptions = getRequestOptions(testRecBadPath, 'POST');
+                    // Submit a bad test recommendation.
+                    client.request(testRecOptions, response => {
+                      // Initialize a collection of data from the response.
+                      const chunks = [];
+                      response
+                      // If the response throws an error:
+                      .on('error', async error => {
+                        // Report it.
+                        console.log(error.message);
+                      })
+                      // If the response delivers data:
+                      .on('data', chunk => {
+                        // Add them to the collection.
+                        chunks.push(chunk);
+                      })
+                      // When the response is completed:
+                      .on('end', async () => {
+                        const content = chunks.join('');
+                        try {
+                          // Output it.
+                          const contentObj = JSON.parse(content);
+                          console.log(JSON.stringify(contentObj, null, 2));
+                        }
+                        catch (error) {
+                          console.log(error.message);
+                          console.log(
+                            `Test recommendation response content: ${content || 'No content'}`
+                          );
+                        }
+                      })
+                    })
+                    // Finish sending the bad test recommendation request.
+                    .end(JSON.stringify({
+                      what: 'Page Wrongly Recommended',
+                      url: reports[Math.floor(Math.random() * reports.length)]
+                      ['tested web page']
+                      .URL,
+                      why: 'This URL has already been tested'
+                    }));
                   }
                   catch (error) {
                     console.log(error.message);
                     console.log(`Test recommendation response content: ${content || 'No content'}`);
                   }
-                });
+                })
               })
-              // Finish sending the test recommendation request.
+              // Finish sending the good test recommendation request.
               .end(JSON.stringify({
                 what: 'Page Not Already Tested',
                 url: 'https://pagenotalreadytested.info',

package/targets/api.js

@@ -9,8 +9,7 @@ const {
   getLogs,
   getNowStamp,
   getRandomString,
-  getReportData,
-  researchAgents
+  getReportData
 } = require('../util');
 
 // CONSTANTS

package/testRecForm/api.js

@@ -20,7 +20,7 @@ exports.response = async (what, url, why) => {
   await updateRecs(what, url, why);
   // Get a response.
   const content = {
-    summary: `This response acknowledges a request made by an agent to the Kilotest service. The agent recommended that Kilotest test the ${what} web page at ${url} for accessibility, usability, and standard-conformity. A Kilotest manager usually approves a recommendation within a day. When the recommendation is approved, the testing will be performed and results will become available. You can check for the availability of the results at ${thisHost}/api/targets. Kilotest performs its testing with the help of Testaro, Testilo, and an ensemble of ten testing tools, using a combination of rule- and machine-learning-based methods. Kilotest exposes several API endpoints for agents and several web UI URLs for humans to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of tools, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
+    summary: `This response acknowledges a request made by an agent to the Kilotest service. The agent recommended that Kilotest test, for the first time, the ${what} web page at ${url} for accessibility, usability, and standard-conformity. A Kilotest manager usually approves a recommendation within a day. When the recommendation is approved, the testing will be performed and results will become available. You can check for the availability of the results at ${thisHost}/api/targets. Kilotest performs its testing with the help of Testaro, Testilo, and an ensemble of ten testing tools, using a combination of rule- and machine-learning-based methods. Kilotest exposes several API endpoints for agents and several web UI URLs for humans to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of tools, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
     'tool name': 'Kilotest',
     request: {
       'type of request': {

package/.well-known/ai-plugin.json

@@ -1,18 +0,0 @@
-{
-  "schema_version": "v1",
-  "name_for_human": "Kilotest",
-  "name_for_model": "kilotest",
-  "description_for_human": "Ensemble testing of web pages for accessibility, usability, and standard conformance.",
-  "description_for_model": "Use Kilotest to retrieve test results about the accessibility, usability, and standard conformance of any web page that has been tested by Kilotest at selectable levels of granularity.",
-  "auth": {
-    "type": "none"
-  },
-  "api": {
-    "type": "openapi",
-    "url": "https://kilotest.com/openapi.yaml",
-    "is_user_authenticated": false
-  },
-  "logo_url": "https://kilotest.com/favicon.ico",
-  "contact_email": "info@kilotest.com",
-  "legal_info_url": "https://github.com/jrpool/kilotest/blob/main/LICENSE"
-}