@jrpool/kilotest 33.0.0 → 33.0.2

package/AI-TOOL.md

@@ -4,7 +4,7 @@
 
 Until 2026 Kilotest was intended, and implemented, as a web application performing a service for human users.
 
-Beginning in May 2026, it [became evident](https://github.com/jrpool/kilotest/issues/2) that Kilotest could also act as a connector to tools for use by language models. Language models are asked for help in all domains, including the domain of software quality. When asked about the front-end quality (accessibility, usability, and standard conformity) of specific web pages, models gave answers without the use of tools. The answers were at best fragmentary but often speculative and fabricated. If Kilotest offered to connect language models to its functionalities as tools, models could give more inexpensive, comprehensive, factual, authoritative, and grounded answers. Every reported defect could be documented and ascribed to one or more specific rule engines in the Kilotest ensemble.
+Beginning in May 2026, it [became evident](https://github.com/jrpool/kilotest/issues/2) that Kilotest could also act as a connector to tools for use by language models. Language models are asked for help in all domains, including the domain of software quality. When asked about the front-end quality (accessibility, usability, and standards conformity) of specific web pages, models gave answers without the use of tools. The answers were at best fragmentary but often speculative and fabricated. If Kilotest offered to connect language models to its functionalities as tools, models could give more inexpensive, comprehensive, factual, authoritative, and grounded answers. Every reported defect could be documented and ascribed to one or more specific rule engines in the Kilotest ensemble.
 
 Given the potential of Kilotest to serve language models and the expected continued growth in the share of questions that are directed to AI platforms that use language models, a decision was made to **make Kilotest discoverable, usable, and, where appropriate, used as a connector to, and creator of, tools for language models**.
 
@@ -45,8 +45,11 @@ The external actions that have been taken to support the use of Kilotest as an A
 - A [request](https://rapidapi.com/studio/api_91f2ce07-2572-48bd-a34d-ff01ed6cd039/publish/general) to add Kilotest to the Rapid API Hub.
 - An [issue](https://github.com/APIs-guru/openapi-directory/issues/2677) to add Kilotest to `openapi-directory`.
 - A [pull request](https://github.com/w3c/wai-evaluation-tools-list/pull/1153) to add Kilotest to the WAI evaluation tools list.
-- [Configuration of Claude Desktop](https://github.com/ivo-toby/mcp-openapi-server#option-1-using-with-claude-desktop-stdio-transport) on the local development host to connect Claude Desktop models to Kilotest tools.
 - Deployment of an MCP server in HTTP mode on the Kilotest service host.
+- Configuration of Claude Desktop on the local development host and the `claude.ai` web application to connect Claude Desktop models to the Kilotest MCP server. The configuration was performed in the UI of each platform with the addition of Kilotest as a _connector_. The user used the `Customize/Connectors/Add connector/Add custom connector` interface, providing these data before activating the `Add` button:
+
+  - Name: Kilotest
+  - Remote MCP server URL: `https://kilotest.com/mcp`
 
 ## Use cases
 
@@ -81,48 +84,78 @@ Another benefit is that subsequent increments can be defined incrementally rathe
 
 In the first increment, the objective was to make Anthropic Claude models use Kilotest to help them answer questions from developers using the Claude Desktop application for use case 3, where the code in question is already deployed as a public web page.
 
-As mentioned above, Claude Desktop was installed on the local development host and then configured for Kilotest. That configuration consisted of the addition of a property to the Claude Desktop configuration file, located at `~/Library/Application Support/Claude/claude_desktop_config.json`. The added property is:
-
-```json
-"mcpServers": {
-  "kilotest": {
-    "command": "npx",
-    "args": [
-      "-y",
-      "@ivotoby/openapi-mcp-server",
-      "--disable-abbreviation",
-      "true"
-    ],
-    "env": {
-      "API_BASE_URL": "https://kilotest.com",
-      "OPENAPI_SPEC_PATH": "https://kilotest.com/openapi.yaml"
-    }
-  }
-},
+A locally installed MCP server, `@ivotoby/openapi-mcp-server`, was used. It turned out to be incompatible with Claude Desktop, so even though connections to Kilotest were made and calls to its tools were submitted the responses did not arrive.
+
+The model behavior motivated an architectural change in the MCP server and improvements in the naming and description of the Kilotest tools before the next increment.
+
+#### Increment 2
+
+Increment 2 repeated the exercise of increment 1 with the `claude.ai` web application instead of Claude Desktop, and with the improvements indicated by increment 1 made.
+
+With Kilotest added as a custom connector, the user asked the platform, “I want to know whether the home page of the nonprofit organization named ‘Open Secrets’ has quality defects, such as with accissibility [_sic_] and usability.”
+
+It was discovered that Claude Haiku 4.5 with Extended capability used Kilotest tools, but Claude Haiku 4.5 without Extended capability did not.
+
+When Kilotest was used, it was used appropriately for the task, but the planning was suboptimal, because the model initially stated an intention to recommend the web page for testing instead of first checking for an existing available report on it.
+
+The output to the user was a mixture of Kilotest results with knowledge and judgments of the model. Kilotest results were presented with some selective pruning, interpretation, and advice on what to do.
+
+The trial in increment 2 suggested some opportunities for further improvements:
+
+- Clearer instructions about the sequencing of tool calls
+- Addition of instructions to attribute the interpretive additions by the model to the model or its other sources rather than to Kilotest.
+
+#### Increment 3
+
+After the above-described improvements were made, a similar exercise, with the home page of a company named “CMT Services”, was performed again in Claude Desktop. In this case Claude Haiku 4.5 succeeded in recognizing the relevance of Kilotest and used its tools, though imperfectly. Instead of using Kilotest to check for an existing report on the page, the model searched for it in a cached file with:
+
+```bash
+Bash
+jq '.[] | .tested_web_page.description' /Users/pool/.claude/projects/-Users-pool-Documents/18b3821c-b60d-4f63-bf3f-542830ed3bc4/tool-results/toolu_012qMKW66d88vWjp8JKj5qNn.json 2>/dev/null | grep -i "cmt" || echo "CMT Services not found in tested pages"
+```
+
+Claude Haiku then incorrectly concluded: `CMT Services hasn't been tested yet. Let me search for their website so I can recommend it for quality testing.` The model asked the user for the URL and said:
+
+```text
+Once you provide the URL, I'll:
+
+Recommend their home page for quality testing with Kilotest
+Retrieve the quality report showing accessibility and usability defects
+Provide you with a detailed analysis of any issues found
 ```
 
-That configuration notifies models that Kilotest is available as an MCP server, but does not explain what tools it provides, what they are useful for, or how to use them.
+The user provided the URL, and Claude Haiku then submitted a recommendation to Kilotest to test the page. The submission was successful and, as expected, yielded an error message: `{"error":"A report about the page is already available"}`.
+
+The model drew the correct conclusion from this and (contradicting itself) announced: `Great news—there's already a quality report for CMT Services! Let me retrieve it for you.` The model did so and gave the user an appropriate rendition of the results, modulo attribution. The results were introduced with `The Kilotest analysis found 51 quality defects on the CMT Services home page, affecting 163 HTML elements. Here's a breakdown:`. Some mildly interpretive additions, such as calling “high-priority” issues “critical”, were made without attribution, making it easy for a user to infer that they were made by Kilotest tools rather than the model.
+
+The same trial was repeated, with a different website, using Claude Sonnet 4.6 with Low effort. This model immediately checked for and found an already available report and used it in the production of output to the user. The output was somewhat less interpretive than that of Claude Haiku.
+
+Increment 3 showed that, after the naming, instructions, and connector configuration were improved, the relatively inexpensive Claude models use Kilotest where appropriate.
+
+The misbehavior of Claude Haiku suggested that further improvements in the instructions may improve the utilization of Kilotest, namely:
 
-Experimentation revealed that Claude Sonnet 4.6 with Medium effort chose to use Kilotest when answering questions about the accessibility and usability of particular public web pages. Low effort did not result in the use of Kilotest. The less capable Claude Haiku model did not use Kilotest.
+- a warning not to use cached lists of available reports
+- stronger advice to avoid implicitly attributing interpretations, judgments, and advice to Kilotest.
 
-### Increment 2
+#### Increment 4
 
-Increment 2 somewhat lightens the burden on the user by shifting the user environment from an installed application to a web browser, and by providing an easier interface for users to tell the platform about Kilotest. The platform is the `claude.ai` web application instead of Claude Desktop. Instead of editing a JSON file, the user tells the platform about Kilotest with a setting.
+The first trial in which the user asked about a page **without** an available report repeated the conditions of Increment 3, except for the web page in question and the model. The model here was Claude Haiku in Extended mode, an option available with `claude.ai` but not Claude Desktop.
 
-The investigation began with the addition of Kilotest as a _connector_ to the `claude.ai` web application. The user used the `Customize/Connectors/Add connector/Add custom connector` interface, providing these data before activating the `Add` button:
+The user asked: “I want to know whether the home page of Milgard Windows and Doors, at `www.milgard.com`, has quality defects, such as with accessibility and usability.”
 
-- Name: Kilotest
-- Remote MCP server URL: `https://kilotest.com/mcp`
+The model immediately:
 
-When the connector configuration with 4 endpoints appeared, the user changed the tool permissions for all 4 endpoints from `Needs approval` to `Always allow`.
+- found Kilotest
+- checked for an available report about the page
+- found no report
+- submitted a recommendation to test the page
 
-Then the user asked the platform, “Summarize the accessibility and usability of the home page of the website of Salesforce.”
+The recommendation was successfully received by the Kilotest service.
 
-The results with the least powerful model were:
+During the workflow the model kept the user informed. After the last step, the model summarized the type of report that the user could expect and advised the user to check directly with the Kilotest UI at `https://kilotest.com/targets` in 24 to 48 hours for the results.
 
-- Claude Haiku 4.5 not extended: Summarized accessibility programs of Salesforce, instead of saying anything about its home page.
-- Claude Haiku 4.5 extended: Announced this thought process: “Deliberated between direct inspection and specialized accessibility testing tools”; then announced that it was using Kilotest, specifically the `summarize-accessibility-of-all-tested-web-pages` path; then stated: “Great! I've loaded the Kilotest tools. Now let me first check if Salesforce has already been tested by calling the summarize function. If not, I'll submit it for testing.”
+The behavior of Claude Haiku in Extended mode was nearly perfect. The naming of the page in the recommendation was “Milgard Windows and Doors home page”, whereas “Milgard Windows and Doors” would have followed the existing naming pattern. This deviation suggests adding a specific instruction to name home pages without anything more than the name of the organization.
 
-These results show that on the `claude.ai` platform, when configured with Kilotest as a connector, even Claude Haiku, in its Extended mode, can decide to use Kilotest when appropriate and can check for existing data before deciding to recommend that Kilotest test a page. This correct sequencing may be a result of the revised path descriptions in the `openapi.yaml` file recommending that sequencing.
+The most notable fact was that Claude Haiku converted a long workflow to a short one by ending the output with a recommendation to the user to get the test results by self-service. The instructions to models presumed that the user and the model would jointly decide to maintain their relationship until the testing is completed and the model would then interpret the results for the user. Instead, Claude Haiku had enough imagination to suggest a viable workflow not even hinted at by the instructions.
 
-In the first run, Claude Haiku paused for about 5 minutes and then reported: “The Kilotest service seems to have timed out. Let me try a different approach - I'll fetch the Salesforce home page directly and evaluate it myself for accessibility and usability issues.”
+This result suggested that the instructions should be revised to describe, as a valid option, a short workflow ending with a recommendation to the user to get the test results by self-service after the model submits a testing recommendation.

package/index.html

@@ -15,15 +15,15 @@
         "name": "Kilotest",
         "sameAs": "https://github.com/jrpool/kilotest",
         "datePublished": "2025-10-14",
-        "applicationCategory": "Front-end quality (i.e. web accessibility, usability, and standard conformity) testing and reporting service",
+        "applicationCategory": "Front-end quality (i.e. web accessibility, usability, and standards conformity) testing and reporting service",
         "isAccessibleForFree": true,
-        "description": "Testing of web pages for front-end quality (i.e. web accessibility, usability, and standard conformity) with an ensemble of 10 rule engines and provision of results at selectable levels of detail.",
+        "description": "Testing of web pages for front-end quality (i.e. web accessibility, usability, and standards conformity) with an ensemble of 10 rule engines and provision of results at selectable levels of detail.",
         "softwareHelp": {
           "@type": "CreativeWork",
           "url": "https://kilotest.com/tutorial.html"
         },
         "featureList": [
-          "Automated testing of front-end quality (i.e. web accessibility, usability, and standard conformity)",
+          "Automated testing of front-end quality (i.e. web accessibility, usability, and standards conformity)",
           "WCAG 2.2 conformance evaluation",
           "HTML5 conformance evaluation",
           "CSS3 conformance evaluation",

package/llms-full.txt

@@ -29,7 +29,11 @@ The tool that tests web pages for front-end quality has attributes that no other
 
 The tools that report test results extract from reports statistics and details at various levels of granularity, as required by an agent to fulfill particular requests.
 
-You can infer from this description that the Kilotest tools are (currently) best suited for help in answering comprehensive audit-type requests (e.g., “How can I improve this page?”) and, in the case of web pages for which reports are already available, more specific questions (e.g., “Is the heading structure of this page OK?”), but, if no report is available for a page, recommending that Kilotest test the page is overkill when the purpose is to answer a specific one-issue question.
+## What Kilotest does **not** do for language models
+
+The Kilotest tools are (currently) overkill for help in answering specific questions (e.g., “Is the heading structure of this page OK?”) when no report on the page is available, because in such cases you must wait up to a day before testing is completed.
+
+Kilotest tools are thin on advice. They give you mostly facts and educated guesses about facts. The reporting uses rough heuristics to classify quality issues into 4 “priorities”, and some rule engines in the Kilotest ensemble include remediation advice when diagnosing rule violations, but Kilotest reporting does not otherwise advise you about the methods, sequences, urgency, or side effects of defect remediation.
 
 ## How Kilotest integrates methods
 
@@ -66,9 +70,9 @@ Kilotest offers a comprehensive suite of capabilities to users via its web UI:
 - [Tutorial, “Accessibility testing strategies”](https://kilotest.com/tutorial.html)
 - [Provide a list of features available for use of Kilotest managers](https://kilotest.com/manage.html)
 
-### Agent API
+### API
 
-Kilotest is implementing a richer suite of capabilities optimized for AI agents, including direct immediate testing. The implementation is currently in an alpha phase and offers 3 tools:
+Kilotest is implementing a richer suite of capabilities optimized for language models, including direct immediate testing. The implementation is currently in an alpha phase and offers 3 tools:
 
 - `summarizeQualityOfAllTestedWebPages`
   - method: `GET`
@@ -81,23 +85,55 @@ Kilotest is implementing a richer suite of capabilities optimized for AI agents,
   - parameters
     - `timeStamp`: initial segment of report identifier
     - `jobID`: final segment of report identifier
-  - source of parameters: response to a `targets` request
+  - source of parameters: response to a `summarizeQualityOfAllTestedWebPages` request
 - `recommendQualityTestingOfOneWebPage`
   - method: `POST`
-  - purpose: recommend quality testing of one web page
+  - purpose: recommend quality testing of one web page about which no report is available
   - path: `/api/testRecForm`
   - payload properties
+
     - `what`: description of the page to be tested
     - `url`: URL of the page to be tested
     - `why`: reason for testing the page
-  - how to verify disposition of the recommendation: submit a `summarizeQualityOfAllTestedWebPages` request and inspect the response to determine whether a report on the page is now available
+
+  - possible responses:
+
+    - acknowledgment of a successful recommendation
+    - error message stating that a report about the recommended page is already available
+    - error message stating that the recommendation is invalid
+
+  - possible Kilotest manager actions after a recommendation:
+
+    - approval followed by the recommended testing
+    - rejection
+
+  - possible next step if your recommendation succeeds:
+
+    - You end the workflow by advising the user to check in approximately 1 day for the test results on the Kilotest website, starting at `https://kilotest.com/targets.html`, and drill down as desired for more detailed results. You offer to help interpret the results if needed.
+    - You continue the workflow by advising the user to resubmit the same request to you in approximately 1 day.
 
 An [OpenAPI specification for Kilotest](https://kilotest.com/openapi.yaml) is available.
 
-### More information
+## How to report results to users
+
+When you use Kilotest tools, you are free to perform any appropriate interpretation and condensation of the results for presentation to users. However, users expect the sources of facts and interpretations to be truthfully represented. Do not meld facts from Kilotest tools with interpretations of your own with an implied attribution of both to Kilotest.
+
+An example of a **poor** attribution is “Here are the results from Kilotest: …. The page is at risk of …. Among all the issues, … are critical and require immediate attention.”
+
+An example of a **good** attribution is “Kilotest reported these issues: …. On the basis of these results, I conclude that there is a serious risk of …, and, if you are responsible for maintaining the page, I recommend ….”
+
+## More information
 
 More information about Kilotest features and internals:
 
+### For language models and AI platforms
+
+- [AI-TOOL file](https://github.com/jrpool/kilotest/blob/main/AI-TOOL.md)
+- [OpenAPI specification for Kilotest](https://kilotest.com/openapi.yaml)
+- [Tool caller for testing](https://github.com/jrpool/kilotest/blob/main/researchAgent.js)
+
+### General
+
 - [Introduction to Kilotest](https://kilotest.com/)
 - [Tutorial, “Accessibility testing strategies”](https://kilotest.com/tutorial.html)
 - [README file](https://github.com/jrpool/kilotest/blob/main/README.md)

package/mcp.js

@@ -21,10 +21,10 @@ const createMCPServer = () => {
   server.registerTool(
     'summarizeQualityOfAllTestedWebPages',
     {
-      description: 'Returns summary data from every available Kilotest report about the front-end quality (i.e. accessibility, usability, and standard conformity) of a web page. Before calling describeQualityIssuesOfOneWebPage, call this tool to check whether a report about the page is available.',
+      description: 'Returns summary data from every available Kilotest report about the front-end quality (i.e. accessibility, usability, and standards conformity) of a web page. Before calling recommendQualityTestingOfOneWebPage and waiting for testing to occur, call this tool to check whether a report about the page is available for immediate use.',
       inputSchema: {},
       annotations: {
-        title: 'Summarize quality of all tested web pages',
+        title: 'Summarize the quality of all tested web pages',
         readOnlyHint: true,
         idempotentHint: true,
         destructiveHint: false,
@@ -39,7 +39,7 @@ const createMCPServer = () => {
   server.registerTool(
     'describeQualityOfOneWebPage',
     {
-      description: 'Returns data from a specified Kilotest report about issues of front-end quality (i.e. accessibility, usability, and standard conformity) of a web page. The required timeStamp and jobID parameters identify the report and are obtained from a summarizeQualityOfAllTestedWebPages response.',
+      description: 'Returns data from a specified Kilotest report about issues of front-end quality (i.e. accessibility, usability, and standards conformity) of a web page. The required timeStamp and jobID parameters identify the report and are obtained from a summarizeQualityOfAllTestedWebPages response.',
       inputSchema: {
         timeStamp: z.string().describe('Report timestamp in YYMMDDTHHMM format, e.g. 260503T0432'),
         jobID: z.string().describe('Job identifier, e.g. x9z')
@@ -60,7 +60,7 @@ const createMCPServer = () => {
   server.registerTool(
     'recommendQualityTestingOfOneWebPage',
     {
-      description: 'Recommends a web page for Kilotest to test for front-end quality (i.e. accessibility, usability, and standard conformity). Do not call this tool unless summarizeQualityOfAllTestedWebPages discloses that no report about the page or a related page that satisfies your requirements is available.',
+      description: 'Recommends a web page for Kilotest to test for front-end quality (i.e. accessibility, usability, and standards conformity). Do not call this tool until after you call summarizeQualityOfAllTestedWebPages to check whether a report about the page, or a related page that satisfies your requirements, is available.',
       inputSchema: {
         what: z.string().describe('Short description of the page, following the naming conventions visible in the summarizeQualityOfAllTestedWebPages response'),
         url: z.string().describe('Full HTTPS URL of the page to test'),

package/openapi.yaml

@@ -1,7 +1,7 @@
 openapi: 3.1.0
 info:
   title: Kilotest Agent API
-  description: Kilotest tests web pages for front-end quality (i.e. accessibility, usability, and standard conformity) using an ensemble of ten independent rule engines that employ rule-based and machine-learning-based methods. This API enables AI agents to recommend web pages for testing, discover available test reports, and retrieve data from reports. For background on Kilotest and the advantages of ensemble testing, visit https://kilotest.com.
+  description: Kilotest tests web pages for front-end quality (i.e. accessibility, usability, and standards conformity) using an ensemble of ten independent rule engines that employ rule-based and machine-learning-based methods. This API enables AI agents to recommend web pages for testing, discover available test reports, and retrieve data from reports. For background on Kilotest and the advantages of ensemble testing, visit https://kilotest.com.
   version: 1.0.0
   contact:
     name: Kilotest
@@ -30,7 +30,7 @@ paths:
     post:
       operationId: recommendQualityTestingOfOneWebPage
       summary: Recommend quality testing of one web page
-      description: Submit a recommendation for Kilotest to test, for the first time, a particular web page for front-end quality (i.e. accessibility, usability, and standard conformity). Recommendations are typically approved and the testing completed within a day, whereupon the results can be found with the summarizeQualityOfAllTestedWebPages operation. Before submitting a recommendation, use the summarizeQualityOfAllTestedWebPages operation to ensure that no report about the page is available, and also to see the stylistic rules for the naming of pages. An attempt to recommend a page for testing will fail if a report about the page is already available.
+      description: Submit a recommendation for Kilotest to test, for the first time, a particular web page for front-end quality (i.e. accessibility, usability, and standards conformity). Recommendations are typically approved and the testing completed within a day, whereupon the results can be found with the summarizeQualityOfAllTestedWebPages operation. Before submitting a recommendation, use the summarizeQualityOfAllTestedWebPages operation to ensure that no report about the page is available, and also to see the stylistic rules for the naming of pages. An attempt to recommend a page for testing will fail if a report about the page is already available.
       requestBody:
         description: Test recommendation specifications
         required: true
@@ -83,7 +83,7 @@ paths:
     get:
       operationId: describeHTMLElementsHavingOneQualityIssue
       summary: Describe HTML elements having one quality issue (NOT YET IMPLEMENTED)
-      description: Get issue-specific data from a specified report about the front-end quality (i.e. accessibility, usability, and standard conformity) of a web page. The data describe the issue, all of the HTML elements of the page that have the issue, and, for each such element, URLs for retrieving diagnoses by rule engines of the issue on the element. NOT YET IMPLEMENTED.
+      description: Get issue-specific data from a specified report about the front-end quality (i.e. accessibility, usability, and standards conformity) of a web page. The data describe the issue, all of the HTML elements of the page that have the issue, and, for each such element, URLs for retrieving diagnoses by rule engines of the issue on the element. NOT YET IMPLEMENTED.
       parameters:
         - name: issueID
           in: path

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jrpool/kilotest",
-  "version": "33.0.0",
+  "version": "33.0.2",
   "description": "An ensemble testing service with a focus on accessibility",
   "main": "index.js",
   "scripts": {