@jrpool/kilotest 31.2.4 → 33.0.0

package/AI-TOOL.md

@@ -1,27 +1,40 @@
-# Kilotest as an AI Tool
+# Kilotest as connector to and creator of AI Tools
 
 ## Introduction
 
-Until 2026 Kilotest was intended, and implemented, as a web application for human users.
+Until 2026 Kilotest was intended, and implemented, as a web application performing a service for human users.
 
-Beginning in May 2026, it [became evident](https://github.com/jrpool/kilotest/issues/2) that Kilotest could also act as a tool for use by language models. Language models are asked for help in all domains, including the domain of software quality. When asked about the front-end quality (accessibility, usability, and standard conformity) of specific web pages, models gave answers without the use of tools, and the answers were inferior: simplistic, speculative, and fabricated. If a model were to use Kilotest as a tool, the model could give comprehensive, authoritative, grounded, and factual answers. Every reported defect could be documented and ascribed to one or more specific tools in the Kilotest ensemble.
+Beginning in May 2026, it [became evident](https://github.com/jrpool/kilotest/issues/2) that Kilotest could also act as a connector to tools for use by language models. Language models are asked for help in all domains, including the domain of software quality. When asked about the front-end quality (accessibility, usability, and standard conformity) of specific web pages, models gave answers without the use of tools. The answers were at best fragmentary but often speculative and fabricated. If Kilotest offered to connect language models to its functionalities as tools, models could give more inexpensive, comprehensive, factual, authoritative, and grounded answers. Every reported defect could be documented and ascribed to one or more specific rule engines in the Kilotest ensemble.
 
-Given the potential of Kilotest as an AI tool and the expected continued growth in the share of questions that are directed to language models, a decision was made to make Kilotest discoverable and usable as a tool for language models.
+Given the potential of Kilotest to serve language models and the expected continued growth in the share of questions that are directed to AI platforms that use language models, a decision was made to **make Kilotest discoverable, usable, and, where appropriate, used as a connector to, and creator of, tools for language models**.
+
+## Terms
+
+- **Language model**: A model (e.g., Claude Haiku 4.5, Kimi K2.6, GPT-5.4, Gemini 3.1 Pro) that can consume and generate text, images, or other content.
+- **AI platform**: A platform (e.g., Claude Desktop, Perplexity, ChatGPT, Gemini 3.5 Flash) that gives human users access to the services of language models and connects language models to productivity resources.
+- **tool**: A specialized productivity resource providing a capability that the language model itself needs but does not have. In the OpenAPI specification, the term _operation_ is used to mean tool.
+- **connector**: A service that allows AI platforms to enable their language models to discover, evaluate, and use tools.
+
+Tools can be independent of connectors that connect them to language models, but Kilotest can be both. It already provides capabilities to human users; it can re-use the same functions that underlie those capabilities and reconfigure those functions as tools for language models. Kilotest can also provide connectors to those tools for language models and make those connectors compatible with AI platforms.
 
 ## Internal additions
 
-The internal changes that have been made in the Kilotest codebase to support the use of Kilotest as an AI tool are:
+The internal changes that have been made in the Kilotest codebase to support the use of Kilotest as a set of tools and connectors to them are:
 
 - An API, consisting of:
   - Additions to `index.js`.
-  - Addition to `util.js`.
+  - Additions to `util.js`.
   - Within directories providing API functionality:
     - `api.js` modules.
     - `util.js` modules, if needed, containing resources shared by the `index.js` and `api.js` modules in those directories.
+- A [`JSON-LD`](https://json-ld.org/) script in the `index.html` file, providing structured data about the Kilotest API.
 - Additions to the `env.example` file.
 - An `llms.txt` file and an `llms-full.txt` file, documenting the use of Kilotest by language models, conforming to the [llms-txt](https://llmstxt.org/) specification.
 - An `openapi.yaml` file, documenting the Kilotest API, conforming to the [OpenAPI specification](https://spec.openapis.org/oas/v3.1.0).
+- An `mcp.js` file, providing an MCP server for Kilotest.
 - A `sitemap.xml` file.
+- Additions to the `README.md` file.
+- This `AI-TOOL.md` file.
 - A `researchAgent.js` file, testing the API.
 
 ## External actions
@@ -32,35 +45,84 @@ The external actions that have been taken to support the use of Kilotest as an A
 - A [request](https://rapidapi.com/studio/api_91f2ce07-2572-48bd-a34d-ff01ed6cd039/publish/general) to add Kilotest to the Rapid API Hub.
 - An [issue](https://github.com/APIs-guru/openapi-directory/issues/2677) to add Kilotest to `openapi-directory`.
 - A [pull request](https://github.com/w3c/wai-evaluation-tools-list/pull/1153) to add Kilotest to the WAI evaluation tools list.
-- [Configuration of Claude Desktop](https://github.com/ivo-toby/mcp-openapi-server#option-1-using-with-claude-desktop-stdio-transport) on the local development host to let Claude models find Kilotest.
+- [Configuration of Claude Desktop](https://github.com/ivo-toby/mcp-openapi-server#option-1-using-with-claude-desktop-stdio-transport) on the local development host to connect Claude Desktop models to Kilotest tools.
+- Deployment of an MCP server in HTTP mode on the Kilotest service host.
 
 ## Use cases
 
-The rationale for Kilotest as a tool for language models is set forth in the `llms-full.txt` file and is not repeated here.
+The rationale for Kilotest as a connector to tools for language models is set forth in the `llms-full.txt` file and is not repeated here.
 
 Some common anticipated use cases for this role are:
 
-- A user of a web-builder platform with responsibility for a website asks an AI platform for help in creating or improving the website.
-- A prospective customer of a web development service asks an AI platform to evaluate the quality of websites in the portfolios of candidate vendors.
-- A professional web developer within an organization asks an AI platform for a code review.
-- A risk-management professional within an organization asks an AI platform to report on any web accessibility defects that could expose the organization to prosecution or civil litigation.
-- An person who depends on web accessibility because of disabilities asks an AI platform to provide documentary support for a complaint to the owner of a website about accessibility defects.
-- A disability-rights advocate or attorney concerned with inaccessibility in a particular industry asks an AI platform to perform a front-end-quality comparison of some websites in that industry.
+1. A user of a web-builder platform with responsibility for a website asks an AI platform for help in creating or improving the website.
+1. A prospective customer of a web development service asks an AI platform to evaluate the quality of websites in the portfolios of candidate vendors.
+1. A professional web developer within an organization asks an AI platform for a code review.
+1. A risk-management professional within an organization asks an AI platform to report on any web accessibility defects that could expose the organization to prosecution or civil litigation for disability discrimination.
+1. A person who depends on web accessibility because of disabilities asks an AI platform to provide documentary support for a complaint to the owner of a website about accessibility defects.
+1. A disability-rights advocate or attorney concerned with inaccessibility in a particular industry asks an AI platform to perform a front-end-quality comparison of some websites in that industry.
 
-## Strategy
+Among these use cases, case 3 would make it feasible for the user to tell an AI platform explicitly and formally that Kilotest is an available connector to tools that are relevant to the task. All 5 of the other use cases would not make that feasible. In those 5 use cases, the user has a question but relies on the AI platform to know or discover which relevant connectors exist, to select appropriate connectors, and to provide language models that can use those connectors and tools that they connect to.
+
+Use cases 1, 2, 4, 5, and 6 exemplify a widespread expectation and demand for AI platform capability. The commonality is: “I have a question; answer it.” If Kilotest can be employed as an expert for AI platforms in relevant cases, platforms will be more successful in satisfying that demand. At present this is a difficult problem because of platform limitations and a lack of standardization.
 
-### Ease of use
+## Strategy
 
-Any use of Kilotest as a tool of language models will fail for some of the anticipated users if they are obligated to be aware of Kilotest and to translate that awareness into instructions or documentation. Therefore, for success in all the use cases, users should be able to tell models what is wanted and rely on models to figure out whether they need tools and, if so, which ones and how to use them.
+The Kilotest project is attempting to solve the just-described problem. That attempt follows the following strategy.
 
 ### Increments
 
-The standardization of tool discovery and utilization by and for language models and AI platforms is partial. Major differences in protocols exist among model and platform families. Therefore, small testable increments of improvement in the tool functionality of Kilotest can best be defined by model and platform family. For example, working on discoverability first and then on usability would not be effective, because both are complex and testability would be postponed until both were complete. Instead, a coherent model and platform family should be identified and any and all improvements to make use cases successful within that family should be made and tested, before development proceeds to the next family.
+Connector and tool discovery and utilization have been only partly standardized. Major differences in protocols exist among model and platform families. Therefore, small testable increments of improvement in the connector and tool functionality of Kilotest can best be defined by model and platform family and by use-case characteristics.
 
-One benefit of this type of incrementalism is that, after the first increment succeeds, it becomes possible to make and test external changes publicizing the fact that a particular platform-model combination delivers unprecedentedly comprehensive, truthful, and low-cost answers to questions about front-end web quality.
+One benefit of such incrementalism is that, after the first increment succeeds, it becomes possible to make and test external changes publicizing the fact that a particular platform-model combination delivers unprecedentedly inexpensive, comprehensive, and truthful answers in a particular class of use cases to questions about front-end web quality.
 
 Another benefit is that subsequent increments can be defined incrementally rather than in advance. Lessons learned from the work on each increment can inform the choice of what to work on next.
 
 #### Increment 1
 
-In the first increment, the objective is to make Kilotest automatically discovered and used by Anthropic Claude models when used within the Claude Desktop application. That investigation is under way. Results will be summarized here.
+In the first increment, the objective was to make Anthropic Claude models use Kilotest to help them answer questions from developers using the Claude Desktop application for use case 3, where the code in question is already deployed as a public web page.
+
+As mentioned above, Claude Desktop was installed on the local development host and then configured for Kilotest. That configuration consisted of the addition of a property to the Claude Desktop configuration file, located at `~/Library/Application Support/Claude/claude_desktop_config.json`. The added property is:
+
+```json
+"mcpServers": {
+  "kilotest": {
+    "command": "npx",
+    "args": [
+      "-y",
+      "@ivotoby/openapi-mcp-server",
+      "--disable-abbreviation",
+      "true"
+    ],
+    "env": {
+      "API_BASE_URL": "https://kilotest.com",
+      "OPENAPI_SPEC_PATH": "https://kilotest.com/openapi.yaml"
+    }
+  }
+},
+```
+
+That configuration notifies models that Kilotest is available as an MCP server, but does not explain what tools it provides, what they are useful for, or how to use them.
+
+Experimentation revealed that Claude Sonnet 4.6 with Medium effort chose to use Kilotest when answering questions about the accessibility and usability of particular public web pages. Low effort did not result in the use of Kilotest. The less capable Claude Haiku model did not use Kilotest.
+
+### Increment 2
+
+Increment 2 somewhat lightens the burden on the user by shifting the user environment from an installed application to a web browser, and by providing an easier interface for users to tell the platform about Kilotest. The platform is the `claude.ai` web application instead of Claude Desktop. Instead of editing a JSON file, the user tells the platform about Kilotest with a setting.
+
+The investigation began with the addition of Kilotest as a _connector_ to the `claude.ai` web application. The user used the `Customize/Connectors/Add connector/Add custom connector` interface, providing these data before activating the `Add` button:
+
+- Name: Kilotest
+- Remote MCP server URL: `https://kilotest.com/mcp`
+
+When the connector configuration with 4 endpoints appeared, the user changed the tool permissions for all 4 endpoints from `Needs approval` to `Always allow`.
+
+Then the user asked the platform, “Summarize the accessibility and usability of the home page of the website of Salesforce.”
+
+The results with the least powerful model were:
+
+- Claude Haiku 4.5 not extended: Summarized accessibility programs of Salesforce, instead of saying anything about its home page.
+- Claude Haiku 4.5 extended: Announced this thought process: “Deliberated between direct inspection and specialized accessibility testing tools”; then announced that it was using Kilotest, specifically the `summarize-accessibility-of-all-tested-web-pages` path; then stated: “Great! I've loaded the Kilotest tools. Now let me first check if Salesforce has already been tested by calling the summarize function. If not, I'll submit it for testing.”
+
+These results show that on the `claude.ai` platform, when configured with Kilotest as a connector, even Claude Haiku, in its Extended mode, can decide to use Kilotest when appropriate and can check for existing data before deciding to recommend that Kilotest test a page. This correct sequencing may be a result of the revised path descriptions in the `openapi.yaml` file recommending that sequencing.
+
+In the first run, Claude Haiku paused for about 5 minutes and then reported: “The Kilotest service seems to have timed out. Let me try a different approach - I'll fetch the Salesforce home page directly and evaluate it myself for accessibility and usability issues.”

package/IDEAS.md

@@ -1,4 +1,4 @@
-# development
+# ideas
 
 Notes about the further development of this project.
 

package/README.md

@@ -4,7 +4,7 @@ An ensemble testing service with a focus on accessibility
 
 ## Features
 
-This application uses an ensemble of 10 tools to test public web pages for standards conformity, usability, and accessibility.
+This application uses an ensemble of 10 rule engines to test public web pages for front-end quality (i.e. accessibility, usability, and conformity to standards).
 
 The testing paradigm employed by Kilotest is discussed in these papers:
 

package/SERVICE.md

@@ -239,7 +239,7 @@ In case Testaro again becomes a dependency of Kilotest, the notes below on secur
 
 ### Browser privileges
 
-Testaro uses Playwright to launch and control headless browsers, often `chromium`. Those browsers navigate to web pages that are tested by the tools that Testaro integrates. The `qualWeb` tool launches its own browser via Puppeteer as a dependency to perform its tests.
+Testaro uses Playwright to launch and control headless browsers, often `chromium`. Those browsers navigate to web pages that are tested by the rule engines that Testaro integrates. The `qualWeb` tool launches its own browser via Puppeteer as a dependency to perform its tests.
 
 When either Playwright or Puppeteer launches a `chromium` browser, in most environments it is [sandboxed](https://www.geeksforgeeks.org/ethical-hacking/what-is-browser-sandboxing/). Sandboxing is a security feature that prevents the browser from accessing potentially unsafe system resources. But in the Ubuntu Linux operating system that was installed on the Vultr Cloud Compute host a sandboxed browser requires an [unprivileged user namespace](https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces), and when Ubuntu was installed its configuration disallowed such namespaces. The file `/etc/sysctl.d/99-kilotest-userns.conf` with the content `kernel.apparmor_restrict_unprivileged_userns = 1` prohibited unprivileged user namespaces and thereby made sandboxed browsers unlaunchable.
 

package/diagnoses/index.html

@@ -37,7 +37,7 @@
         <li>Related WCAG standard: __wcag__</li>
       </ul>
       <h2>Diagnoses</h2>
-      <p>Here is how tools diagnose the <q>__issue__</q> issue for HTML element __catalogIndex__ of the __target__ page.</p>
+      <p>Here is how rule engines diagnose the <q>__issue__</q> issue for HTML element __catalogIndex__ of the __target__ page.</p>
       <ol>
         __diagnoses__
       </ol>

package/index.html

@@ -15,22 +15,22 @@
         "name": "Kilotest",
         "sameAs": "https://github.com/jrpool/kilotest",
         "datePublished": "2025-10-14",
-        "applicationCategory": "Web accessibility, usability, and standard conformity testing and reporting service",
+        "applicationCategory": "Front-end quality (i.e. web accessibility, usability, and standard conformity) testing and reporting service",
         "isAccessibleForFree": true,
-        "description": "Testing of web pages for accessibility, usability, and standard conformity with an ensemble of 10 tools and provision of results at selectable levels of detail.",
+        "description": "Testing of web pages for front-end quality (i.e. web accessibility, usability, and standard conformity) with an ensemble of 10 rule engines and provision of results at selectable levels of detail.",
         "softwareHelp": {
           "@type": "CreativeWork",
           "url": "https://kilotest.com/tutorial.html"
         },
         "featureList": [
-          "Automated testing of web accessibility, usability, and standard conformity",
+          "Automated testing of front-end quality (i.e. web accessibility, usability, and standard conformity)",
           "WCAG 2.2 conformance evaluation",
           "HTML5 conformance evaluation",
           "CSS3 conformance evaluation",
-          "Ensemble of 10 testing tools performing more than a thousand tests",
+          "Ensemble of 10 rule engines performing more than a thousand tests",
           "Multi-page, single-page, single-issue, and single-element reporting",
           "Priority-ranked issue lists",
-          "Public API for agent integration"
+          "Public API, connector, and tools for language-model integration"
         ],
         "url": "https://kilotest.com/",
         "maintainer": {
@@ -63,17 +63,17 @@
         <summary>More than a thousand? Really?</summary>
         <h2>Ensemble testing</h2>
         <p>Kilotest does <strong>ensemble testing</strong> of web pages for <a href="https://www.w3.org/mission/accessibility/">accessibility</a>, usability, and conformity to HTML standards.</p>
-        <p>Efficient testing <strong>first</strong> leverages <strong>automated tools</strong> and corrects the defects that they discover, <strong>then</strong> employs human testers.</p>
-        <p>Choosing an automated tool is hard, because they <a href="https://www.w3.org/WAI/test-evaluate/tools/list/">are numerous</a> and <a href="https://arxiv.org/pdf/2304.07591">vary in what and how they test</a>.</p>
-        <p><strong>Ensemble testing</strong>, instead of choosing a tool, combines multiple tools. It can discover more issues than single-tool testing, and the tools can check one another&rsquo;s quality.</p>
-        <h2>The tools</h2>
-        <p>The Kilotest ensemble is <a href="https://github.com/jrpool/testaro?tab=readme-ov-file#dependencies">10 tools</a> developed by teams in the USA, Canada, Denmark, Australia, and Portugal.</p>
-        <p>The tools perform about 1,300 tests in total. <a href="https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3">Open-source software</a> developed initially at CVS Health drives the tools, tries to weed out invalid results, and distills the findings into about 350 <q>issues</q>.</p>
+        <p>Efficient testing <strong>first</strong> leverages <strong>automated rule engines</strong> and corrects the defects that they discover, <strong>then</strong> employs human testers.</p>
+        <p>Choosing an automated rule engine is hard, because they <a href="https://www.w3.org/WAI/test-evaluate/tools/list/">are numerous</a> and <a href="https://arxiv.org/pdf/2304.07591">vary in what and how they test</a>.</p>
+        <p><strong>Ensemble testing</strong>, instead of choosing a rule engine, combines multiple rule engines. It can discover more issues than single-rule-engine testing, and the rule engines can check one another&rsquo;s quality.</p>
+        <h2>The rule engines</h2>
+        <p>The Kilotest ensemble is <a href="https://github.com/jrpool/testaro?tab=readme-ov-file#dependencies">10 rule engines</a> developed by teams in the USA, Canada, Denmark, Australia, and Portugal.</p>
+        <p>The rule engines perform about 1,300 tests in total. <a href="https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3">Open-source software</a> developed initially at CVS Health drives the rule engines, tries to weed out invalid results, and distills the findings into about 350 <q>issues</q>.</p>
         <h2>Status</h2>
         <p>Kilotest is a proof of concept, launched in October 2025 and being actively developed. Your suggestions, bug reports, and collaboration proposals are welcome! Post them on <a href="https://github.com/jrpool/kilotest/issues">GitHub</a> or email them to <a href="mailto:info@kilotest.com">info@kilotest.com</a>.</p>
         <p>In the current version, users can get results of already performed tests and can recommend new tests. Kilotest managers can approve recommendations.</p>
         <h2 id="a11y">Accessibility</h2>
-        <p>Kilotest intends to be accessible. At a minimum, Kilotest intends to conform to the rules of the tools in the Kilotest ensemble. Those rules enforce various standards and best practices, including many success criteria of conformance levels AA and AAA of the <a href="https://www.w3.org/WAI/standards-guidelines/wcag/">Web Content Accessibility Guidelines (WCAG)</a>.</p>
+        <p>Kilotest intends to be accessible. At a minimum, Kilotest intends to conform to the rules of the rule engines in the Kilotest ensemble. Those rules enforce various standards and best practices, including many success criteria of conformance levels AA and AAA of the <a href="https://www.w3.org/WAI/standards-guidelines/wcag/">Web Content Accessibility Guidelines (WCAG)</a>.</p>
         <p>Humans and automated agents encountering any accessibility barriers during the use of Kilotest are encouraged to report them on <a href="https://github.com/jrpool/kilotest/issues">GitHub</a> or by email to <a href="mailto:info@kilotest.com">info@kilotest.com</a>.</p>
       </details>
       <h2>What do you want to know?</h2>