@jrpool/kilotest 46.0.11 → 46.1.0

package/llms-full.txt

@@ -22,10 +22,10 @@ To provide a more inexpensive, thorough, and accurate assessment, the agent coul
 
 The tool that tests web pages for front-end quality has attributes that no other such tool has. It:
 
-- selects an ensemble of 10 rule engines that implement tests for, in total, more than a thousand rules for front-end quality
+- selects an ensemble of 12 rule engines that implement tests for, in total, more than a thousand rules for front-end quality
 - runs the tests of the rule engines
 - combines the reports of the rule engines into a single integrated report
-- consolidates the 1000+ rules having 10 different naming systems into 300+ “issues” with a uniform naming system
+- consolidates the 1000+ rules having 12 different naming systems into 300+ “issues” with a uniform naming system
 
 The tools that report test results extract from reports statistics and details at various levels of granularity, as required by an agent to fulfill particular user requests.
 
@@ -46,9 +46,9 @@ The rule engines in the Kilotest ensemble take responsibility for determining wh
 
 ## Kilotest versus other MCP servers
 
-In the marketplace for MCP servers that test for front-end quality, all servers other than that of Kilotest leverage at most 1 rule engine, most often `axe-core`. The Kilotest testing tool leverages `axe-core` plus 9 other rule engines. This has these main effects:
+In the marketplace for MCP servers that test for front-end quality, all servers other than that of Kilotest leverage at most 1 rule engine, most often `axe-core`. The Kilotest testing tool leverages `axe-core` plus 11 other rule engines. This has these main effects:
 
-- For web pages that have not yet been tested, a single-rule-engine MCP can provide faster results than Kilotest. A tool running only the `axe-core` tests can complete its work in about 5 seconds or less. The testing tool of Kilotest usually completes its work in 2 to 3 minutes, because all 10 rule engines are run, and some of the tests involve navigation and interaction with the page and LLM inference. Moreover, Kilotest currently allows users (both humans and models) to **recommend** new pages for testing but authorizes only Kilotest managers to make the tool proceed with the testing. The wait for manager action can take up to a day. A feature permitting immediate testing ordered by AI agents is planned, but, until it is implemented, Kilotest will be useful for not-yet-tested pages only in long-running (multi-day) workflows.
+- For web pages that have not yet been tested, a single-rule-engine MCP can provide faster results than Kilotest. A tool running only the `axe-core` tests can complete its work in about 5 seconds or less. The testing tool of Kilotest usually completes its work in 2 to 3 minutes, because all 12 rule engines are run, and some of the tests involve navigation and interaction with the page and LLM inference. Moreover, Kilotest currently allows users (both humans and models) to **recommend** new pages for testing but authorizes only Kilotest managers to make the tool proceed with the testing. The wait for manager action can take up to a day. A feature permitting immediate testing ordered by AI agents is planned, but, until it is implemented, Kilotest will be useful for not-yet-tested pages only in long-running (multi-day) workflows.
 - For web pages about which reports are already available, Kilotest can provide faster results than a single-rule-engine MCP server, because Kilotest stores test results for subsequent retrieval. A retrieval from a Kilotest reporting tool can be completed in less than 2 seconds.
 - Kilotest results are more comprehensive than single-rule-engine MCP server results. Every rule engine provides limited coverage of front-end quality, so false negatives (missed defects) are more common with single-rule-engine MCP servers. This difference [has been documented in research](https://arxiv.org/pdf/2304.07591).
 
@@ -165,7 +165,6 @@ More information about Kilotest features and internals:
 
 - [Kilotest as connector to and collection of AI Tools](https://github.com/jrpool/kilotest/blob/main/docs/AI-TOOLS.md)
 - [OpenAPI specification for Kilotest](https://kilotest.com/openapi.yaml)
-- [Tool caller for testing](https://github.com/jrpool/kilotest/blob/main/api/testSuite.js)
 
 ### General
 
@@ -173,7 +172,8 @@ More information about Kilotest features and internals:
 - [Tutorial, “Accessibility testing strategies”](https://kilotest.com/tutorial.html)
 - [Tutorial on authorizing language models to use Kilotest](https://kilotest.com/qai/)
 - [README file](https://github.com/jrpool/kilotest/blob/main/README.md)
-- [Testaro rule-engine orchestrator](https://github.com/jrpool/testaro)
-- [Testaro rule classifier](https://github.com/jrpool/testaro-issues)
+- [Summary of Kilotest capabilities](https://kilotest.com/capability.md)
+- [Testaro rule-engine orchestrator](https://github.com/YRA-Tech/testaro)
+- [Testaro rule classifier](https://github.com/YRA-Tech/testaro-issues)
 - [Research report on Testaro](https://arxiv.org/abs/2309.10167)
 - [Blog post on the Kilotest use case](https://medium.com/cvs-health-tech-blog/how-to-run-a-thousand-accessibility-tests-63692ad120c3)

package/llms.txt

@@ -1,6 +1,6 @@
 # Kilotest
 
-Kilotest helps language models and AI platforms to give thorough, accurate and truthful answers when users ask about the front-end quality (i.e. accessibility, usability, and standards conformity) of web pages. Kilotest integrates an ensemble often independent rule engines to test any public web page. Kilotest creates a structured report of the results and saves the report for future reference. Kilotest API endpoints and MCP tools can provide multipage, single-page, and element-level test results.
+Kilotest helps language models and AI platforms to give thorough, accurate and truthful answers when users ask about the front-end quality (i.e. accessibility, usability, and standards conformity) of web pages. Kilotest integrates an ensemble of twelve independent rule engines to test any public web page. Kilotest creates a structured report of the results and saves the report for future reference. Kilotest API endpoints and MCP tools can provide multipage, single-page, and element-level test results.
 
 ## API
 
@@ -33,6 +33,7 @@ Kilotest helps language models and AI platforms to give thorough, accurate and t
 ## More documents
 
 - [Detailed explanation of Kilotest and its capabilities for LLMs](https://kilotest.com/llms-full.txt)
+- [Summary of Kilotest capabilities](https://kilotest.com/capability.md)
 - [OpenAPI specification for Kilotest](https://kilotest.com/openapi.yaml)
 - [Kilotest as connector to and collection of AI Tools](https://github.com/jrpool/kilotest/blob/main/docs/AI-TOOLS.md)
 - [Tutorial on authorizing language models to use Kilotest](https://kilotest.com/qai/)

package/openapi.yaml

@@ -1,7 +1,7 @@
 openapi: 3.1.0
 info:
   title: Kilotest API
-  description: 'Kilotest is an ensemble testing and reporting service for front-end web quality. It uses an ensemble of 10 rule engines to test public web pages for front-end quality (accessibility, usability, and standards conformity). Kilotest acts as a web server for human users, an MCP server for AI platforms, and an API for programmatic access. With this API you can request that a page be tested or retested, discover available reports, incrementally retrieve facts from a report: from what reports are available, to what issues were reported in one report, to what elements violated one issue, to what diagnoses rule engines gave for one violation.'
+  description: 'Kilotest is an ensemble testing and reporting service for front-end web quality. It uses an ensemble of 12 rule engines to test public web pages for front-end quality (accessibility, usability, and standards conformity). Kilotest acts as a web server for human users, an MCP server for AI platforms, and an API for programmatic access. With this API you can request that a page be tested or retested, discover available reports, incrementally retrieve facts from a report: from what reports are available, to what issues were reported in one report, to what elements violated one issue, to what diagnoses rule engines gave for one violation.'
   version: 2.1.0
   license:
     name: MIT

package/package.json

@@ -1,16 +1,17 @@
 {
   "name": "@jrpool/kilotest",
-  "version": "46.0.11",
+  "version": "46.1.0",
   "description": "Ensemble testing for front-end web quality",
   "main": "index.js",
   "engines": {
-    "node": ">=22.12.0"
+    "node": ">=24.16.0"
   },
   "scripts": {
     "start": "node index",
-    "lint": "npx eslint && npx markdownlint-cli2",
-    "test": "c8 node --require ./test/setup.cjs --test --test-concurrency=1 --test-timeout=70",
+    "lint": "npx eslint && out=$(npx markdownlint-cli2 2>&1); code=$?; if [ $code -ne 0 ]; then echo \"$out\"; exit $code; fi",
+    "test": "c8 node --require ./test/setup.cjs --test --test-concurrency=1 --test-timeout=100 --test-randomize",
     "test:smoke": "node smokeTest",
+    "checkinfo": "node checkinfo",
     "generate:openapi": "node scripts/generate-openapi"
   },
   "repository": {

package/scripts/generate-openapi.js

@@ -58,7 +58,7 @@ const generate = () => {
     openapi: '3.1.0',
     info: {
       title: 'Kilotest API',
-      description: 'Kilotest is an ensemble testing and reporting service for front-end web quality. It uses an ensemble of 10 rule engines to test public web pages for front-end quality (accessibility, usability, and standards conformity). Kilotest acts as a web server for human users, an MCP server for AI platforms, and an API for programmatic access. With this API you can request that a page be tested or retested, discover available reports, incrementally retrieve facts from a report: from what reports are available, to what issues were reported in one report, to what elements violated one issue, to what diagnoses rule engines gave for one violation.',
+      description: 'Kilotest is an ensemble testing and reporting service for front-end web quality. It uses an ensemble of 12 rule engines to test public web pages for front-end quality (accessibility, usability, and standards conformity). Kilotest acts as a web server for human users, an MCP server for AI platforms, and an API for programmatic access. With this API you can request that a page be tested or retested, discover available reports, incrementally retrieve facts from a report: from what reports are available, to what issues were reported in one report, to what elements violated one issue, to what diagnoses rule engines gave for one violation.',
       version,
       license: {
         name: license,

package/smokeTest.js

@@ -14,50 +14,53 @@ const host = process.env.SMOKE_HOST || 'kilotest.com';
 // Concrete paths matching each wildcard pattern in the routes table, for smoke testing.
 const concretePaths = {
   GET: {
-    '/mcp': '/mcp',
+    '*.html*': '/listReports.html',
     '/': '/',
-    '/index.html': '/index.html',
-    '/robots.txt': '/robots.txt',
-    '/openapi.yaml': '/openapi.yaml',
-    '/openapi.json': '/openapi.json',
-    '/swagger.yaml': '/swagger.yaml',
-    '/swagger.json': '/swagger.json',
     '/api-docs': '/api-docs',
-    '/llms.txt': '/llms.txt',
+    '/api/*': '/api/listReports',
+    '/capability.md': '/capability.md',
+    '/favicon.*': '/favicon.ico',
+    '/fullReport.json/*': '/fullReport.json/260101T0000/mix',
+    '/index.html': '/index.html',
     '/llms-full.txt': '/llms-full.txt',
+    '/llms.txt': '/llms.txt',
+    '/mcp': '/mcp',
+    '/openapi.json': '/openapi.json',
+    '/openapi.yaml': '/openapi.yaml',
+    '/robots.txt': '/robots.txt',
     '/sitemap.xml': '/sitemap.xml',
     '/style.css': '/style.css',
-    '/fullReport.json/*': '/fullReport.json/260101T0000/mix',
-    '/api/*': '/api/listReports',
-    '/tutorial/images/*': '/tutorial/images/diagram.png',
-    '/favicon.*': '/favicon.ico',
-    '*.html*': '/listReports.html'
+    '/swagger.json': '/swagger.json',
+    '/swagger.yaml': '/swagger.yaml',
+    '/tutorial/images/*': '/tutorial/images/diagram.png'
   },
   POST: {
+    '/api/*': '/api/requestFeature',
     '/mcp': '/mcp',
-    '/requestTest.html': '/requestTest.html',
-    '/requestRetest.html/*': '/requestRetest.html/260101T0001/ct',
-    '/recAction.html': '/recAction.html',
     '/reannotate.html': '/reannotate.html',
+    '/recAction.html': '/recAction.html',
     '/renewWCAG.html': '/renewWCAG.html',
+    '/requestRetest.html/*': '/requestRetest.html/260101T0001/ct',
+    '/requestTest.html': '/requestTest.html',
+    '/tutorialComment.html': '/tutorialComment.html',
     '/worker/job': '/worker/job',
-    '/worker/report': '/worker/report',
-    '/api/*': '/api/requestFeature',
-    '/tutorialComment.html': '/tutorialComment.html'
+    '/worker/report': '/worker/report'
   }
 };
 // Minimal POST bodies for paths that require them.
 const postBodies = {
   '/api/requestFeature': {feature: 'smoke test'},
-  '/requestTest.html': {what: 'Smoke Test Page', url: 'https://smoketest.example.com', why: 'smoke test'},
-  '/requestRetest.html/260101T0001/ct': {why: 'smoke test'},
-  '/recAction.html': {target: 'https://smoketest.example.com\tSmoke Test Page', authCode: 'invalid'},
+  '/mcp': {},
   '/reannotate.html': {authCode: 'invalid'},
+  '/recAction.html': {
+    target: 'https://smoketest.example.com\tSmoke Test Page', authCode: 'invalid'
+  },
   '/renewWCAG.html': {authCode: 'invalid'},
-  '/worker/job': {},
-  '/worker/report': {},
+  '/requestRetest.html/260101T0001/ct': {why: 'smoke test'},
+  '/requestTest.html': {what: 'Smoke Test Page', url: 'https://smoketest.example.com', why: 'smoke test'},
   '/tutorialComment.html': {content: 'smoke test'},
-  '/mcp': {}
+  '/worker/job': {},
+  '/worker/report': {}
 };
 
 // FUNCTIONS

package/util.test.js

@@ -398,7 +398,7 @@ test('processTestRequest returns an error for a mismatched directory name', asyn
   assert.equal(result.message, 'Invalid recommendation');
 });
 
-test('processTestRequest succeeds and populates the template for a valid request', async () => {
+test('processTestRequest succeeds and populates the template for a valid request', {timeout: 500}, async () => {
   // Reset recs.json to empty before the test.
   await fs.writeFile(recsPath(), '{}\n');
   const result = await processTestRequest('test', requestTestDir, 'Example Page', 'https://example.com', 'because accessibility');
@@ -409,7 +409,7 @@ test('processTestRequest succeeds and populates the template for a valid request
   await fs.writeFile(recsPath(), '{}\n');
 });
 
-test('processTestRequest returns a duplicate error for a repeated request', async () => {
+test('processTestRequest returns a duplicate error for a repeated request', {timeout: 500}, async () => {
   // Reset recs.json to empty, then make a successful request.
   await fs.writeFile(recsPath(), '{}\n');
   await processTestRequest('test', requestTestDir, 'Example Page', 'https://example.com', 'because accessibility');

package/web/listReports/listReports.test.js

@@ -88,7 +88,7 @@ test('listReports includes a link to request testing a new page', async () => {
   assert.ok(testLink);
 });
 
-test('listReports shows recommendations when recs.json has entries', async () => {
+test('listReports shows recommendations when recs.json has entries', {timeout: 500}, async () => {
   const dbDir = path.join(__dirname, '..', '..', 'test', 'fixtures', 'db');
   const recsPath = path.join(dbDir, 'jobs', 'recs.json');
   const originalRecs = await fs.readFile(recsPath, 'utf8');

package/web/reannotate/reannotate.test.js

@@ -87,7 +87,7 @@ test('answer returns an error when no reports are available', async () => {
   }
 });
 
-test('answer returns an error when a report fails annotation', async () => {
+test('answer returns an error when a report fails annotation', {timeout: 2000}, async () => {
   // Back up all fixture reports, because annotateReport modifies good reports
   // in place before it reaches the bad one and returns an error.
   const backups = await backupReports();
@@ -122,7 +122,7 @@ test('answer returns an error when a report fails annotation', async () => {
   }
 });
 
-test('answer returns ok with an answer page when all reports annotate successfully', async () => {
+test('answer returns ok with an answer page when all reports annotate successfully', {timeout: 2000}, async () => {
   // Back up all fixture reports, because annotateReport modifies them in place.
   const backups = await backupReports();
   try {

package/web/requestTest/requestTest.test.js

@@ -61,7 +61,7 @@ after(async () => {
 
 // TESTS
 
-test('answer returns ok with a populated answer page for a valid test request', async () => {
+test('answer returns ok with a populated answer page for a valid test request', {timeout: 500}, async () => {
   // Require the module after DB_DIR is set.
   const {answer} = require('./index');
   const result = await answer(
@@ -81,7 +81,7 @@ test('answer returns ok with a populated answer page for a valid test request',
   assert.ok(h1.textContent.includes('Test Page'));
 });
 
-test('answer returns an error when the URL is already queued', async () => {
+test('answer returns an error when the URL is already queued', {timeout: 500}, async () => {
   const {answer} = require('./index');
   // Create a queued job with the target URL.
   const queueDir = path.join(fixtureDBDir, 'jobs', 'queue');
@@ -107,7 +107,7 @@ test('answer returns an error when the URL is already queued', async () => {
   }
 });
 
-test('answer returns an error for an invalid recommendation', async () => {
+test('answer returns an error for an invalid recommendation', {timeout: 500}, async () => {
   const {answer} = require('./index');
   const result = await answer('', 'not-a-url', 'why');
   assert.equal(result.status, 'error');

package/web/rewindReportsForm/rewindReportsForm.test.js

@@ -72,7 +72,7 @@ test('rewindReportsForm deletes a latest superseding report with valid auth code
   }
 });
 
-test('rewindReportsForm returns an error when a report file is corrupt', async () => {
+test('rewindReportsForm returns an error when a report file is corrupt', {timeout: 500}, async () => {
   const reportPath = path.join(reportsPath(), '260101T0001-ct.json');
   const backup = await fs.readFile(reportPath, 'utf8');
   try {

package/web/tutorial/index.html

@@ -334,7 +334,7 @@
         <section id="testing-page">
           <h3>Testing the page</h3>
           <h4>By rule engines</h4>
-          <p>What do rule engines find about this issue when they test this page? For a quick answer, Kilotest subjected the page to testing by all 10 rule engines in its ensemble. Two of the rule engines reported rule violations belonging to this issue:</p>
+          <p>What do rule engines find about this issue when they test this page? For a quick answer, Kilotest subjected the page to testing by the 10 rule engines that were then in its ensemble (since then, 2 more were added, so now there are 12). Two of the rule engines reported rule violations belonging to this issue:</p>
           <ul>
             <li>The Testaro rule engine reported, <q>input has no autocomplete="email" attribute</q>. In other words, Testaro recognized this input as asking for the user&rsquo;s email address and therefore as requiring an <code>autocomplete="email"</code> attribute, but not having one.</li>
             <li>The HTML CodeSniffer rule engine, referencing this <code>form</code> element, reported, <q>Invalid autocomplete value: new-password. Element does not belong to Password control group.</q>. In other words, HTML CodeSniffer recognized that this form is not a password input (a form is not an input at all!) and therefore is not eligible for an <code>autocomplete="new-password"</code> attribute, but has one.</li>
@@ -346,7 +346,7 @@
           </ul>
           <p>You might think that this shows rule engines were effective in revealing the defects related to this issue, but this is not a success story. Yes, two rule engines found related defects, but the other eight rule engines did not. Inspection of those eight rule engines reveals that six of them include rules enforcing this standard in some way. So, did these six rule engines report false negatives? Did they fail to enforce their applicable rules here? Perhaps, but perhaps not. Suppose a rule engine has a rule that requires the value of any <code>autocomplete</code> attribute to be in the list of allowed values. Well, <code>new-password</code> is in the list. So that rule engine would, properly, not report a violation of <strong>that</strong> rule.</p>
           <p>We can clearly see from this example that no rule engine will find all accessibility defects, not even all defects that <strong>can</strong> be detected by rule engines. That is why Kilotest uses an <strong>ensemble</strong> of rule engines.</p>
-          <p>Of course, running ten rule engines also creates a risk of false positives reported by ten rule engines. In this particular case, there were none. For example, no rule engine claimed to find a defect related to this issue someplace else on the page. Kilotest, like any platform that integrates multiple rule engines, faces greater risks of false positives than a single rule engine would. It combats false positives by investigating user complaints and deprecating rules whose tests are found faulty.</p>
+          <p>Of course, running twelve rule engines also creates a risk of false positives reported by twelve rule engines. In this particular case, there were none. For example, no rule engine claimed to find a defect related to this issue someplace else on the page. Kilotest, like any platform that integrates multiple rule engines, faces greater risks of false positives than a single rule engine would. It combats false positives by investigating user complaints and deprecating rules whose tests are found faulty.</p>
           <h4>By AI agents</h4>
           <p>An AI agent powered by a frontier model was asked to visit the page and report any accessibility defect related to <code>autocomplete</code> attributes. The agent worked for about 3 minutes and then reported a missing <code>autocomplete="email"</code> attribute in the newsletter subscription form. But the page has no newsletter subscription form! It has a form to request a guide, not to subscribe to anything. Moreover, the agent provided code to support its conclusion, and the code differed pervasively from the actual code on the page. This response was a complete fabrication. The agent confessed to it, explaining that it resorted to hallucination when its security restrictions prevented it from accessing the necessary page data.</p>
           <p>A similar AI agent returned correct diagnoses when it was fed the page as an HTML document. That worked, because the two defects relevant to this practicum are discoverable from the HTML of the rendered page. Any defects arising from stylesheets or scripts would have been missed.</p>
@@ -418,7 +418,7 @@
               <legend>Choose the best answer</legend>
               <p>In this practicum, which rule engines discovered both <code>autocomplete</code> defects?</p>
               <ol>
-                <li><label><input type="radio" name="q4" value="a"> All 10 rule engines</label></li>
+                <li><label><input type="radio" name="q4" value="a"> All 12 rule engines</label></li>
                 <li><label><input type="radio" name="q4" value="b"> Six of the rule engines</label></li>
                 <li><label><input type="radio" name="q4" value="c"> Only Testaro and HTML CodeSniffer</label></li>
                 <li><label><input type="radio" name="q4" value="d"> None of the rule engines</label></li>

package/web/tutorial/tutorial.test.js

@@ -59,7 +59,7 @@ test('handleComment returns an error for content that is empty after sanitizatio
   assert.equal(result.message, 'Comment is empty after sanitization');
 });
 
-test('handleComment saves a sanitized comment and returns ok', async () => {
+test('handleComment saves a sanitized comment and returns ok', {timeout: 500}, async () => {
   // Replace comments with an empty array for a clean test.
   await fs.writeFile(commentsPath, '[]\n');
   const result = await handleComment('This is a <b>test</b> comment.');
@@ -70,7 +70,7 @@ test('handleComment saves a sanitized comment and returns ok', async () => {
   assert.ok(comments[0].timeStamp);
 });
 
-test('handleComment strips HTML tags and control characters', async () => {
+test('handleComment strips HTML tags and control characters', {timeout: 500}, async () => {
   await fs.writeFile(commentsPath, '[]\n');
   const result = await handleComment('<img src=x onerror=alert(1)>\x00\x07Hello');
   assert.equal(result.status, 'ok');
@@ -78,7 +78,7 @@ test('handleComment strips HTML tags and control characters', async () => {
   assert.equal(comments[0].content, 'Hello');
 });
 
-test('handleComment truncates content to 500 characters', async () => {
+test('handleComment truncates content to 500 characters', {timeout: 500}, async () => {
   await fs.writeFile(commentsPath, '[]\n');
   const longComment = 'x'.repeat(600);
   const result = await handleComment(longComment);
@@ -87,7 +87,7 @@ test('handleComment truncates content to 500 characters', async () => {
   assert.equal(comments[0].content.length, 500);
 });
 
-test('handleComment creates comments.json when it does not exist', async () => {
+test('handleComment creates comments.json when it does not exist', {timeout: 500}, async () => {
   await fs.unlink(commentsPath).catch(() => {});
   const result = await handleComment('Test comment for missing file');
   assert.equal(result.status, 'ok');