@jrpool/kilotest 63.2.0 → 63.2.2

package/README.md

@@ -88,8 +88,8 @@ When you make a change in a feature branch **without** any API or tool contract
 
 When you **make an API or tool contract change** in a feature branch:
 
-1. `lhm.plugin.json`: Transform it to match `api/schemas.ts` and `mcp.ts`.
-2. `llms.txt`, `llms-full.txt`, `capability.md`, `README.md`: Update parameter and path names.
+1. `llms.txt`, `llms-full.txt`: Update narrative descriptions of the changed tools or parameters.
+2. `capability.md` and `README.md`: Update their prose to describe the changed behavior.
 3. Stage, commit, and optionally push all changes.
 4. Execute `npm run prePR -- <level>` (`major`, `minor`, or `patch`).
 5. Submit a pull request.

package/content.ts

@@ -1,6 +1,6 @@
 /*
   content.ts
-  Canonical prose describing what Kilotest is and does, reused verbatim by mcp.ts, scripts/generate-openapi.ts, and scripts/generate-lhm.ts, so the sentences are written once instead of being hand-copied into mcp.ts, lhm.plugin.json, llms.txt, and openapi.yaml.
+  Canonical prose describing what Kilotest is and does, reused verbatim by mcp.ts, scripts/generate-openapi.ts, scripts/generate-lhm.ts, and scripts/generate-llms.ts, so the sentences are written once instead of being hand-copied into mcp.ts, lhm.plugin.json, llms.txt, and openapi.yaml.
 */
 
 // CONSTANTS
@@ -8,5 +8,5 @@
 // Brief description of the Kilotest tool collection. Used as the MCP server description and as lhm.plugin.json’s top-level description. No length limit.
 export const collectionDescription = 'Tools that test web pages with an ensemble of 12 rule engines for front-end quality (accessibility, usability, and W3C web standards, including HTML, CSS, ARIA, and WCAG)';
 
-// Description of the Kilotest domain and data hierarchy. Used as (1) the server instructions, (2) the opening paragraph of llms.txt, (3) the start of the openapi.yaml info.description, and (4) the getKilotestOverview tool description (but not part of the other tool descriptions, to avoid diluting how models read each tool), because some MCP clients do not propagate server-level instructions or the server description to their models.
+// Description of the Kilotest domain and data hierarchy. Used as (1) the McpServer instructions, which is the only place most tools' own descriptions (everything but getKilotestOverview) get this context from, since repeating it in each of their descriptions would dilute how models read each tool, (2) the first paragraph of the kilotestOverview prose that mcp.ts returns from the getKilotestOverview tool and the kilotest-overview resource, (3) the opening paragraph of llms.txt, (4) the start of the openapi.yaml info.description, and (5) the getKilotestOverview tool's own description and the kilotest-overview resource's own description, because some MCP clients do not propagate the McpServer instructions to their models, so getKilotestOverview and the kilotest-overview resource (unlike the other tools) carry the context redundantly, in both their own description and their returned body.
 export const sharedContext = 'Kilotest integrates an ensemble of 12 rule engines, with about 1,300 rules, to test web pages for front-end quality (accessibility, usability, and W3C web standards, including HTML, CSS, ARIA, and WCAG). Each test job generates a standardized report describing which elements violated which rules and grouping the violations into about 380 issues. Kilotest can list the tested pages, the reports about each page, the issues in each report, the violators of each issue, and the diagnoses of each violation. Kilotest also accepts requests to test pages not yet tested and to retest already tested pages.';

package/docs/AI-TOOLS.md

@@ -97,8 +97,10 @@ A change to the tool or API contract (tool names, parameter names, schemas, resp
 - `openapi-3.0.yaml`: an OpenAPI 3.0.3 conversion of `openapi.yaml`, regenerated by the same hook, for registries whose importers reject 3.1 documents; never edit it by hand.
 - `server.json`: contains only the server descriptor (name, description, repository, remotes, version), no tool schemas. Its version is synced by `update-versions.ts`; edit it manually only if the server description or remote URL changes.
 - `lhm.plugin.json`: embeds every tool’s input and output schema and its descriptions, in the LobeHub manifest format. Its `description` and every tool’s `description`, `inputSchema`, `outputSchema`, and `annotations` are regenerated from `mcp.ts`/`api/schemas.ts` by `scripts/generate-lhm.ts` (run by `npm run generate`, part of `npm run prePR`); never edit those fields by hand. The remaining fields (`author`, `cloudEndpoint`, `homepage`, `identifier`, `name`, `prompts`, `resources`, `tags`, `version`, and each tool’s `execution` block) have no canonical source elsewhere and are still edited manually. See `docs/decisions/010-content-generation.md`.
-- `content.ts`: canonical prose shared verbatim by `mcp.ts`, `scripts/generate-openapi.ts`, and `scripts/generate-lhm.ts`. Also hand-check `llms.txt`’s opening paragraph, which is supposed to match `content.ts`’s `sharedContext` verbatim but is not generated.
+- `content.ts`: canonical prose shared verbatim by `mcp.ts`, `scripts/generate-openapi.ts`, `scripts/generate-lhm.ts`, and `scripts/generate-llms.ts` (which regenerates `llms.txt`'s opening paragraph from `sharedContext`); all four are run by `npm run generate`, so no hand-checking is needed there. `scripts/check-docs.ts` (also run by `npm run lint`) separately verifies that `llms.txt` and `llms-full.txt`'s quoted `/api/...` paths and parameters still match `api/routes.ts`.
 - `llms-full.txt`, `capability.md`, `README.md`, `docs/openai-plugin/skills/kilotest-overview/SKILL.md`: hand-maintained, intentionally paraphrased rather than copied verbatim; search them for stale parameter and path names after a contract change.
+- `docs/openai-plugin/plugin.json`: hand-maintained, submitted to OpenAI's plugin review (see "Registrations" above); its `review.test_cases[].tools_triggered` fields name actual tools verbatim (e.g. `"listPages, listIssues"`), so check them, not just the paraphrased prose fields, after a contract change. Its `version` and `release_notes` track revisions to this plugin submission (the manifest, the skill, and the test cases) and are unrelated to `package.json`'s version; `update-versions.ts` does not touch this file, so bump `version` and add a `release_notes` entry by hand whenever the submission changes.
+- `docs/openai-plugin/mcp.json`: hand-maintained, but contains only the server's type and URL, so a contract change never requires editing it.
 
 #### External registries that update themselves, needing no maintainer action
 

package/lhm.plugin.json

@@ -1,7 +1,7 @@
 {
+  "_generated": "Do not edit the _generated, version, description, or tools property. They are automatically generated from content.ts, api/schemas.ts/mcp.ts, scripts/generate-lhm.ts, and scripts/update-versions.ts.",
   "author": "Jonathan Robert Pool",
   "cloudEndpoint": "https://kilotest.com/mcp",
-  "description": "Tools that test web pages with an ensemble of 12 rule engines for front-end quality (accessibility, usability, and W3C web standards, including HTML, CSS, ARIA, and WCAG)",
   "homepage": "https://github.com/jrpool/kilotest",
   "identifier": "jrpool-kilotest",
   "name": "Kilotest",
@@ -18,6 +18,8 @@
     "wcag",
     "web standards"
   ],
+  "version": "63.2.2",
+  "description": "Tools that test web pages with an ensemble of 12 rule engines for front-end quality (accessibility, usability, and W3C web standards, including HTML, CSS, ARIA, and WCAG)",
   "tools": [
     {
       "name": "getKilotestOverview",
@@ -4860,6 +4862,5 @@
         "taskSupport": "forbidden"
       }
     }
-  ],
-  "version": "63.2.0"
+  ]
 }

package/llms-full.txt

@@ -101,6 +101,13 @@ Kilotest offers these tools:
     - `issueID`: identifier of the issue
     - `reportID`: identifier of the report, with YYMMDDTHHmm format for its initial segment (example: 260503T0432-x9z)
   - source of parameters: response to a `listViolators` request
+- `getReport`
+  - method: `GET`
+  - purpose: get one full report in JSON
+  - path: `/api/getReport/{reportID}`
+  - parameters
+    - `reportID`: identifier of the report, with YYMMDDTHHmm format for its initial segment (example: 260503T0432-x9z)
+  - source of parameters: response to a `listIssues` request
 - `requestNewTest`
   - method: `POST`
   - purpose: request testing of one web page about which no report is available

package/llms.txt

@@ -1,6 +1,6 @@
 # Kilotest
 
-Kilotest integrates an ensemble of 12 rule engines, with about 1,300 rules, to test web pages for front-end quality (accessibility, usability, and web standards, including HTML, CSS, ARIA, and WCAG). Each test job generates a standardized report describing which elements violated which rules and grouping the violations into about 380 issues. Kilotest can list the tested pages, the reports about each page, the issues in each report, the violators of each issue, and the diagnoses of each violation. Kilotest also accepts requests to test pages not yet tested and to retest already tested pages. Kilotest features are available via a web UI, an API, and an MCP server with streamable HTTP and no authentication.
+Kilotest integrates an ensemble of 12 rule engines, with about 1,300 rules, to test web pages for front-end quality (accessibility, usability, and W3C web standards, including HTML, CSS, ARIA, and WCAG). Each test job generates a standardized report describing which elements violated which rules and grouping the violations into about 380 issues. Kilotest can list the tested pages, the reports about each page, the issues in each report, the violators of each issue, and the diagnoses of each violation. Kilotest also accepts requests to test pages not yet tested and to retest already tested pages. Kilotest features are available via a web UI, an API, and an MCP server with streamable HTTP and no authentication.
 
 ## API
 

package/mcp.ts

@@ -57,8 +57,7 @@ export const mcpPath = '/mcp';
 // Detailed overview of the Kilotest domain, data hierarchy, and tool workflow, returned by
 // the getKilotestOverview tool and served as the docs://kilotest/overview resource.
 const kilotestOverview = [
-  'Kilotest tests web pages for front-end quality: accessibility, usability, and web standards, including HTML, CSS, ARIA, and WCAG.',
-  'Kilotest integrates an ensemble of twelve independent rule engines to test a web page and stores the results as a structured report.',
+  sharedContext,
   'Results are organized as a hierarchy: tested pages; for each tested page, reports from the tests of the page; for each report, issues reported; for each issue, violators of rules belonging to the issue (a violator is an element or the whole page); for each violator, diagnoses by rule engines of its violations.',
   'Typical workflow: call the listPages tool to learn whether the page has been tested. If it has, drill down from its latest report with the listIssues, listViolators, and listDiagnoses tools, or retrieve the entire report with the getReport tool. The listIssues output also lists all reports about the same page, so you can examine earlier reports. If the page has not been tested, request a test with the requestNewTest tool or order a test with the orderNewTest tool. If the latest report is obsolete, request a retest with the requestRetest tool or order a retest with the orderRetest tool.',
   'The request tools submit requests for manager approval. The order tools obtain automated review and approval or rejection. Approval produces a report identifier and an opportunity to wait for completion. To wait, call the awaitTest tool with that identifier; it blocks until the report is ready, the job fails, or a maximum wait time elapses. To check later instead, call the listIssues tool with the same identifier.',

package/openapi-3.0.yaml

@@ -2,7 +2,7 @@ openapi: 3.0.3
 info:
   title: Kilotest API
   description: 'Kilotest integrates an ensemble of 12 rule engines, with about 1,300 rules, to test web pages for front-end quality (accessibility, usability, and W3C web standards, including HTML, CSS, ARIA, and WCAG). Each test job generates a standardized report describing which elements violated which rules and grouping the violations into about 380 issues. Kilotest can list the tested pages, the reports about each page, the issues in each report, the violators of each issue, and the diagnoses of each violation. Kilotest also accepts requests to test pages not yet tested and to retest already tested pages. Kilotest acts as a web server for human users, an MCP server for AI platforms, and an API for programmatic access. With this API you can request that a page be tested or retested, discover available reports, and incrementally retrieve facts from a report: from what reports are available, to what issues were reported in one report, to what elements violated one issue, to what diagnoses rule engines gave for one violation.'
-  version: 63.2.0
+  version: 63.2.2
   license:
     name: MIT
     url: https://github.com/jrpool/kilotest/blob/main/LICENSE

package/openapi.yaml

@@ -2,7 +2,7 @@ openapi: 3.1.0
 info:
   title: Kilotest API
   description: 'Kilotest integrates an ensemble of 12 rule engines, with about 1,300 rules, to test web pages for front-end quality (accessibility, usability, and W3C web standards, including HTML, CSS, ARIA, and WCAG). Each test job generates a standardized report describing which elements violated which rules and grouping the violations into about 380 issues. Kilotest can list the tested pages, the reports about each page, the issues in each report, the violators of each issue, and the diagnoses of each violation. Kilotest also accepts requests to test pages not yet tested and to retest already tested pages. Kilotest acts as a web server for human users, an MCP server for AI platforms, and an API for programmatic access. With this API you can request that a page be tested or retested, discover available reports, and incrementally retrieve facts from a report: from what reports are available, to what issues were reported in one report, to what elements violated one issue, to what diagnoses rule engines gave for one violation.'
-  version: 63.2.0
+  version: 63.2.2
   license:
     name: MIT
     identifier: MIT

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jrpool/kilotest",
-  "version": "63.2.0",
+  "version": "63.2.2",
   "description": "Ensemble testing for front-end web quality",
   "type": "module",
   "main": "index.ts",
@@ -10,11 +10,11 @@
   "scripts": {
     "start": "node index.ts",
     "typecheck": "tsc --noEmit",
-    "lint": "npx eslint && npx htmlhint web error.html && out=$(npx markdownlint-cli2 2>&1); code=$?; if [ $code -ne 0 ]; then echo \"$out\"; exit $code; fi",
+    "lint": "npx eslint && npx htmlhint web error.html && out=$(npx markdownlint-cli2 2>&1); code=$?; if [ $code -ne 0 ]; then echo \"$out\"; exit $code; fi && node scripts/check-docs.ts",
     "test": "c8 node --require ./test/setup.ts --experimental-test-module-mocks --test --test-concurrency=1 --test-randomize '**/*.test.ts'",
     "test:smoke": "node smokeTest.ts",
     "checkinfo": "node checkinfo.ts",
-    "generate": "node scripts/generate-openapi.ts && node scripts/generate-lhm.ts && node scripts/update-versions.ts",
+    "generate": "node scripts/generate-openapi.ts && node scripts/generate-lhm.ts && node scripts/generate-llms.ts && node scripts/update-versions.ts",
     "build:media": "node scripts/buildDemoVideo.ts",
     "publish:lhm": "npx -y @lobehub/market-cli plugin update --dir .",
     "prePR": "node scripts/prePR.ts",

package/scripts/check-docs.ts

@@ -0,0 +1,59 @@
+/*
+  check-docs.ts
+  Verifies that the `/api/...` path templates (including path-parameter names and order)
+  quoted in llms.txt and llms-full.txt match api/routes.ts, the single source of truth for
+  API methods and paths. Fails with a listing of mismatches if any API path in those files is
+  missing, stale, or renders a route not present in api/routes.ts, and fails if any route in
+  api/routes.ts is not documented in a file. Run via `npm run lint`.
+*/
+
+// IMPORTS
+
+import fs from 'node:fs';
+import path from 'node:path';
+import {routes} from '../api/routes.ts';
+
+// CONSTANTS
+
+const rootDir = path.join(import.meta.dirname, '..');
+const targetFiles = ['llms.txt', 'llms-full.txt'];
+const apiPathPattern = /\/api\/[A-Za-z]+(?:\/\{[A-Za-z]+\})*/g;
+const expectedPaths = new Set(routes.map(route => route.path));
+
+// FUNCTIONS
+
+// Returns the operationId of a path template, i.e. the segment right after "/api/".
+const operationIdOf = (apiPath: string): string => apiPath.split('/')[2] ?? '';
+
+// EXECUTION
+
+const errors: string[] = [];
+for (const fileName of targetFiles) {
+  const filePath = path.join(rootDir, fileName);
+  const content = fs.readFileSync(filePath, 'utf8');
+  const found = new Set(content.match(apiPathPattern) ?? []);
+  for (const apiPath of found) {
+    if (!expectedPaths.has(apiPath)) {
+      const operationId = operationIdOf(apiPath);
+      const route = routes.find(candidate => candidate.operationId === operationId);
+      errors.push(
+        route
+          ? `${fileName}: "${apiPath}" does not match api/routes.ts, which defines ${operationId} as "${route.path}"`
+          : `${fileName}: "${apiPath}" does not match any operationId in api/routes.ts`
+      );
+    }
+  }
+  for (const route of routes) {
+    if (!found.has(route.path)) {
+      errors.push(`${fileName}: missing the path "${route.path}" documented in api/routes.ts for ${route.operationId}`);
+    }
+  }
+}
+if (errors.length) {
+  console.error('Documentation path/parameter mismatches found:');
+  for (const error of errors) {
+    console.error(`  ${error}`);
+  }
+  process.exit(1);
+}
+console.log(`API paths in ${targetFiles.join(' and ')} match api/routes.ts.`);

package/scripts/generate-lhm.ts

@@ -2,11 +2,13 @@
   generate-lhm.ts
   Regenerates the description and tool entries (description, inputSchema, outputSchema,
   annotations) of lhm.plugin.json from the single sources of truth: content.ts for prose and
-  api/schemas.ts (via the same registrations mcp.ts makes) for schemas. Everything else in
-  lhm.plugin.json (author, cloudEndpoint, homepage, identifier, name, prompts, resources,
-  tags, version, and each tool's execution block) is left untouched, because it has no
-  canonical source elsewhere in the repository; version is kept in sync separately by
-  update-versions.ts.
+  api/schemas.ts (via the same registrations mcp.ts makes) for schemas, and (re)writes a
+  "_generated" warning property so editors of the raw JSON see that those fields are
+  overwritten on the next generation. version has its own generator, update-versions.ts, which
+  copies it from package.json. Every remaining property (author, cloudEndpoint, homepage,
+  identifier, name, prompts, resources, tags, and each tool's execution block) has no
+  generator anywhere in the repository, so it is left untouched here and is meant to be
+  edited by hand.
 */
 
 // IMPORTS
@@ -50,10 +52,10 @@ type ZodRawShape = Record<string, z.ZodTypeAny>;
 
 const outputPath = path.join(import.meta.dirname, '..', 'lhm.plugin.json');
 const jsonSchemaOptions = {target: 'draft-7' as const, unrepresentable: 'any' as const};
+// Edit warning (disregarded on publication per https://market.lobehub.com/s/publish-mcp/references/manifest.md).
+const generatedWarning = 'Do not edit the _generated, version, description, or tools property. They are automatically generated from content.ts, api/schemas.ts/mcp.ts, scripts/generate-lhm.ts, and scripts/update-versions.ts.';
 
-// The input and output schemas mcp.ts passes to registerTool for each tool, keyed by tool
-// name, matching mcp.ts's own registerTool calls exactly (mcp.ts remains the source for
-// descriptions and annotations, read at runtime below via createMCPServer).
+// The input and output schemas mcp.ts passes to registerTool for each tool, keyed by tool name, matching mcp.ts's own registerTool calls (mcp.ts remains the source for descriptions and annotations, read at runtime below via createMCPServer).
 const toolSchemas: Record<string, {input: ZodRawShape; output?: z.ZodTypeAny}> = {
   getKilotestOverview: {input: {}},
   listPages: {input: {}, output: listPagesResponseSchema},
@@ -72,16 +74,14 @@ const toolSchemas: Record<string, {input: ZodRawShape; output?: z.ZodTypeAny}> =
 
 // FUNCTIONS
 
-// Builds the lhm.plugin.json inputSchema for a tool: like outputSchema below, but LobeHub's
-// existing input schemas never carry additionalProperties, so that key is dropped.
+// Builds the lhm.plugin.json inputSchema for a tool: like outputSchema below, but LobeHub’sexisting input schemas never carry additionalProperties, so that key is dropped.
 const buildInputSchema = (shape: ZodRawShape): Record<string, unknown> => {
   const generated = z.toJSONSchema(z.object(shape), jsonSchemaOptions) as Record<string, unknown>;
   delete generated.additionalProperties;
   const {$schema, ...withoutSchemaKeyword} = generated;
   return {...withoutSchemaKeyword, $schema};
 };
-// Builds the lhm.plugin.json outputSchema for a tool, reordering keys to match the file's
-// existing convention ($schema placed just before additionalProperties and definitions).
+// Builds the lhm.plugin.json outputSchema for a tool, reordering keys to match the file’s existing convention ($schema placed just before additionalProperties and definitions).
 const buildOutputSchema = (schema: z.ZodTypeAny): Record<string, unknown> => {
   const generated = z.toJSONSchema(schema, jsonSchemaOptions) as Record<string, unknown>;
   const {$schema, additionalProperties, definitions, ...rest} = generated;
@@ -97,7 +97,10 @@ const buildOutputSchema = (schema: z.ZodTypeAny): Record<string, unknown> => {
 
 const server = createMCPServer();
 const registeredTools = (server as unknown as {_registeredTools: Record<string, {description: string; annotations: unknown}>})._registeredTools;
-const manifest = JSON.parse(fs.readFileSync(outputPath, 'utf8'));
+const parsedManifest = JSON.parse(fs.readFileSync(outputPath, 'utf8'));
+// Places "_generated" first regardless of where (or whether) it already appears, so it is the first thing a reader of the raw JSON sees. The explicit reassignment after the spread is required even though "_generated" is already a key: were the input file's own (possibly stale) "_generated" value to spread in after it, the spread’s last-write-wins rule would silently keep that stale value in place, even though the key's position stays first either way. Reassigning the other keys below does not move their position, since they already exist on this object.
+const manifest = {_generated: generatedWarning, ...parsedManifest};
+manifest._generated = generatedWarning;
 manifest.description = collectionDescription;
 manifest.tools = manifest.tools.map((tool: {name: string; execution: unknown}) => {
   const registered = registeredTools[tool.name];

package/scripts/generate-llms.ts

@@ -0,0 +1,34 @@
+/*
+  generate-llms.ts
+  Regenerates the opening paragraph of llms.txt from content.ts's sharedContext, the single
+  source of truth content.ts already documents that paragraph as reusing (along with the MCP
+  server instructions, the openapi.yaml description, and the getKilotestOverview tool
+  description). Leaves the rest of llms.txt untouched, because it has no canonical source
+  elsewhere in the repository.
+*/
+
+// IMPORTS
+
+import fs from 'node:fs';
+import path from 'node:path';
+import {sharedContext} from '../content.ts';
+
+// CONSTANTS
+
+const outputPath = path.join(import.meta.dirname, '..', 'llms.txt');
+// Text appended after sharedContext to form the full opening paragraph; unlike sharedContext,
+// this sentence is specific to llms.txt (it is not reused anywhere else), so it lives here
+// rather than in content.ts.
+const openingSuffix = ' Kilotest features are available via a web UI, an API, and an MCP server with streamable HTTP and no authentication.';
+const openingParagraph = `${sharedContext}${openingSuffix}`;
+
+// EXECUTION
+
+const lines = fs.readFileSync(outputPath, 'utf8').split('\n');
+const paragraphIndex = lines.findIndex(line => line.startsWith('Kilotest integrates an ensemble of'));
+if (paragraphIndex === -1) {
+  throw new Error(`Could not find the opening paragraph to regenerate in ${outputPath}`);
+}
+lines[paragraphIndex] = openingParagraph;
+fs.writeFileSync(outputPath, lines.join('\n'));
+console.log(`Wrote ${outputPath}`);

package/scripts/prePR.ts

@@ -1,6 +1,6 @@
 /*
   prePR.ts
-  Prepares a release pull request on the current feature branch: verifies a clean working tree, bumps the package version, regenerates the generated files (openapi.yaml, openapi-3.0.yaml, server.json, and lhm.plugin.json), commits and pushes the result, records the branch name for postPR.ts, and prints the remaining manual steps. The generation is run explicitly because the project's .npmrc sets ignore-scripts=true, which disables the npm "version" lifecycle hook that would otherwise run it.
+  Prepares a release pull request on the current feature branch: verifies a clean working tree, bumps the package version, regenerates the generated files (openapi.yaml, openapi-3.0.yaml, lhm.plugin.json, the opening paragraph of llms.txt, server.json, and the version field of lhm.plugin.json), commits and pushes the result, records the branch name for postPR.ts, and prints the remaining manual steps. The generation is run explicitly because the project's .npmrc sets ignore-scripts=true, which disables the npm "version" lifecycle hook that would otherwise run it.
 */
 
 // IMPORTS

package/server.json

@@ -6,7 +6,7 @@
     "url": "https://github.com/jrpool/kilotest",
     "source": "github"
   },
-  "version": "63.2.0",
+  "version": "63.2.2",
   "remotes": [
     {
       "type": "streamable-http",