@jrpool/kilotest 36.1.1 → 37.0.0

package/{AI-TOOL.md → docs/AI-TOOL.md}

@@ -185,4 +185,4 @@ The [Perplexity documentation](https://www.perplexity.ai/help-center/en/articles
 
 Once the compatibility of Kilotest as a connector with AI platforms of two different vendors had been confirmed, the focus of work returned to completion of the missing endpoints of the API. The first was an endpoint that would let a model efficiently search for an available report about a particular web page.
 
-The new `identifyReportsAboutMatchingWebPages` tool provides inputs for fragments of a description and of a URL hostname of a web page and responds with summary data about all and only the available reports of pages that match at least one of those fragments, where matching means either including or being included by the provided fragment, case-insensitively.
+The new `summarizeQualityOfMatchingWebPages` tool provides inputs for fragments of a description and of a URL hostname of a web page and responds with summary data about all and only the available reports of pages that match at least one of those fragments, where matching means either including or being included by the provided fragment, case-insensitively.

package/docs/decisions/001-api-design.md

@@ -0,0 +1,110 @@
+---
+date: 2026-07-09
+status: proposed
+---
+
+# API design
+
+## Context and problem
+
+The UI of Kilotest offers human non-manager users opportunities to influence and to interrogate the data managed by the application. The data are organized into reports. Each report contains the results of one session in which 10 rule engines were used for testing the front-end quality of one web page.
+
+The opportunities to interrogate the data in the UI are organized as a hierarchy of granularity:
+
+1. All available reports
+2. All issues from one report
+3. All violators of one issue from one report
+4. All diagnoses for one violator of one issue from one report
+
+A violator is a particular element (HTML, SVG, etc.) on the page that some rule engine reported having violated a particular rule of the rule engine. If the rule classifier used by Kilotest classifies that rule as belonging to a particular issue, then that element is referred to as a violator of that issue.
+
+This hierarchical design is based on a realistic assumption about how human users interact with Kilotest. They visit its home page and begin what to them is a session. Although requests to Kilotest are stateless, the response at each level includes links that contain all the specifications required for a request at the next level. For example, the response to a request at level 3 includes, for each violator, a link that makes a request at level 4 and includes identifiers of the page, the issue and the violator.
+
+The responses to requests in the UI at any level include a summary of some of the information about the previous levels. For example, a response at level 3 includes a summary of information about the report and about the issues in that report. The prior-level summarized information is based on assumptions about what a typical human user would value. For example, at level 3, the prior-level information does not summarize all the available reports, but does include a list of the rule engines that succeeded and failed in testing the page and the counts of issues reported.
+
+These assumptions for human UI users may or may not be valid for how LLMs interact with the Kilotest API. Those interactions may vary depending on the tasks that they are performing. Consider three scenarios:
+
+1. A user asks an LLM for all available information about the front-end quality of a particular web page.
+2. A user asks an LLM for all the available information about issues related to a particular topic, such as keyboard navigation, for a particular web page.
+3. A user asks an LLM to summarize the front-end quality of a particular web page; having consumed the summary, asks the LLM for a summary of information about one of the reported issues; having consumed that summary, asks the LLM for a summary of information about one of the reported violators of that issue; and, having consumed that summary, asks the LLM for a summary of information about the diagnoses for that violator of that issue.
+
+Scenario 1 seems to justify an API that can aggregate all the information about the latest report about the page into a single response.
+
+Scenario 2 seems to justify an API that offers enough information about all the available reports and all the defined issues to allow the LLM to make a request at level 3 for a particular issue in a particular report.
+
+Scenario 3 seems to justify an API that mimics the UI in structure, but perhaps not in the prior-levels information provided at each level.
+
+Even these apparently justified API designs depend on the assumption that users mean what they say, particularly in scenario 1. A user who asks for **all** available information about the front-end quality of a particular web page may do so because of ignorance about how voluminous that information is, and the real wish may be more conditional: all the information if there is not much, but otherwise a summary so I can select what I want to learn more about.
+
+Uncertainty about how LLMs will interact with the API creates a design problem. How should the API be further developed?
+
+As of the date of this decision, a partial initial API version already exists: an API that mimics the UI for levels 1, 2, and 3. For each level, the MCP server provides a tool.
+
+## Considered options
+
+1. **Iterate and instrument.** (1) Add a tool that implements level 4; then (2) add a tool that asks LLMs to comment on how the API design could better satisfy their needs; then (3) add metrics to make API usage and submitted LLM comments available to Kilotest managers.
+2. **Research first.** (1) Pause further API development while doing fundamental research to predict LLM needs; then (2) resume API and metrics development, including API redesign if the research results warrant.
+
+## Decision
+
+The decision is to adopt **option 1: iterate and instrument**.
+
+## Empirical basis
+
+The decision is based on the fact that, even without a pause for fundamental research, there is enough empirical evidence to justify this decision.
+
+### Completion of the 4-level hierarchy
+
+There is empirical support for an API that assumes scenario 3, i.e. incremental rather than one-shot interrogation. Both single-shot comprehensive requests and incremental drill-downs occur in significant numbers in real-world LLM use.
+
+- Jian et al. (2024) studied experienced ChatGPT users and found that **43.8% of prompts were single-component**, **38.2% were dual-component**, and only **18.0% were multi-component**. A request such as scenario 1 ("tell me everything about page X") or scenario 2 ("tell me about defect class X on page Y") is structurally common because it is typically a single-component prompt.
+- At the same time, follow-up refinement is routine. Mysore et al. (2025) analyzed large-scale Bing Copilot and WildChat writing sessions and report: “Rather than passively accepting output, users actively refine, explore, and co-construct text.” This matches scenario 3.
+
+For scenario 3, current LLM tool-use patterns do not require the model to restart at level 1. Anthropic's tool-use documentation describes an agentic loop in which the full conversation history is kept and “every turn sees the complete prior context”. Cohere’s multi-step tool-use documentation explains that the model can use information learned from one tool call to parameterize a subsequent call. MCP’s typed tool schemas support this pattern. Therefore the existing hierarchy can be expected to be practical in scenario 3.
+
+### Usage and suggestion metrics
+
+There is also empirical support for modesty in any estimate of the adequacy of the current or any other single API design. As found by Zhu et al. (2025), “users do not transfer existing conversational norms to LLM interactions, but instead develop new registers with distinct pragmatic features.” Metrics will help to inform the contributors to the API about how LLMs are actually using it and how they want it to be improved.
+
+## Cost and context-window constraints
+
+A user-friendly rendering of an entire Kilotest report is estimated at about 5 MB of text, or roughly 1.25 million tokens. This exceeds the 200,000-token context window of Claude 3.5 Sonnet, so the model cannot consume the whole report in one prompt. If it could be passed whole, each pass would cost roughly $3.75 in input tokens at current pricing, and every subsequent question about it would require re-sending it. By contrast, a four-level drill-down through report → issue → violator → diagnosis costs roughly $0.07 for a typical path. An intuitive guess is that a typical conversation would involve 1 report, 2 issues, and 3 violators per issue. Delivering such results with the full-report method would require 2 issue passes and 6 violator passes beyond the initial pass, for a total of about $33.75. Doing so incrementally would cost about $0.26, which is only **1/130th** of the cost of the same result with the full-report strategy. Even if a user initially wanted a comprehensive report and then wanted an LLM to find and interpret it, the impact of that strategy on cost would likely motivate the user to switch to an incremental strategy.
+
+## Sub-decision: ancestor information in level responses
+
+Given this decision, a secondary decision to make is what information from prior levels to repeat in a response. Three alternatives were considered:
+
+1. **Identifiers only:** The report, issue, and violator would be described by their unique identifiers only. The LLM would use the identifiers to retrieve from its conversation-history context any information needed about them.
+2. **Identifiers plus labels:** The report, issue, and violator would be described by their unique identifiers and human-interpretable labels. If the LLM needs more information about them than what their labels provide, the LLM would use the identifiers to retrieve from its conversation-history context that additional information.
+3. **Cumulative responses:** All the information from the prior responses would be repeated, so that any needed information would be available in the current response and no search of the conversation history would be needed.
+
+The sub-decision was **identifiers plus labels** (option 2). This decision was based on cost and reliability, as follows.
+
+- **Cost:** Option 3 would probably make a four-level drill-down several times as expensive as option 2, because of the large response volumes.
+- **Reliability:** Option 1 would entail a greater risk of information loss and hallucination than option 2, because the LLM could be unable to retrieve the needed information using identifiers if its history were truncated, if a tool were invoked in isolation, or if the model were asked later to summarize findings in natural language.
+
+## Implementation plan
+
+- The existing MCP tools for levels 1, 2, and 3 will be revised to remove excess information about prior levels and include only identifiers and labels for them.
+- Level 4 will be implemented as an MCP tool that returns the diagnoses for one violator of one issue in one report.
+- A suggestions tool will be provided so that LLM-based clients can report friction or suggest improvements to the API design.
+- Usage metrics will be defined, captured, and recorded so that Kilotest managers can observe which levels, issues, and patterns are most common and what suggestions have been received.
+- If the empirical data later show that the API does not serve some common scenario well, the API will be redesigned accordingly.
+
+### Risks and mitigations
+
+- **Risk:** The hierarchy may not match the way LLMs actually want to query the data.
+  - **Mitigation:** The suggestions tool and usage metrics will surface mismatches early.
+- **Risk:** The cost estimate assumes a well-behaved client that preserves conversation history; a poorly implemented client could still incur higher-than-necessary costs.
+  - **Mitigation:** MCP clients are expected to maintain message history by protocol; the ancestor labels provide a minimal human-friendly substitute.
+- **Risk:** Scenario 1 users may still want an aggregated "everything about page X" response.
+  - **Mitigation:** A future tool can be added to aggregate the hierarchy on the server side and return a structured summary, but that decision is deferred until usage data justify it.
+
+## References
+
+- Jian, J., et al. (2024). [Understanding User Prompting Behavior in Generative AI](https://dl.acm.org/doi/abs/10.1002/pra2.1318).
+- Mysore, S., et al. (2025). [Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wild](https://aclanthology.org/2025.emnlp-main.852/).
+- Zhu, S., et al. (2025). [Show or Tell? Modeling the evolution of request-making in human-LLM conversations](https://arxiv.org/html/2508.01213v2).
+- Anthropic. [Tutorial: Build a tool-using agent](https://platform.claude.com/docs/en/agents-and-tools/tool-use/build-a-tool-using-agent).
+- Cohere. [Multi-step Tool Use](https://docs.cohere.com/v1/docs/multi-step-tool-use).
+- [Model Context Protocol specification](https://modelcontextprotocol.io/specification/2025-06-18/server/tools).

package/index.js

@@ -342,10 +342,10 @@ const requestHandler = async (request, response) => {
     // Otherwise, if it is for an API service:
     else if (pageName === 'api') {
       const [service,  ... specs] = pathTail.split('/');
-      // If the service is provision of facts about the available reports:
-      if (service === 'targets') {
+      // If the service is provision of a list of all available reports:
+      if (service === 'reportList') {
         // Get the response (potentially error) data.
-        const responseData = await require(path.join(__dirname, 'targets', 'api')).response(specs);
+        const responseData = await require(path.join(__dirname, 'reportList', 'api')).response(specs);
         // Send them.
         setHeaders('application/json', null, 'ultra');
         response.end(JSON.stringify(responseData));
@@ -702,7 +702,7 @@ const requestHandler = async (request, response) => {
         // Otherwise, if the first segment is the report finding service:
         else if (segments[0] === 'target') {
           const {what = '', hostname = ''} = postData;
-          // If the payload contains a description fragment or URL fragment:
+          // If the payload contains a description fragment or hostname fragment:
           if (what || hostname) {
             // Process the request and get the response data.
             const responseData = await require(path.join(__dirname, 'target', 'api'))

package/llms-full.txt

@@ -82,10 +82,10 @@ The implementation is currently in an alpha phase and offers 4 tools:
   - payload properties
     - `what`: all or part of a description of the page
     - `hostname`: all or part of the hostname of the page
-- `summarizeQualityOfAllTestedWebPages`
+- `getListOfAllAvailableReports`
   - method: `GET`
-  - purpose: summarize front-end quality test results from all available reports
-  - path: `/api/targets`
+  - purpose: get a list of all available reports
+  - path: `/api/reportList`
   - caveat: Cached results are likely to be stale and should not be used
 - `describeQualityOfOneWebPage`
   - method: `GET`
@@ -94,7 +94,7 @@ The implementation is currently in an alpha phase and offers 4 tools:
   - parameters
     - `timeStamp`: initial segment of report identifier
     - `jobID`: final segment of report identifier
-  - source of parameters: response to a `summarizeQualityOfAllTestedWebPages` request
+  - source of parameters: response to a `getListOfAllAvailableReports` request
 - `describeOneIssueFromOneReport`
   - method: `GET`
   - purpose: provide details about one issue from one report
@@ -156,7 +156,7 @@ More information about Kilotest features and internals:
 
 ### For language models and AI platforms
 
-- [AI-TOOL file](https://github.com/jrpool/kilotest/blob/main/AI-TOOL.md)
+- [AI-TOOL file](https://github.com/jrpool/kilotest/blob/main/docs/AI-TOOL.md)
 - [OpenAPI specification for Kilotest](https://kilotest.com/openapi.yaml)
 - [Tool caller for testing](https://github.com/jrpool/kilotest/blob/main/researchAgent.js)
 

package/llms.txt

@@ -8,8 +8,8 @@ There is a [detailed explanation of Kilotest and its capabilities for LLMs](http
 
 ## API
 
-- [Find any available reports about a specified web page](/api/identifyReportsAboutMatchingWebPages)
-- [Get summaries of all available reports](/api/targets)
+- [Get summaries of reports about a specified web page](/api/target)
+- [Get a list of all available reports](/api/reportList)
 - Get summarized test results from one report: `/api/reportIssues/{timeStamp}/{jobID}`
 - Get details about one issue from one report: `/api/reportIssue/{issueID}/{timeStamp}/{jobID}`
 - Recommend testing of a web page: `/api/testRecForm` (requires `POST` request)

package/mcp.js

@@ -10,7 +10,7 @@ const {StreamableHTTPServerTransport} = require('@modelcontextprotocol/sdk/serve
 const {z} = require('zod');
 const {isReportAvailable, isURL} = require('./util');
 const targetAPI = require('./target/api');
-const targetsAPI = require('./targets/api');
+const reportListAPI = require('./reportList/api');
 const reportIssuesAPI = require('./reportIssues/api');
 const reportIssueAPI = require('./reportIssue/api');
 const testRecFormAPI = require('./testRecForm/api');
@@ -27,10 +27,10 @@ const createMCPServer = () => {
   server.registerTool(
     'summarizeQualityOfMatchingWebPages',
     {
-      description: 'Returns summary data from every available Kilotest report about the front-end quality (i.e. accessibility, usability, and standards conformity) of web pages that match the description or hostname fragment of a web page that you have provided. Matching is case-insensitive and succeeds if the page property either is included by or includes the specified fragmant. Before calling recommendQualityTestingOfOneWebPage and waiting for testing to occur, call this tool to check whether a report about the page is available now.',
+      description: 'Returns summary data from every available Kilotest report about the front-end quality (i.e. accessibility, usability, and standards conformity) of web pages that match the description or hostname fragment of a web page that you have provided. Matching is case-insensitive and succeeds if the page property either is included by or includes the specified fragment.',
       inputSchema: {
         what: z.string().describe('All or part of a description of the web page.'),
-        url: z.string().describe('All or part of the hostname of the URL of the web page.')
+        hostname: z.string().describe('All or part of the hostname of the URL of the web page.')
       },
       annotations: {
         title: 'Summarize the quality of matching web pages',
@@ -40,18 +40,18 @@ const createMCPServer = () => {
         openWorldHint: false
       }
     },
-    async ({what, url}) => {
-      const result = await targetAPI.response([what, url]);
+    async ({what, hostname}) => {
+      const result = await targetAPI.response([what, hostname]);
       return {content: [{type: 'text', text: JSON.stringify(result)}]};
     }
   );
   server.registerTool(
-    'summarizeQualityOfAllTestedWebPages',
+    'getListOfAllAvailableReports',
     {
-      description: 'Returns summary data from every available Kilotest report about the front-end quality (i.e. accessibility, usability, and standards conformity) of a web page.',
+      description: 'Returns a list of all available Kilotest reports; each report describes the results of a job that tested one web page for front-end quality (i.e. accessibility, usability, and standards conformity).',
       inputSchema: {},
       annotations: {
-        title: 'Summarize the quality of all tested web pages',
+        title: 'Get a list of all available reports',
         readOnlyHint: true,
         idempotentHint: true,
         destructiveHint: false,
@@ -59,14 +59,14 @@ const createMCPServer = () => {
       }
     },
     async () => {
-      const result = await targetsAPI.response();
+      const result = await reportListAPI.response();
       return {content: [{type: 'text', text: JSON.stringify(result)}]};
     }
   );
   server.registerTool(
     'describeQualityOfOneWebPage',
     {
-      description: 'Returns data from a specified Kilotest report about issues for the front-end quality (i.e. accessibility, usability, and standards conformity) of a web page. The required timeStamp and jobID parameters identify the report and are obtained from a summarizeQualityOfAllTestedWebPages response.',
+      description: 'Returns data from a specified Kilotest report about issues for the front-end quality (i.e. accessibility, usability, and standards conformity) of a web page. The required timeStamp and jobID parameters identify the report and are obtained from a getListOfAllAvailableReports response.',
       inputSchema: {
         timeStamp: z.string().describe('Report timestamp in YYMMDDTHHMM format, e.g. 260503T0432'),
         jobID: z.string().describe('Job identifier, e.g. x9z')
@@ -109,9 +109,9 @@ const createMCPServer = () => {
   server.registerTool(
     'recommendQualityTestingOfOneWebPage',
     {
-      description: 'Recommends a web page for Kilotest to test for front-end quality (i.e. accessibility, usability, and standards conformity). Do not call this tool until after you call summarizeQualityOfAllTestedWebPages to check whether a report about the page, or a related page that satisfies your requirements, is available.',
+      description: 'Recommends a web page for Kilotest to test for front-end quality (i.e. accessibility, usability, and standards conformity). Do not call this tool until after you call getListOfAllAvailableReports to check whether a report about the page, or a related page that satisfies your requirements, is available and to understand the naming conventions for pages.',
       inputSchema: {
-        what: z.string().describe('Short description of the page, following the naming conventions visible in the summarizeQualityOfAllTestedWebPages response'),
+        what: z.string().describe('Short description of the page, following the naming conventions visible in the getListOfAllAvailableReports response'),
         url: z.string().describe('Full HTTPS URL of the page to test'),
         why: z.string().describe('Reason for recommending this page for testing')
       },

package/openapi.yaml

@@ -17,7 +17,7 @@ paths:
     post:
       operationId: summarizeQualityOfMatchingWebPages
       summary: Summarize the quality of matching web pages
-      description: Get summary data from the available reports about web pages whose descriptions or hostnames match your specifications. Matching is case-insensitive and succeeds at least 1 specification is included by or includes the corresponding property of a tested page. The data include the name and URL of each tested web page, when the testing was performed, how many front-end quality (i.e. accessibility, usability, and standards conformity) issues were reported, and URLs for retrieving more detailed data from the report. Before calling the recommendQualityTestingOfOneWebPage tool and waiting for testing to occur, call this tool to check whether a report about the page is available now. If so, you can retrieve data from it. If not, you can use the recommendQualityTestingOfOneWebPage tool to recommend the page for testing.
+      description: Get summary data from the available reports about web pages whose descriptions or hostnames match your specifications. Matching is case-insensitive and succeeds if at least 1 specification is included by or includes the corresponding property of a tested page. The data include the name and URL of each tested web page, when the testing was performed, how many front-end quality (i.e. accessibility, usability, and standards conformity) issues were reported, and URLs for retrieving more detailed data from the report.
       requestBody:
         description: Page specifications
         required: true
@@ -33,24 +33,24 @@ paths:
               schema:
                 $ref: '#/components/schemas/TargetsResponse'
 
-  /api/targets:
+  /api/reportList:
     get:
-      operationId: summarizeQualityOfAllTestedWebPages
-      summary: Summarize the quality of all tested web pages
-      description: Get summary data about every non-hidden report available from Kilotest, including the name and URL of the tested web page, when the testing was performed, how many front-end quality (i.e. accessibility, usability, and standard-conformity) issues were reported, and URLs for retrieving more detailed data from the report.
+      operationId: getListOfAllAvailableReports
+      summary: Get a list of all available reports
+      description: Get a list of all available reports. Each report describes the results of a job that tested a web page for front-end quality (i.e. accessibility, usability, and standards conformity). For each report, the list identifies which page was tested and when and provides URLs to begin incremental retrieval of the test results.
       responses:
         '200':
-          description: Summaries of available reports
+          description: List of all available reports
           content:
             application/json:
               schema:
-                $ref: '#/components/schemas/TargetsResponse'
+                $ref: '#/components/schemas/ReportListResponse'
 
   /api/testRecForm:
     post:
       operationId: recommendQualityTestingOfOneWebPage
       summary: Recommend quality testing of one web page
-      description: Submit a recommendation for Kilotest to test, for the first time, a particular web page for front-end quality (i.e. accessibility, usability, and standards conformity). Recommendations are typically approved and the testing completed within a day, whereupon the results can be found with the summarizeQualityOfAllTestedWebPages operation. Before submitting a recommendation, use the summarizeQualityOfAllTestedWebPages operation to ensure that no report about the page is available, and also to see the stylistic rules for the naming of pages. An attempt to recommend a page for testing will fail if a report about the page is already available.
+      description: Submit a recommendation for Kilotest to test, for the first time, a particular web page for front-end quality (i.e. accessibility, usability, and standards conformity). Recommendations are typically approved and the testing completed within a day, whereupon the results can be found with the getListOfAllAvailableReports operation. Before submitting a recommendation, use the getListOfAllAvailableReports operation to ensure that no report about the page is available, and also to see the stylistic rules for the naming of pages. An attempt to recommend a page for testing will fail if a report about the page is already available.
       requestBody:
         description: Test recommendation specifications
         required: true
@@ -70,7 +70,7 @@ paths:
     get:
       operationId: describeQualityOfOneWebPage
       summary: Describe quality of one web page
-      description: Get data about the quality of a specified Kilotest report. The data about each issue include its priority, the rule engines that reported it, the number of HTML elements exhibiting it, and URLs for retrieving element-level detail. The timeStamp and jobID parameters identify the report and are available in the response from the summarizeQualityOfAllTestedWebPages operation.
+      description: Get data about the quality of a specified Kilotest report. The data about each issue include its priority, the rule engines that reported it, the number of HTML elements exhibiting it, and URLs for retrieving element-level detail. The timeStamp and jobID parameters identify the report and are available in the response from the getListOfAllAvailableReports operation.
       parameters:
         - name: timeStamp
           in: path
@@ -146,10 +146,14 @@ components:
         summary:
           type: string
           description: Natural-language description of this response and of Kilotest. Provides context for an agent encountering Kilotest for the first time.
-        tool name:
+        tool collection name:
           type: string
           examples:
             - Kilotest
+        tool name:
+          type: string
+          examples:
+            - getListOfAllAvailableReports
         request:
           type: object
           properties:
@@ -158,6 +162,8 @@ components:
               properties:
                 identifier:
                   type: string
+                  examples:
+                    - reportList
                 description:
                   type: string
             method:
@@ -167,7 +173,7 @@ components:
                 - POST
             payload:
               type: object
-              description: Present only for POST requests. The request payload.
+              description: The request payload, if method type is POST, or otherwise omitted.
             URLs:
               type: object
               properties:
@@ -197,6 +203,7 @@ components:
               description: Unique identifier for this response instance.
             date and time:
               type: string
+              description: UTC date and time when the response was generated, in ISO 8601 format.
               format: date-time
 
     TestRecForm:
@@ -306,12 +313,44 @@ components:
           $ref: '#/components/schemas/RuleEnginesSummary'
         rule engines that reported issues:
           $ref: '#/components/schemas/RuleEnginesSummary'
-        URLs for getting data on the reported issues:
+        URLs for getting data about the reported issues:
           $ref: '#/components/schemas/NextTierURLs'
         URL for getting the full technical report as JSON:
           type: string
           format: uri
 
+    ReportListItem:
+      type: object
+      description: Summary data about a single available report.
+      properties:
+        identifier:
+          type: string
+          examples:
+            - 260504T1659-029
+        creation date and time:
+          type: string
+          format: date-time
+        days since the creation date:
+          type: integer
+        tested web page:
+          type: object
+          properties:
+            description:
+              type: string
+            URL:
+              type: string
+              format: uri
+        whether a later report about the same page exists:
+          type: boolean
+          description: If true, a more recent report about this page is available. If you want the latest results, use that report instead.
+        URLs for incrementally getting facts from the report:
+          $ref: '#/components/schemas/NextTierURLs'
+        size of the report in bytes:
+          type: integer
+        URL to get the entire report as machine-oriented JSON:
+          type: string
+          format: uri
+
     Target:
       type: object
       properties:
@@ -322,7 +361,7 @@ components:
           type: string
           description: All or part of the hostname of the web page. The hostname omits the scheme and the path.
 
-    TargetsResponse:
+    ReportListResponse:
       allOf:
         - $ref: '#/components/schemas/CommonResponseFields'
         - type: object
@@ -330,6 +369,17 @@ components:
             available reports:
               type: array
               description: One entry per available Kilotest report, in alphabetical order by page description and, in case of multiple reports per page, in order of creation date and time. Reports are matched by page description, not page URL, so, if only the URL of a page has changed between reports, the reports are treated as reports about the same page.
+              items:
+                $ref: '#/components/schemas/ReportListItem'
+
+    TargetsResponse:
+      allOf:
+        - $ref: '#/components/schemas/CommonResponseFields'
+        - type: object
+          properties:
+            matching reports:
+              type: array
+              description: One entry per available matching Kilotest report. A report is matching if its page description or hostname contains or is contained by a specified description or hostname fragment, respectively.
               items:
                 $ref: '#/components/schemas/TargetsItem'
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jrpool/kilotest",
-  "version": "36.1.1",
+  "version": "37.0.0",
   "description": "An ensemble testing service with a focus on accessibility",
   "main": "index.js",
   "scripts": {

package/reportIssue/api.js

@@ -106,13 +106,13 @@ exports.response = async args => {
   const thisHost = process.env.THIS_KILOTEST_HOST;
   // Get a response.
   const content = {
-    summary: `This document fulfills a request made by a language model to a Kilotest tool. The model requested data, drawn from a Kilotest report, about one of the issues for the front-end quality (i.e. accessibility, usability, and standard-conformity) of a web page. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten rule engines, performs tests on web pages, using a combination of rule- and machine-learning-based methods, and produces reports. Kilotest exposes several API endpoints to recommend web pages for testing and to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
+    summary: `This document fulfills a request made by a language model to a Kilotest tool. The model requested data, drawn from a Kilotest report, about one of the issues for the front-end quality (i.e. accessibility, usability, and standards conformity) of a web page. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten rule engines, performs tests on web pages, using a combination of rule- and machine-learning-based methods, and produces reports. Kilotest exposes several API endpoints to recommend web pages for testing and to obtain information from Kilotest reports. To learn more about Kilotest and the advantages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
     'tool collection name': 'Kilotest',
     'tool name': 'describeOneIssueFromOneReport',
     request: {
       'type of request': {
         identifier: 'reportIssue',
-        description: 'Describe one issue from one repeort.'
+        description: 'Describe one issue from one report.'
       },
       method: 'GET',
       URLs: {
@@ -123,8 +123,8 @@ exports.response = async args => {
         identifier: 'describeQualityOfOneWebPage',
         description: 'Describe the quality of one web page.',
         URLs: {
-          'for you': `${thisHost}/api/reportIssues`,
-          'for humans': `${thisHost}/reportIssues.html`
+          'for you': `${thisHost}/api/reportIssues/${timeStamp}/${jobID}`,
+          'for humans': `${thisHost}/reportIssues.html/${timeStamp}/${jobID}`
         }
       }
     },

package/reportIssues/api.js

@@ -63,7 +63,7 @@ exports.response = async args => {
   const thisHost = process.env.THIS_KILOTEST_HOST;
   // Get a response.
   const content = {
-    summary: `This document fulfills a request made by a language model to a Kilotest tool. The model requested data from a Kilotest report about the front-end quality (i.e. accessibility, usability, and standard-conformity) of a web page. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten rule engines, performs tests on web pages, using a combination of rule- and machine-learning-based methods, and produces reports. Kilotest exposes several API endpoints to recommend web pages for testing and to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
+    summary: `This document fulfills a request made by a language model to a Kilotest tool. The model requested data from a Kilotest report about the front-end quality (i.e. accessibility, usability, and standards conformity) of a web page. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten rule engines, performs tests on web pages, using a combination of rule- and machine-learning-based methods, and produces reports. Kilotest exposes several API endpoints to recommend web pages for testing and to obtain information from Kilotest reports. To learn more about Kilotest and the advantages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
     'tool collection name': 'Kilotest',
     'tool name': 'describeQualityOfOneWebPage',
     request: {
@@ -77,10 +77,10 @@ exports.response = async args => {
         'equivalent URL for humans': `${thisHost}/reportIssues.html/${timeStamp}/${jobID}`
       },
       'closest ancestor request': {
-        identifier: 'summarizeQualityOfAllTestedWebPages',
-        description: 'Summarize the quality of all tested web pages.',
+        identifier: 'getListOfAllAvailableReports',
+        description: 'Get a list of all available reports.',
         URLs: {
-          'for you': `${thisHost}/api/targets`,
+          'for you': `${thisHost}/api/reportList`,
           'for humans': `${thisHost}/targets.html`
         }
       }

package/reportList/api.js

@@ -0,0 +1,75 @@
+/*
+  api.js
+  Responds to the reportList API request.
+*/
+
+// IMPORTS
+
+const {
+  getAgoDays,
+  getDateTime,
+  getLogs,
+  getNowStamp,
+  getRandomString,
+  getReportSize
+} = require('../util');
+
+// CONSTANTS
+
+const thisHost = process.env.THIS_KILOTEST_HOST;
+
+// FUNCTIONS
+
+// Returns a response to an API request for reports.
+exports.response = async () => {
+  const reportsFacts = [];
+  // Get the available logs, with added job names.
+  const availableLogs = await getLogs();
+  // For each of them:
+  for (const availableLog of availableLogs) {
+    const {jobName} = availableLog;
+    const [timeStamp, jobID] = jobName.split('-');
+    const reportSize = await getReportSize(timeStamp, jobID);
+    const {superseded, url, what} = availableLog;
+    reportsFacts.push({
+      identifier: jobName,
+      'creation date and time': getDateTime(timeStamp),
+      'days since the creation date': getAgoDays(timeStamp),
+      'tested web page': {
+        description: what,
+        URL: url
+      },
+      'whether a later report about the same page exists': !! superseded,
+      'URLs for incrementally getting facts from the report': {
+        'for you': `${thisHost}/api/reportIssues/${timeStamp}/${jobID}`,
+        'for humans': `${thisHost}/reportIssues.html/${timeStamp}/${jobID}`
+      },
+      'size of the report in bytes': reportSize,
+      'URL to get the entire report as machine-oriented JSON': `${thisHost}/fullReport.json/${timeStamp}/${jobID}`
+    });
+  }
+  // Get a response.
+  const content = {
+    summary: `This document fulfills a request made by a language model to a Kilotest tool. The model asked which reports about the front-end quality (i.e. accessibility, usability, and standards conformity) of web pages are available from Kilotest. Each report describes the results of a job that tested one web page. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten rule engines, performs more than a thousand tests on a public web page, using a combination of rule- and machine-learning-based methods. Kilotest exposes API endpoints and web UI URLs to recommend web pages for testing and to obtain information from Kilotest reports. You and your users can learn more about Kilotest and the advantages of testing with an ensemble of rule engines at the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}).`,
+    'tool collection name': 'Kilotest',
+    'tool name': 'getListOfAllAvailableReports',
+    request: {
+      'type of request': {
+        identifier: 'reportList',
+        description: 'Get a list of all available reports. Each report describes the results of a job that tested a web page for front-end quality (i.e. accessibility, usability, and standards conformity). For each report, the list identifies which page was tested and when and provides URLs to begin incremental retrieval of the test results.'
+      },
+      method: 'GET',
+      URLs: {
+        'URL of your request': `${thisHost}/api/reportList`,
+        'equivalent URL for humans': `${thisHost}/targets.html`
+      },
+      'closest ancestor request': null
+    },
+    'response metadata': {
+      identifier: `${getNowStamp()}-${getRandomString(3)}`,
+      'date and time': new Date().toISOString(),
+    },
+    'available reports': reportsFacts
+  };
+  return content;
+};

package/researchAgent.js

@@ -115,9 +115,9 @@ const requestService = async () => {
   if (content.error) {
     return;
   }
-  console.log('======================\nRequest 2: Summarize all available reports');
+  console.log('======================\nRequest 2: List all available reports');
   method = 'GET';
-  path = '/api/targets';
+  path = '/api/reportList';
   content = await submitRequest(results, path, method);
   reports = content?.['available reports'] ?? [];
   if (content.error || ! Array.isArray(reports) || ! reports.length) {

package/target/api.js

@@ -78,7 +78,7 @@ exports.response = async args => {
         'number of HTML elements reported as exhibiting issues': violatorCount,
         'URLs for getting data about the reported issues': {
           'for you': `${thisHost}/api/reportIssues/${timeStamp}/${jobID}`,
-          'for humans': `${thisHost}/reportIssues/${timeStamp}/${jobID}`
+          'for humans': `${thisHost}/reportIssues.html/${timeStamp}/${jobID}`
         },
         'URL for getting the full technical report as JSON': `${thisHost}/fullReport.json/${timeStamp}/${jobID}`
       });
@@ -86,13 +86,13 @@ exports.response = async args => {
   }
   // Get a response.
   const content = {
-    summary: `This document fulfills a request made by a language model to the Kilotest API. The model asked whether Kilotest had tested a specified web page for front-end quality (i.e. accessibility, usability, and standards conformity) and had made a report of the test results available. The response provides summary information about all and only the available reports of pages that match at least one of the provided fragments, where matching means either including or being included by the provided fragment, case-insensitively. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten rule engines, performs tests on web pages, using a combination of rule- and machine-learning-based methods, and produces reports. Kilotest exposes API endpoints to recommend web pages for testing and to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), whose home page contains an introduction and a link to a tutorial.`,
+    summary: `This document fulfills a request made by a language model to the Kilotest API. The model asked whether Kilotest had tested a specified web page for front-end quality (i.e. accessibility, usability, and standards conformity) and had made a report of the test results available. The response provides summary information about all and only the available reports of pages that match at least one of the provided fragments, where matching means either including or being included by the provided fragment, case-insensitively. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten rule engines, performs tests on web pages, using a combination of rule- and machine-learning-based methods, and produces reports. Kilotest exposes API endpoints to recommend web pages for testing and to obtain information from Kilotest reports. To learn more about Kilotest and the advantages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), whose home page contains an introduction and a link to a tutorial.`,
     'tool collection name': 'Kilotest',
-    'tool name': 'identifyReportsAboutMatchingWebPages',
+    'tool name': 'summarizeQualityOfMatchingWebPages',
     request: {
       'type of request': {
-        identifier: 'targets',
-        description: 'Identify reports about matching web pages.'
+        identifier: 'target',
+        description: 'Summarize the quality of matching web pages.'
       },
       method: 'POST',
       payload: {
@@ -109,7 +109,7 @@ exports.response = async args => {
       identifier: `${getNowStamp()}-${getRandomString(3)}`,
       'date and time': new Date().toISOString(),
     },
-    'reports, if any, that may match your page specifications': matchingReports
+    'matching reports': matchingReports
   };
   return content;
 };

package/testRecForm/api.js

@@ -26,7 +26,7 @@ exports.response = async (what, url, why) => {
   );
   // Get a response.
   const content = {
-    summary: `This response acknowledges a request made by a language model to a Kilotest tool. The model recommended that Kilotest test, for the first time, the ${what} web page at ${url} for front-end quality (i.e. accessibility, usability, and standards conformity). A Kilotest manager usually approves a recommendation within a day. When the recommendation is approved, the testing will be performed and results will become available. You can check for the availability of the results at ${thisHost}/api/targets. Kilotest performs its testing with the help of Testaro, Testilo, and an ensemble of ten rule engines, using a combination of rule- and machine-learning-based methods. Kilotest exposes several API endpoints and several web UI URLs to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
+    summary: `This response acknowledges a request made by a language model to a Kilotest tool. The model recommended that Kilotest test, for the first time, the ${what} web page at ${url} for front-end quality (i.e. accessibility, usability, and standards conformity). A Kilotest manager usually approves a recommendation within a day. When the recommendation is approved, the testing will be performed and results will become available. You can check for the availability of the results at ${thisHost}/api/reportList. Kilotest performs its testing with the help of Testaro, Testilo, and an ensemble of ten rule engines, using a combination of rule- and machine-learning-based methods. Kilotest exposes several API endpoints and several web UI URLs to obtain information from Kilotest reports. To learn more about Kilotest and the advantages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), which contains an introduction on its home page and a tutorial.`,
     'tool collection name': 'Kilotest',
     'tool name': 'recommendQualityTestingOfOneWebPage',
     request: {

package/util.js

@@ -489,7 +489,7 @@ exports.getPOSTData = request => new Promise(resolve => {
 });
 // Returns the waiting test and retest recommendations.
 const getRecs = exports.getRecs = async () => {
-  let recs = {};
+  let recs;
   let recsJSON;
   try {
     recsJSON = await fs.readFile(recsPath, 'utf8');
@@ -764,3 +764,9 @@ exports.isReportAvailable = async (what, url) => {
   const miniURLs = logs.map(log => minifyURL(log.url));
   return whats.includes(what) || miniURLs.includes(minifyURL(url));
 };
+// Returns the size of a report.
+exports.getReportSize = async (timeStamp, jobID) => {
+  const reportStat = await fs.stat(path.join(reportsPath, `${timeStamp}-${jobID}.json`));
+  const reportSize = reportStat.size;
+  return reportSize;
+};

package/targets/api.js

@@ -1,99 +0,0 @@
-/*
-  api.js
-  Responds to the targets request.
-*/
-
-// IMPORTS
-
-const {
-  getLogs,
-  getNowStamp,
-  getRandomString,
-  getReportData
-} = require('../util');
-
-// CONSTANTS
-
-const thisHost = process.env.THIS_KILOTEST_HOST;
-
-// FUNCTIONS
-
-// Returns a response to a targets request.
-exports.response = async () => {
-  const availableReports = [];
-  // Get the non-hidden logs.
-  const targetLogs = await getLogs();
-  // For each log:
-  for (const targetLog of targetLogs) {
-    const {jobName} = targetLog;
-    const [timeStamp, jobID] = jobName.split('-');
-    const {superseded, url, what} = targetLog;
-    // Get data about its report.
-    const data = await getReportData(timeStamp, jobID);
-    const {
-      creationDate,
-      daysAgo,
-      issueCount,
-      toolNames,
-      toolCount,
-      reporterNames,
-      reporterCount,
-      violatorCount,
-      preventedToolNames,
-      preventedToolCount
-    } = data;
-    availableReports.push({
-      identifier: jobName,
-      'creation date': creationDate,
-      'days since the creation date': daysAgo,
-      'tested web page': {
-        description: what,
-        URL: url
-      },
-      'whether a later report about the same page exists': !! superseded,
-      'rule engines that tried to test the page': {
-        number: toolCount,
-        names: toolNames
-      },
-      'rule engines that were unable to test the page': {
-        number: preventedToolCount,
-        names: preventedToolNames
-      },
-      'rule engines that reported issues': {
-        number: reporterCount,
-        names: reporterNames
-      },
-      'number of issues reported': issueCount,
-      'number of HTML elements reported as exhibiting issues': violatorCount,
-      'URLs for getting data about the reported issues': {
-        'for you': `${thisHost}/api/reportIssues/${timeStamp}/${jobID}`,
-        'for humans': `${thisHost}/reportIssues/${timeStamp}/${jobID}`
-      },
-      'URL for getting the full technical report as JSON': `${thisHost}/fullReport.json/${timeStamp}/${jobID}`
-    });
-  }
-  // Get a response.
-  const content = {
-    summary: `This document fulfills a request made by a language model to a Kilotest tool. The model requested data about the web pages that Kilotest had tested for front-end quality (i.e. accessibility, usability, and standards conformity) and, for each page, statistics about the results of the tests. Kilotest, with the help of Testaro, Testilo, and an ensemble of ten rule engines, performs tests on web pages, using a combination of rule- and machine-learning-based methods, and produces reports. Kilotest exposes API endpoints and web UI URLs to recommend web pages for testing and to obtain information from Kilotest reports. To learn more about Kilotest and the advangages of testing with an ensemble of rule engines, visit the deployed instance of Kilotest (${process.env.DEPLOYED_KILOTEST_HOST}), whose home page contains an introduction and a link to a tutorial.`,
-    'tool collection name': 'Kilotest',
-    'tool name': 'summarizeQualityOfAllTestedWebPages',
-    request: {
-      'type of request': {
-        identifier: 'targets',
-        description: 'Summarize the quality of all tested web pages.'
-      },
-      method: 'GET',
-      URLs: {
-        'URL of your request': `${thisHost}/api/targets`,
-        'equivalent URL for humans': `${thisHost}/targets.html`
-      },
-      'closest ancestor request': null
-    },
-    'response metadata': {
-      identifier: `${getNowStamp()}-${getRandomString(3)}`,
-      'date and time': new Date().toISOString(),
-    },
-    'available reports': availableReports
-  };
-  return content;
-};