@jrpool/kilotest 46.1.2 → 48.0.0

package/docs/SERVICE.md

@@ -16,24 +16,29 @@ Connection to the host is made with `ssh linuxuser@kilotest.com`. Periodically t
 
 Kilotest is installed at `/opt/jpdev/kilotest` on the server.
 
-Any other application `xyz` can be installed at `/opt/jpdev/xyz` on the server.
+Any other application `xyz` can be installed at `/opt/jpdev/xyz` on the server. (Previously, the QAI tutorial application was deployed as a separate service at `/opt/jpdev/qai`; it has been integrated into Kilotest itself and no longer runs as a separate process, though the documentation structure remains generic to accommodate future independent applications should they be needed.)
 
 ## Process management
 
 ### Process manager
 
-Kilotest is managed with [PM2](https://pm2.keymetrics.io) on the server (not on the local development host). The PM2 configuration is specified in the repository as `pm2.config.js`:
+Kilotest is managed with [PM2](https://pm2.keymetrics.io) on the server (not on the local development host). The PM2 configuration is specified in the repository as `pm2.config.cjs` (CommonJS is required because PM2 loads configuration files with `require()`, which cannot load ES modules):
 
 ```javascript
 module.exports = {
   apps: [
     {
       name: 'kilotest',
-      script: 'index.js',
+      // index.ts cannot be the PM2 script: its autostart guard (import.meta.main) is
+      // false inside PM2's process container, so the server would never start.
+      script: 'serve.ts',
+      // PM2 maps the .ts extension to the bun interpreter, so node must be specified explicitly.
+      interpreter: 'node',
       instances: 1,
       autorestart: true,
       watch: false,
       max_memory_restart: '500M',
+      time: true,
       env: {
         NODE_ENV: 'production',
         BASE_PATH: '/',
@@ -51,6 +56,14 @@ pm2 restart kilotest --time --update-env
 pm2 save
 ```
 
+If the `script` entry point itself has changed, `pm2 restart` reuses the stored path and cannot pick up the new one; delete and recreate the process instead:
+
+```text
+pm2 delete kilotest
+pm2 start pm2.config.cjs
+pm2 save
+```
+
 The management of the PM2 logs is [documented by PM2](https://pm2.keymetrics.io/docs/usage/log-management/).
 
 ### Process tuning
@@ -132,6 +145,26 @@ The DNS records for `kilotest.com` are configured as follows. In each case, `TTL
    - **Host**: _acme-challenge.kilotest.com
    - **Answers**: validation tokens provided by Porkbun ACME client
 
+6. **TXT** (SPF for Porkbun-handled mail)
+   - **Host**: kilotest.com
+   - **Answer**: `v=spf1 include:_spf.porkbun.com ~all`
+
+7. **MX and TXT** (Resend MAIL FROM domain; provides SPF alignment for Resend mail)
+   - **Host**: send.kilotest.com
+   - **Answers**: MX `feedback-smtp.us-east-1.amazonses.com` (priority 10); TXT `v=spf1 include:amazonses.com ~all`
+
+8. **TXT** (Resend DKIM public key)
+   - **Host**: resend._domainkey.kilotest.com
+   - **Answer**: an RSA public key in `p=...` form, issued by Resend
+
+9. **TXT** (DMARC policy)
+   - **Host**: _dmarc.kilotest.com
+   - **Answer**: `v=DMARC1; p=none;`
+
+10. **TXT** (site verification)
+    - **Host**: kilotest.com
+    - **Answers**: `google-site-verification=...` and `smithery-verification=...` tokens issued by Google and Smithery
+
 The DNS configuration as CSV:
 
 ```csv
@@ -140,7 +173,13 @@ kilotest.com,kilotest.com,A,149.28.208.106,3600,0
 kilotest.com,www.kilotest.com,CNAME,kilotest.com,3600,0
 kilotest.com,kilotest.com,MX,fwd1.porkbun.com,3600,10
 kilotest.com,kilotest.com,MX,fwd2.porkbun.com,3600,20
+kilotest.com,send.kilotest.com,MX,feedback-smtp.us-east-1.amazonses.com,3600,10
 kilotest.com,kilotest.com,TXT,"v=spf1 include:_spf.porkbun.com ~all",3600,0
+kilotest.com,send.kilotest.com,TXT,"v=spf1 include:amazonses.com ~all",3600,0
+kilotest.com,resend._domainkey.kilotest.com,TXT,"p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDWjPkYLQ+xYcmMRODdXoXl7QTXK5TkhNQXjl5HcPvk1D5A5eqtjuw17lgZOQy1wiUq8BGId+Iq3BWO32kYTJo+lWMQJ8/Me4M0xQ0qaYPmtTJ9cQ8oyrFUuheu7T0MqTYEwYRbHwE74WkQSQTHnCx0Uy98pcXqOaucA3n1v3YkjQIDAQAB",3600,0
+kilotest.com,_dmarc.kilotest.com,TXT,"v=DMARC1; p=none;",3600,0
+kilotest.com,kilotest.com,TXT,google-site-verification=EnNe2chUyUaV645ENd3c6pHs1PZ_Zb3u5516HSirvrA,3600,0
+kilotest.com,kilotest.com,TXT,smithery-verification=6a3a2327c26bc478be748041f340e14fa50c4386eea3b2692eeffda703b8b3f9,3600,0
 kilotest.com,_acme-challenge.kilotest.com,TXT,GIzAHOFW416fHp7cuTkg4gvJDsyuPZvsPlSsnyViFLQ,3600,0
 kilotest.com,_acme-challenge.kilotest.com,TXT,qQdIIiUSC76PvYuku-AxPsRPY-fJV7T7i3b8fimlFjU,3600,0
 ```
@@ -187,7 +226,7 @@ curl -sv https://example.com/ -o /dev/null
 
 Requests to `https://kilotest.com` are received on port 443 and processed by [Caddy](https://caddyserver.com/), which forwards them via HTTP to the application at `localhost:3000`. Caddy manages, provisions, and renews a TLS certificate via Let’s Encrypt. Any request to `http://kilotest.com` is received on port 80, and Caddy redirects it to an `https` request. Caddy forwards `https` requests to the [local server](http://localhost:3000), where it is processed by the Kilotest service.
 
-The Caddy configuration is maintained and tracked in `/etc/caddy/Caddyfile`. Leading 2 spaces below represent 1 Tab character. The `handle_path` block is used for the QAI application, which is accessed at `https://kilotest.com/qai` and listens on port 3001.
+The Caddy configuration is maintained and tracked in `/etc/caddy/Caddyfile`. Leading 2 spaces below represent 1 Tab character. Requests are forwarded to Kilotest on port 3000, which handles all paths including tutorial content and backward-compatibility redirects.
 
 ```caddyfile
 # Configuration of the Caddy web server.
@@ -196,32 +235,25 @@ The Caddy configuration is maintained and tracked in `/etc/caddy/Caddyfile`. Lea
 kilotest.com {
   # Enable Zstandard and Gzip compression of responses.
   encode zstd gzip
-  # Specify the only paths of forwardable requests.
+  # Specify the only paths of forwardable requests. UptimeRobet uses HEAD.
   @allowedGET {
-    method GET
-    path /mcp / /index.html /robots.txt /openapi.yaml /openapi.json /swagger.yaml /swagger.json /api-docs /llms.txt /llms-full.txt /capability.md *.html* /fullReport.json/* /api/* /tutorial/images/* /favicon.* /style.css /sitemap.xml
+    method GET HEAD
+    path /mcp / /index.html /robots.txt /openapi.yaml /openapi.json /swagger.yaml /swagger.json /api-docs /llms.txt /llms-full.txt /capability.md *.html* /fullReport.json/* /api/* /tutorialWeb/images/* /tutorialAI/images/* /qai /qai/comments /favicon.* /style.css /sitemap.xml
   }
   @allowedPOST {
     method POST
-    path /mcp /requestRetest.html/* /requestTest.html /recAction.html /reannotate.html /renewWCAG.html /api/* /tutorialComment.html /worker/job /worker/report
+    path /mcp /requestRetest.html/* /requestTest.html /requestAction.html /reannotate.html /renewWCAG.html /api/* /tutorialWebComment.html /tutorialAIComment.html /worker/job /worker/report
   }
   # Respond to OPTIONS requests.
   @allowedOPTIONS method OPTIONS
   handle @allowedOPTIONS {
     header {
       Access-Control-Allow-Origin *
-      Access-Control-Allow-Methods GET,POST,OPTIONS
+      Access-Control-Allow-Methods GET,HEAD,POST,OPTIONS
       Access-Control-Allow-Headers Content-Type,Authorization
     }
     respond 204
   }
-  # Redirect any /qai request to /qai/.
-  redir /qai /qai/ 301
-  # Truncate any initial /qai from the forwarded path.
-  handle_path /qai* {
-    # Forward any QAI request to port 3001.
-    reverse_proxy localhost:3001
-  }
   # Forward any other GET, POST, or OPTIONS request, if allowed, to port 3000.
   handle @allowedGET {
     reverse_proxy localhost:3000 {
@@ -283,6 +315,127 @@ When a pull request adds a new route to `index.js`, the Caddyfile at `/etc/caddy
 1. The smoke test workflow runs and verifies that all paths, including the new one, are forwarded by Caddy.
 1. If the Caddyfile was not updated, the smoke test reports a bare 404 for the new path, the required status check fails, and the pull request cannot merge.
 
+## Smoke testing
+
+A periodic GitHub Actions workflow runs smoke tests daily against the deployed service. This workflow is defined in `.github/workflows/periodic-smoke-tests.yml` and runs on a daily schedule at 14:14 UTC. It can also be triggered manually from the GitHub Actions interface.
+
+The periodic smoke tests validate that the deployed Kilotest service is functioning correctly end-to-end, independent of code deployments. Code and infrastructure are deployed together, and this periodic check validates them independently. If a deployment introduces a regression—for example, a missing Caddyfile update or a broken route—the periodic tests will detect it within 24 hours.
+
+The workflow checks all valid GET and POST paths by running `smokeTest.ts` against the deployed service at `kilotest.com`, verifying that Caddy forwards each path to Kilotest rather than returning a bare 404.
+
+## Health monitoring
+
+The deployment is monitored for external health and availability using [UptimeRobot](https://dashboard.uptimerobot.com/). The monitoring target is the site root `/`. UptimeRobot checks this target hourly with a `HEAD` request and sends an email alert to the maintainer in either of two cases:
+
+- The response status code is 502 (Bad Gateway). This typically indicates that Caddy cannot reach the Kilotest application on `localhost:3000`, suggesting the application process has crashed or become unresponsive.
+- The request times out (connection timeout). This suggests a network issue, a host down, or a severely degraded application.
+
+UptimeRobot also sends a recovery message when the service recovers and a subsequent check succeeds after a prior failure.
+
+This monitoring service requires `Caddyfile` to permit `HEAD` requests, not only `GET` requests, to the root path.
+
+## Alerting
+
+Kilotest sends email alerts to the maintainer when certain events occur during normal operation. This section describes when and how those alerts are configured and sent.
+
+### When alerts are sent
+
+Alerts are sent to the maintainer in the following conditions:
+
+1. **Tutorial comment received**: When a user submits a comment on either the web tutorial page (`/tutorialWeb.html`) or the AI agent tutorial page (`/tutorialAI.html`), an alert is sent with subject `"New web tutorial comment received"` or `"New AI tutorial comment received"`.
+
+2. **Annotations become obsolete**: The `/listTopIssues.html` page displays no individual report; it aggregates violation counts, across all stored reports (one per tested page, its latest only), into a single cross-report summary of the most frequently reported issues. To build that summary, Kilotest reads the `issueID` field that it previously wrote onto each violating standard instance during annotation, and looks that `issueID` up as a key in the current `testaro-issues` package's issue definitions. The alert fires only if that key is entirely absent, i.e., the issue that this violation was previously classified under has since been removed (or renamed to a different key) in `testaro-issues`. It does not fire, and this check cannot detect, a rule being *reclassified* to a different, still-existing issue; that broader discrepancy (any stored `issueID` that no longer matches what `testaro-issues` currently assigns the rule, whether because the issue vanished or because the rule was moved to a different issue) is instead surfaced separately, without an alert, on the `/reannotateForm.html` page, which lists both such "reclassified" rules and rules that remain wholly unclassified. Each violating instance is checked independently, so a single stored report can trigger this alert once per distinct missing issue ID it contains, and the alert body names one such issue ID at a time. Despite naming a specific issue, the alert does not point at a way to reannotate just that issue: Kilotest's only reannotation mechanism, reached via `/reannotateForm.html` and submitted at `/reannotate.html`, always re-annotates every standard instance of every currently stored report in one bulk operation (see `web/reannotate/index.ts`); there is no per-issue or per-report reannotation. The named issue ID is diagnostic detail only, telling the maintainer why reannotation is needed, not a parameter to a targeted fix.
+
+3. **Unclassified rules violated**: When a Testaro report is received and annotated, Kilotest attempts to classify every rule violation by looking up each rule in the `testaro-issues` package, which Kilotest depends on (`testaro-issues` in `package.json`) but does not itself define. If a rule has no classification in the currently installed `testaro-issues` package (i.e., the rule has never been assigned to any issue), that rule is added to an `issuelessRules` list in the report, and an alert is sent with subject `"Kilotest: unclassified rules violated"`, listing which rules in which rule engines lack classifications. Because Kilotest can only look up classifications that `testaro-issues` already provides, it has no means to give such a violation an issue ID until `testaro-issues` is updated to supply one; the remedy is two-fold and cannot be done in Kilotest alone: first update `testaro-issues` to add the missing rule-to-issue mapping, then, once Kilotest's dependency on `testaro-issues` is updated to that new version, reannotate the stored reports (`/reannotateForm.html`, see item 2 above) so their violating instances of that rule are written with the new issue ID. The two packages currently share a maintainer, so coordinating the two steps is not an external obstacle, but both steps remain necessary.
+
+4. **Unusable report received**: When a Testaro worker submits a report that Kilotest cannot parse or use (e.g., malformed data), an alert is sent with subject `"Kilotest: unusable report received"`, indicating which job produced the report and which worker submitted it.
+
+5. **Requested full report unavailable**: When a user requests a full report via the API but the report cannot be retrieved (e.g., the report file is missing from disk), an alert is sent with subject `"Kilotest: requested full report unavailable"`, including details on which report was requested and why it could not be retrieved.
+
+6. **Test request received via UI**: When a user submits a test request through the web interface, an alert is sent with subject `"Kilotest: new test request in the UI"` (via `/requestTest.html`) or `"Kilotest: new retest request in the UI"` (via `/requestRetest.html`), including the target URL, description, and the user's stated reason for the request. `test` and `retest` here are the only two literal values Kilotest passes as the request's type; they are hardcoded at each call site, not user-supplied.
+
+7. **Test request received via API**: When a test request is submitted via the API, an alert is sent with subject `"Kilotest: new test request in the API"` (via the `requestTest` API operation) or `"Kilotest: new retest request in the API"` (via the `requestRetest` API operation), including the target URL, description, and the user's stated reason. A request is treated as a duplicate, and no alert is sent for it, only if a request already pending in `db/jobs/testRequests.json`, i.e., submitted but not yet manually approved by the maintainer into an actual job, for the same URL has the identical description; this check ignores the stated reason and the test-versus-retest distinction, and is not limited to a recent time window, it compares against every request still pending approval for that URL. Once the maintainer approves a pending request (turning it into a job via `/enqueue.html`), all pending requests for that URL are cleared, so this check cannot detect a duplicate against a request that has already been approved, is currently running, or has already completed; an identical request submitted after approval alerts again. The check and its dedup key are identical for the UI and API paths, since both funnel through the same underlying request-queuing function.
+
+8. **MCP feature request received**: When a user submits a feature request via the MCP server's `requestFeature` tool, an alert is sent with subject `"MCP feature request received"`, including the text of the feature request.
+
+9. **WAVE balance low**: When Kilotest detects that the account balance for the WAVE API service (used for accessibility testing) has fallen below a threshold, an alert is sent with subject `"Kilotest: WAVE balance low"`, indicating the number of credits remaining.
+
+10. **AI service balance low**: When Kilotest detects that the account balance for an AI service account (such as Anthropic) has fallen below a threshold, an alert is sent with subject `"Kilotest: AI service [N] balance low"`, indicating the approximate remaining balance in dollars and the estimated cost per job.
+
+### Alert configuration
+
+Alerts are sent via the `sendAlert()` function in [alerts.ts](../alerts.ts), which requires five environment variables to be configured:
+
+1. **`MANAGER_EMAIL`**: The email address to which alerts should be sent. This is typically the maintainer's email address.
+2. **`ALERT_API_HOST`**: The hostname of the email delivery service. For Kilotest, this is configured to use [Resend](https://resend.com/), a transactional email service.
+3. **`ALERT_API_PATH`**: The API endpoint path on the email delivery service. For Resend, this is typically `/emails` or similar, depending on the API version.
+4. **`ALERT_API_KEY`**: An API key issued by the email delivery service for authentication. For Resend, this is the API key generated in the Resend dashboard.
+5. **`ALERT_FROM`**: The sender email address for outgoing alerts. This must be a verified sender address on the email delivery service.
+
+If any of these five variables is missing or empty at alert time, the alert is not sent. Instead, a warning message is logged to the console, and the application continues normal operation. The warning includes the alert subject and body so the maintainer can manually review what would have been sent.
+
+### Resend integration
+
+Kilotest uses [Resend](https://resend.com/) as its transactional email delivery service. Resend is configured through the environment variables above.
+
+**Resend configuration for Kilotest**: `ALERT_API_HOST` and `ALERT_API_PATH` together identify the Resend API endpoint Kilotest posts to; `ALERT_API_KEY` is a Resend-issued API key; `ALERT_FROM` is the sender address Kilotest sends as. The live values of these environment variables are set in the server's `.env` file, not its PM2 configuration (see “Setting environment variables in production” below), and are not duplicated here.
+
+### DNS configuration for Resend
+
+Resend authenticates outgoing mail using DKIM, and mail receivers additionally consult SPF and DMARC records for the sending domain. The `kilotest.com` DNS zone (managed at [porkbun.com](https://porkbun.com/); see the DNS Configuration section above for the full record set) currently contains the following records relevant to Resend, confirmed live on 2026-09-17 by querying DNS directly and by inspecting the full headers of a delivered alert message:
+
+1. **DKIM (DomainKeys Identified Mail)**
+   - **Record type**: TXT (not a CNAME)
+   - **Host**: `resend._domainkey.kilotest.com`
+   - **Value**: an RSA public key in `p=...` form, issued by Resend for this domain.
+
+2. **SPF (Sender Policy Framework)**
+   - **Record types**: MX and TXT
+   - **Host**: `send.kilotest.com` (a dedicated MAIL FROM subdomain, not the root domain)
+   - **Values**: MX `feedback-smtp.us-east-1.amazonses.com` (priority 10) and TXT `v=spf1 include:amazonses.com ~all`
+   - This pair is what Resend’s “Enable SPF” option creates. It makes Resend send with a MAIL FROM on `send.kilotest.com`, which aligns with the `kilotest.com` From domain under relaxed DMARC alignment. Because Resend sends through Amazon SES, the include target is `amazonses.com`, not `resend.com`. The separate root-domain TXT record `v=spf1 include:_spf.porkbun.com ~all` authorizes Porkbun’s servers for mail sent as `@kilotest.com` through Porkbun; SPF is evaluated against the MAIL FROM domain, so the root record plays no role in Resend authentication and needs no Resend mechanism.
+
+3. **DMARC (Domain-based Message Authentication, Reporting and Conformance)**
+   - **Record type**: TXT
+   - **Host**: `_dmarc.kilotest.com`
+   - **Value**: `v=DMARC1; p=none;`
+   - The `p=none` policy is monitoring-only: it asks receivers to take no special action on mail that fails alignment, rather than quarantining or rejecting it. No `rua=` reporting address is configured, so no aggregate DMARC reports are being sent anywhere.
+
+A delivered alert message inspected on 2026-09-17 confirms this configuration end to end: `spf=pass` with `smtp.mailfrom` on `send.kilotest.com`, `dkim=pass` with `d=kilotest.com` (`s=resend`), and `dmarc=pass` at both the Porkbun forwarding hop and the final mailbox. Any future change to Resend’s configuration (for example, rotating the DKIM key, adding a `rua=` reporting address to the DMARC record, or tightening the DMARC policy from `p=none` to `p=quarantine`) should be reflected here and in the DNS Configuration section’s CSV.
+
+### Setting environment variables in production
+
+The deployed `pm2.config.cjs` on the server (`/opt/jpdev/kilotest/pm2.config.cjs`) does not list any of the five alert variables in its `env` object; it matches the repository's own copy of that file, which contains only `NODE_ENV`, `BASE_PATH`, and `DEMO_SSE_DELAY_MS`. Alerting is nonetheless working in production, so the five alert variables are being supplied another way.
+
+`index.ts` calls `dotenv.config({quiet: true})` at startup, which reads a `.env` file from the process's working directory (`/opt/jpdev/kilotest/.env`) and loads its contents into `process.env`, in addition to whatever PM2 itself already set. `.env` is listed in `.gitignore`, so it is never committed and exists only on the server; this is the actual mechanism supplying `MANAGER_EMAIL`, `ALERT_API_HOST`, `ALERT_API_PATH`, `ALERT_API_KEY`, and `ALERT_FROM` to the running process. Its expected contents, including these five variables, are documented by the repository's own [env.example](../env.example) (see also the setup instructions in the project [README](../README.md)); `env.example` already records `ALERT_API_HOST=api.resend.com` and `ALERT_API_PATH=/emails` as committed values (also confirmed live: a direct request to `https://api.resend.com/emails` returns `401 Unauthorized`, i.e., a real endpoint that requires an API key, not a routing failure), while `MANAGER_EMAIL`, `ALERT_API_KEY`, and `ALERT_FROM` are left as placeholders there, to be filled in per deployment with server-side secrets not recorded in this document.
+
+To change any of the five alert variables:
+
+1. Edit `/opt/jpdev/kilotest/.env` on the server directly.
+2. Restart PM2 so the process starts fresh and `dotenv.config()` re-reads the file:
+
+   ```bash
+   pm2 restart kilotest --time --update-env
+   pm2 save
+   ```
+
+### Troubleshooting
+
+**Alerts are not being sent**:
+
+1. Check that all five alert environment variables are set in `/opt/jpdev/kilotest/.env` on the server (not `pm2.config.cjs`, which does not carry them): `MANAGER_EMAIL`, `ALERT_API_HOST`, `ALERT_API_PATH`, `ALERT_API_KEY`, `ALERT_FROM`.
+2. Check the PM2 logs for warning or error messages: `pm2 logs kilotest | grep -i alert`.
+3. Verify that the Resend API key is valid and has not been revoked in the Resend dashboard.
+4. Verify that the sender email address (`ALERT_FROM`) is verified in the Resend dashboard.
+5. Verify that the DNS records (DKIM, SPF, DMARC) are correctly configured for `kilotest.com`.
+
+**Alerts are bouncing or not reaching the maintainer**:
+
+1. Check the Resend dashboard logs for delivery errors.
+2. Verify that the `MANAGER_EMAIL` address is correct and actively monitored.
+3. Check the maintainer's email spam folder; some email providers filter unfamiliar senders.
+4. Verify DMARC, SPF, and DKIM records are correctly configured.
+
 ## Performance
 
 The Cloud Compute host, in initial testing, took about 2.5 as long to process an example job as an Apple M2 Pro MacBook Pro with 16GB of memory. After tuning, the ratio was reduced to about 1.7.
@@ -308,7 +461,7 @@ Experimentation revealed that a high-frequency instance could decrease the elaps
 
 Jobs and reports are not tracked, so there are no duplicates in any other copy of the repository. Jobs are typically ephemeral, but reports typically remain in existence until deemed obsolete and useless even for historical comparison. Therefore, reports risk deletion unless duplicates are made externally. Other files, including application code, dependencies, and `db/reportsExtract.json`, are not at risk, because they can be pushed from the local repository or regenerated.
 
-Reports created on the deployed server are currently protected with an external archive at the [Cloudflare R2 object storage service](https://developers.cloudflare.com/r2/). On that service, the current Kilotest maintainer has an account, subscribes to the R2 service, has created two _buckets_ named `kilotest-reports` and `kilotest-hidden-reports`, and has created an API token scoped to those buckets with object read-write permissions, access and secret access keys for S3 clients, and a restriction to the IPv4 and IPv6 addresses of the server.
+Reports created on the deployed server are currently protected with an external archive at the [Cloudflare R2 object storage service](https://developers.cloudflare.com/r2/). On that service, the current Kilotest maintainer has an account, subscribes to the R2 service, has created two *buckets* named `kilotest-reports` and `kilotest-hidden-reports`, and has created an API token scoped to those buckets with object read-write permissions, access and secret access keys for S3 clients, and a restriction to the IPv4 and IPv6 addresses of the server.
 
 The server host uses `rclone` for file synchronization with external storage locations. Two files on the server host enable `rclone` for use by Kilotest. One file is `/home/linuxuser/.config/rclone/rclone.conf`. Its content, with secrets replaced, is:
 

package/docs/decisions/002-coverage-tool.md

@@ -0,0 +1,40 @@
+---
+date: 2026-09-13
+status: accepted
+---
+
+# Coverage tool: keep c8, revisit when Node.js experimental coverage stabilizes
+
+## Context and problem
+
+Kilotest enforces 100% line, branch, and statement coverage on the production source files in its test suite. During the TypeScript/ESM migration (step 12), the question arose whether to replace the `c8` coverage tool with Node.js's built-in `--experimental-test-coverage` reporter.
+
+## Considered options
+
+1. **Keep `c8`.**
+2. **Switch to Node.js built-in coverage.**
+
+## Decision
+
+**Keep `c8`** for now.
+
+## Rationale
+
+### Why `c8` works
+
+There is exactly one `c8` exclusion in the codebase: the `/* c8 ignore start */` … `/* c8 ignore stop */` block around the import statements at the top of `util.ts`. A comment there explains that `c8` intermittently misreports import lines as uncovered against the installed Node.js version. The exclusion was re-verified empirically during step 12 by temporarily removing it and confirming the failure. It is still necessary.
+
+### Why Node.js built-in coverage does not yet work cleanly
+
+The switch was tested against the full suite. Two problems arose:
+
+1. **`mock.module()` corrupts coverage accounting.** Five test files use `t.mock.module()`. Under `--experimental-test-coverage`, one of them (`web/requestRetest`) reproducibly shows its success-path lines (28–30) as uncovered, despite a passing test that exercises exactly that path. The gap appears only after a later test in the same file runs `import('./index.ts?mockNoExtracts')` alongside `t.mock.module()`. This matches a family of actively-tracked Node.js core bugs where `mock.module()` combined with `--experimental-test-coverage` corrupts coverage accounting (nodejs/node#59112, #61709, #58119). The failure is inconsistent across runs, making it hard to audit or suppress reliably.
+
+2. **Ignore comments do not suppress branch coverage.** The standard `/* node:coverage ignore next [n] */` syntax suppresses line coverage but not branch coverage for the same lines, per nodejs/node#61586. This matters because any exclusion added to work around problem 1 would sit on real, exercised logic (`requestRetest`'s success path), and the standard workaround is documented not to fully apply in exactly that case. So the switch would trade one well-understood, narrowly-scoped exclusion (import lines in `util.ts`) for a new, less predictable one on real logic with a broken workaround.
+
+## Revisit condition
+
+Revisit this decision once `--experimental-test-coverage` stabilizes and the following upstream bugs are resolved:
+
+- nodejs/node#59112, #61709, #58119 — `mock.module()` + coverage accounting
+- nodejs/node#61586 — ignore comments do not suppress branch coverage

package/docs/decisions/003-adr-threshold.md

@@ -0,0 +1,34 @@
+---
+date: 2026-06-30
+status: accepted
+---
+
+# ADR threshold
+
+## Context and problem
+
+Several decisions are being made early in the life of the project. What importance threshold should merit the documentation of a decision in an architectural decision record (ADR)?
+
+## Considered options
+
+- Lower: Create an ADR for each nontrivial decision.
+- Higher: Create an ADR for each decision that is judged strategic. Deem a decision strategic if there are substantially different alternatives and the choice of an alternative is expected to have substantial effects on the maintainability or quality of the project.
+
+## Decision
+
+Higher, because:
+
+- Formally documenting tactical decisions slows development with little benefit.
+- Persons who want to read the ADRs for the project history can better digest the content if it remains parsimonious and strategically focused.
+- Minor decisions are often understandable with inspection of the codebase.
+- Decisions affecting single points in a codebase are more efficiently explained with adjacent comments that do not require navigation to a separate file.
+
+This decision will cause the count of ADRs to be moderate, making a 3-digit naming prefix practical.
+
+### Confirmation
+
+To confirm that the decision is appropriately implemented, ask LLMs and human colleagues to judge whether the ADRs describe all and only the strategic decisions manifested in the codebase.
+
+## ADR mutability convention
+
+Unlike the strict Nygard convention of never editing an ADR once written, Kilotest maintains ADRs as kept-current documents. An ADR describes the strategic decision *as currently implemented in the codebase*, and may be edited to reflect how that decision has evolved. Historical context (for example, "approach X was tried but had defect Y, so it was replaced with Z") belongs inside the ADR when it helps prevent future maintainers from reconsidering a decision that was already evaluated and rejected. This convention allows the ADR set to remain authoritative throughout the project lifecycle without requiring separate, frozen ADRs to document every historical iteration.

package/docs/decisions/004-docs-directory.md

@@ -0,0 +1,23 @@
+---
+date: 2026-06-30
+status: accepted
+---
+
+# `docs` directory
+
+## Context and problem
+
+The project began with several files of planning documentation at the root and is now acquiring ADRs as well. Where should all those documents be located?
+
+## Considered options
+
+- Minimal change: Create a `docs/decisions` directory for ADRs but leave the planning documents at the root.
+- Maximal coherence: Create a `docs` directory for all of those documents, with ADRs in a `decisions` subdirectory.
+
+## Decision
+
+Maximal coherence, because it substantially simplifies navigation for people perusing the filesystem, and its cost is only revising a few links in documentation and README.
+
+### Confirmation
+
+To confirm the correctness of the implementation, find all internal links in the codebase and verify that each link to a file that is now in the `docs` directory has the current file location as its destination.

package/docs/decisions/005-observability.md

@@ -0,0 +1,28 @@
+---
+date: 2026-07-01
+status: accepted
+---
+
+# Observability
+
+## Context and problem
+
+The maintainer will want to review some data about the operation of Kilotest and will want to be alerted to some activities. What data should be collected, and what activities should merit alerts?
+
+## Considered options
+
+- Rich: Collect abundant data and send alerts on all significant events.
+- Minimal: Collect obviously important data and send alerts on events that require maintainer action.
+
+## Decision
+
+Minimal, because:
+
+- Rich data and frequent alerts cause inattention, dismissal, or even blocking.
+- The anticipated interactions between users and Kilotest will gradually increase in complexity and unpredictability, but by the first release only two event types are expected to justify alerts: (1) server errors and (2) comment submissions by users.
+
+### Confirmation
+
+To confirm that the implementation is correct, launch the application, submit requests to it, and verify that any required logs appear. Then temporarily corrupt a request handler to make Kilotest return an error response. Verify that an alert from the alerting module with the expected content has arrived at the expected destination.
+
+Which events merit logging, what the contents of alerts should be, and what packages should be chosen as observability and alerting modules are among the decisions classified as "tactical" in [ADR 003](./003-adr-threshold.md). Therefore, those decisions are not described in this ADR or other ADRs.

package/docs/decisions/006-git-workflow.md

@@ -0,0 +1,28 @@
+---
+date: 2026-07-02
+status: accepted
+---
+
+# Git workflow
+
+## Context and problem
+
+The project uses Git for version control and GitHub as the host of the origin repository. For simplicity, the process of revision and versioning should be consistent. What Git workflow should be chosen?
+
+## Considered options
+
+- Push to main: Merge feature branches into the main branch locally and then push the main branch to the origin.
+- PR to main: Merge feature branches into the main branch at the origin by means of approval of pull requests, and never push from the local to the origin main branch.
+
+In each case, the decision includes mandatory updates to the dependencies and mandatory internal testing before every push or merge to the origin main branch.
+
+## Decision
+
+PR to main, because:
+
+- It is more secure. If pushes from the local to the origin main branch are permitted, a developer can bypass required prerequisites with the `--no-verify` flag.
+- It is more team-friendly. Although initially there is only 1 developer, local main branches could diverge if other developers began to contribute, and pushes could conflict or lose changes.
+
+### Confirmation
+
+To confirm that the implementation is correct, try to push to the origin main branch and verify that the push attempt is rejected.

package/docs/decisions/007-internal-testing.md

@@ -0,0 +1,34 @@
+---
+date: 2026-07-02
+status: accepted
+---
+
+# Internal testing
+
+## Context and problem
+
+Internal testing creates a tradeoff between speed of development and risk of debugging effort. Under what conditions should internal testing be required?
+
+## Considered options
+
+Make testing a prerequisite for any:
+
+- push to an origin feature branch
+- merge to the origin main branch
+
+Any required testing must take place automatically upon a request to perform the merging or pushing action. The choice of test-triggering mechanisms is a tactical decision.
+
+## Decision
+
+Require testing only before merges to the origin main branch, because:
+
+- Merges to the origin main branch are infrequent, so mandatory testing before them is not burdensome.
+- The risk of failure is excessive if the deployed server is permitted to relaunch with code of an untested main branch.
+- Mandatory testing before pushes to the origin of a feature branch would deter routine pushes that protect the local code from accidental deletion.
+- The absence of a local-branch testing requirement still allows developers to perform voluntary testing of any local branches at any time.
+
+Although this decision governs only mandatory testing, developers are advised to precede any merge to an origin feature branch with voluntary testing unless they have not yet corrected known bugs in it.
+
+### Confirmation
+
+To confirm that the implementation is correct, try to merge a defective origin feature branch to the origin main branch. Verify that testing is automatically performed and the merge is prevented by a test failure.

package/docs/decisions/008-dependencies.md

@@ -0,0 +1,28 @@
+---
+date: 2026-07-02
+status: accepted
+---
+
+# Dependencies
+
+## Context and problem
+
+The project will install dependencies. How should their updates be handled?
+
+## Considered options
+
+- Pin to major version: Specify the major version of each dependency and, when dependencies are updated, update it only to the latest minor and patch version of that major version.
+- Stay with latest: As a general rule, when dependencies are updated, update them to their latest versions. Specify a version range only when the latest version has been found incompatible.
+
+## Decision
+
+Stay with latest, because:
+
+- It gives the project access to the latest features and corrections.
+- It eliminates the drudgery of monitoring dependency updates.
+- It makes the reporting of dependency bugs simpler by eliminating the need to retest with the latest version and thereby encourages participation in open-source communities.
+- It does not introduce a risk of malfunction in production, because merges to the main branch require internal testing.
+
+### Confirmation
+
+To confirm that the implementation is correct, update the dependencies and spot-check that the installed versions are the latest published ones.

package/docs/decisions/009-qai-integration.md

@@ -0,0 +1,40 @@
+---
+date: 2026-09-14
+status: accepted
+---
+
+# QAI integration
+
+## Context and problem
+
+The QAI application (at `https://github.com/jrpool/qai`) is a small standalone Node.js server deployed at `https://kilotest.com/qai`. QAI is a tutorial showing users how to connect AI platforms (specifically, Claude on `claude.ai`) to Kilotest's MCP server. The `kilotest` repository already provides a tutorial showing users how to use Kilotest itself. The two-repository split was required by an organizational constraint that no longer applies. QAI's own purpose (teaching users how to connect an AI platform to Kilotest) makes it naturally a part of Kilotest's tutorial content. Should QAI remain a separate deployment, or be folded into Kilotest?
+
+## Considered options
+
+1. **Keep separate:** QAI remains an independent repository with its own deployment, continuous integration, and process management.
+2. **Merge into Kilotest:** Port QAI's tutorial content and comment-collection infrastructure into Kilotest as a second tutorial module, retire QAI as a separate deployment, and simplify the Caddyfile and process management.
+
+## Decision
+
+**Merge into Kilotest**, because:
+
+- QAI is fundamentally tutorial content for Kilotest, not an independent application. Keeping it in a separate repository obscures this relationship.
+- Merging eliminates duplicate infrastructure: both repositories use the same comment-storage pattern, alert environment variables, build configuration, and type-checking standards. A single repository reduces maintenance burden.
+- The existing `web/tutorial/` pattern in Kilotest provides a proven, documented template for modules combining an HTML page with a comment-collection form and email alerts. QAI's infrastructure was originally designed by copying this pattern; reusing it directly instead of maintaining a reimplementation aligns the codebase with DRY principles.
+- Simplifying the Caddyfile and process management reduces the risk of configuration drift, and makes the deployment easier to reason about.
+- Merging improves the documentation standard. QAI includes 9 ADRs and architectural/requirements/deployment documentation that enriches Kilotest's own documentation, which initially had only 2 ADRs.
+
+### Confirmation
+
+To confirm that the implementation is correct:
+
+1. All tests pass with 100% coverage, and linting and type-checking produce no errors.
+2. The new QAI tutorial page and comment form are reachable and functional at their new `/tutorialAI.html` path.
+3. The old QAI paths (`/qai`, `/qai/comments`) redirect correctly to the new paths.
+4. Comments submitted to the new form are stored correctly and trigger email alerts as configured.
+
+## Historical context
+
+QAI was initially developed as an independent deployment partly because Kilotest had not yet implemented monitoring and alerting infrastructure. By the time QAI reached deployment, both QAI and Kilotest had similar comment-collection and alerting patterns. Keeping QAI separate after this convergence created unnecessary duplication. This merge resolves that architectural drift and reestablishes Kilotest as the canonical location for all Kilotest-related tutorial content.
+
+The separate `jrpool/qai` repository is preserved as a historical archive in `jrpool`'s account and can be kept as read-only history for reference.

package/docs/decisions/QAI-HISTORICAL-CONTEXT.md

@@ -0,0 +1,42 @@
+# QAI historical context
+
+This document preserves pointers to documentation and development history from the `jrpool/qai` repository, which was integrated into Kilotest as part of the decision documented in [ADR 009](./009-qai-integration.md).
+
+## QAI architectural decisions (superseded or historical)
+
+The following ADRs from QAI are not ported to Kilotest, but their reasoning and historical context are preserved in the `jrpool/qai` repository for reference:
+
+- **003-initial-routing.md**: QAI originally chose to deploy as an independent service on a separate port, separate from Kilotest's port 3000. This was driven by course requirements and the need for QAI to have evaluable infrastructure of its own. ADR 009 documents the superseding decision to integrate QAI into Kilotest's deployment.
+
+- **004-prerelease-versions.md**: QAI used prerelease versioning (0.1.0, etc.) during initial development. This versioning scheme is not carried forward to the integrated Kilotest; Kilotest's own version scheme applies to all content.
+
+- **005-infrastructure-health.md**: QAI originally required its own external monitoring with UptimeRobot because Kilotest lacked health monitoring infrastructure. This constraint is now resolved; Kilotest's unified monitoring and alerting (covered in ADR 005-observability.md) applies to both the tutorial content and any other components.
+
+## QAI development history
+
+Sprint reports, peer reviews, and development progress documentation are preserved in the `jrpool/qai` repository:
+
+- `docs/progress-reports/` contains sprint reviews and reflections from the course development of QAI (sprints 3–5).
+- `docs/reviews/` contains peer review feedback on QAI's initial implementation.
+
+This historical record is retained in `jrpool/qai`'s git history and is not duplicated in Kilotest's codebase.
+
+## Branch protection and CI configuration
+
+QAI's GitHub branch-protection ruleset (`docs/rulesets/qai-main.json`) enforced requirements similar to those already in place for Kilotest. Kilotest's own branch protection, documented in `docs/SERVICE.md`, supersedes QAI's configuration, and the separate QAI ruleset file is not ported.
+
+## QAI domain-specific documentation
+
+The following documents from QAI are superseded by the integration:
+
+- **`requirements.md`**: Documented QAI's specific functional requirements. These remain valid for the integrated tutorial, but are now framed as part of Kilotest's tutorial offerings rather than QAI-specific product requirements. Tutorial requirements are documented in the code comments and inline in `web/tutorialAI/index.ts` rather than as a separate file.
+
+- **`architecture.md`**: Described QAI as an independent deployment. With integration, QAI's architecture is now part of Kilotest's architecture, and the former QAI-specific architectural overview is superseded by Kilotest's architecture and API design documented in the main codebase and in [ADR 001 (API design)](./001-api-design.md).
+
+- **`deployment.md`**: Documented QAI's guest deployment on Kilotest infrastructure and its PM2 configuration. With QAI integrated into Kilotest, a single deployment configuration applies to both. PM2 and infrastructure details are now documented in `docs/SERVICE.md` with no separate QAI-specific deployment doc.
+
+- **`repository.md`**, **`project-vision.md`**: Project-management framing specific to QAI as a university-course project. These provide historical context but are not carried forward to Kilotest's documentation.
+
+- **`future.md`**, **`ai-implementation-review.md`**: Point-in-time project status and review artifacts. Historical context and any actionable ideas are incorporated into the main codebase and Kilotest's backlog; the original documents are not ported.
+
+- **`manual-verification.md`**: Documented QAI's testing and deployment verification procedures. Kilotest's unified test suite and continuous-integration configuration now cover the integrated tutorial content. Some manual verification procedures from QAI (such as form submission and alerting) are mirrored in Kilotest's `smokeTest.ts` and existing test suites.

package/eslint.config.mjs

@@ -3,6 +3,7 @@ import globals from "globals";
 import json from "@eslint/json";
 import markdown from "@eslint/markdown";
 import css from "@eslint/css";
+import tseslint from "typescript-eslint";
 import {defineConfig} from "eslint/config";
 
 export default defineConfig([
@@ -34,6 +35,32 @@ export default defineConfig([
       ]
     }
   },
+  {
+    files: ["**/*.ts"],
+    plugins: {
+      js
+    },
+    extends: ["js/recommended", tseslint.configs.recommended],
+    languageOptions: {
+      globals: globals.node
+    },
+    rules: {
+      "@typescript-eslint/ban-ts-comment": [
+        "error",
+        {"ts-nocheck": "allow-with-description"}
+      ],
+      "no-control-regex": "off",
+      "@typescript-eslint/no-explicit-any": "off",
+      "rest-spread-spacing": ["error", "never"],
+      "space-unary-ops": [
+        "error",
+        {
+          "words": true,
+          "nonwords": false
+        }
+      ]
+    }
+  },
   {
     files: ["**/*.js"],
     languageOptions: {