turva-llms-txt-validator 0.3.13 → 0.3.15

package/README.md

@@ -54,7 +54,7 @@ Use JSON for automation and strict mode when warnings should fail a CI step:
 npx --yes turva-llms-txt-validator example.com --json --strict
 ```
 
-Completed validation returns `{ target, summary, checks }`. Each check contains `{ id, status, label, detail }`. The `input-path` check, which reports a path given with the domain and then ignored, also carries `value`, that path cut to 120 characters. With `--json`, an invalid domain or a failed fetch returns `{ "error": "...", "target": "..." }` and exits with code `2`. The target is the address given on the command line, with any user name, password and query value masked, and a network error adds `code`, such as `EAI_AGAIN`. An unknown option, a second domain and a missing domain return `{ "error": "..." }` only, with no target, and also exit with code `2`. `--help` exits with code `0`, and `--version` or `-V` prints the installed version and exits with code `0`.
+Completed validation returns `{ target, summary, checks }`. Each check contains `{ id, status, label, detail }`. The `input-path` check, which reports a path given with the domain and then ignored, also carries `value`, that path cut to 120 characters. With `--json`, an invalid domain or a failed fetch returns `{ "error": "...", "target": "..." }` and exits with code `2`. The target is the address given on the command line, with any password and query value masked. A user name is masked too, except in a value such as `user:secret@example.com`, where the part before the colon reads as a URL scheme and is shown as `user:***@example.com`. A value with no scheme and no slash is read as user information when a colon comes before its last `@`, so `user_name:secret@example.com` is shown as `***@example.com`. Without that colon, as in `path@2x.png` or an e-mail address, the value is shown as given and holds no password. A network error adds `code`, such as `EAI_AGAIN`. An unknown option, a second domain and a missing domain return `{ "error": "..." }` only, with no target, and also exit with code `2`. `--help` exits with code `0`, and `--version` or `-V` prints the installed version and exits with code `0`.
 
 ## What it checks
 
@@ -69,9 +69,9 @@ Completed validation returns `{ target, summary, checks }`. Each check contains
 | 5 | H2 sections group the content | none | No H2 sections, no section with a Markdown link list, a heading between the title and the first H2, or a second H1 |
 | 6 | Markdown links have names and absolute HTTP or HTTPS targets | none | Missing links, empty names, relative targets or targets that do not parse as HTTP or HTTPS URLs |
 | 7 | File is small enough to read cheaply | none | Over 50 KB, or truncated at the 256 KB read limit |
-| 8 | File contains no HTML markup | none | HTML tags found |
+| 8 | File contains no HTML markup | none | An opening tag, a closing tag or an HTML comment is found anywhere in the file, including inside code spans and fenced code. Autolinks such as `<https://example.com>` are not tags |
 
-Links are read the way CommonMark reads them for the forms listed here. A link inside inline code, after an escaped bracket or in an image does not count, and a link in an ordered list, a target in angle brackets and a target with a title in double or single quotes do. A title in parentheses is not read, so `[Guide](https://example.org/guide (Overview))` is not collected. An unbracketed target ends at its first closing parenthesis, so `https://example.org/docs_(v2)` is read only up to `https://example.org/docs_(v2`; write a target that contains parentheses in angle brackets. A link reference definition or label that spans several lines, and an image used as a reference, are not handled.
+Links are read the way CommonMark reads them for the forms listed here. A link inside inline code, after an escaped bracket or in an image does not count, and a link in an ordered list, a target in angle brackets and a target with a title in double or single quotes do. A title in parentheses is not read, so `[Guide](https://example.org/guide (Overview))` is not collected. An unbracketed target ends at its first closing parenthesis, so `https://example.org/docs_(v2)` is read only up to `https://example.org/docs_(v2`; write a target that contains parentheses in angle brackets. A link reference definition or label that spans several lines, and an image used as a reference, are not handled. A quote escaped with a backslash inside a link title, as in `[A](https://example.org/a "Ti\"tle\"")`, is not recognised, so that link is dropped. A title that spans a line break is dropped the same way. Nested brackets in a link label, as in `[A [x] B](https://example.org/a)`, drop the link. A heading is an H2 only when `##` is followed by a space, so `##` followed by a tab is not read as one. A link in a four-space indented code block counts as a link, but such a block does not count as a file list under an H2.
 
 Some failures stop the file checks early. For example, an HTML response is reported as a failed plain-text check rather than parsed as Markdown.
 
@@ -84,7 +84,7 @@ These checks are labelled **v2** in the output. They look for link relations in
 | 9 | Home page points to its llms.txt | `rel="describedby"` has a non-empty target | `info` |
 | 10 | Home page points to a Markdown version | `rel="alternate"` has media type `text/markdown` and a non-empty target | `info` |
 
-The validator detects these declarations without fetching their targets. It reads the first 65,536 characters of the head, and when a relation is missing from a longer head the result says that only that part was read. A target is shown with its credentials and fragment removed and its query values masked. It also does not follow the links inside `llms.txt`, crawl the site or test whether an AI agent can complete a task there.
+The validator detects these declarations without fetching their targets. It reads the first 65,536 UTF-16 code units of the head after comments and the contents of script, style, title and template elements have been set aside, and when a relation is missing from a longer head the result says that only that part was read. A target is shown with its credentials and fragment removed and its query values masked. It also does not follow the links inside `llms.txt`, crawl the site or test whether an AI agent can complete a task there.
 
 ## Use from Node
 
@@ -136,7 +136,7 @@ For vulnerability reporting and supported versions, see [SECURITY.md](SECURITY.m
 
 The [hosted validator](https://turva.dev/llms-txt-validator) accepts any public domain in the browser. It requests the same two documents, `/llms.txt` and the home page, and runs the same checks. Its source is in the [turva.dev Cloudflare Worker](https://github.com/erekola/turva-worker). The hosted validator is the canonical implementation: if results diverge, this package is updated to match it.
 
-The hosted version also returns JSON with the same result shape for a completed validation. Its errors are shaped differently: an `error` field with HTTP status 400 and no `target`.
+The hosted version also returns JSON with the same result shape for a completed validation. Its errors are shaped differently: an `error` field with HTTP status 400 and no `target`. The site-wide rate limiter can also answer HTTP 429 as plain text, with a `Retry-After` header, before the validator runs, and that answer is not JSON.
 
 The JSON request:
 

package/bin/cli.mjs

@@ -2,7 +2,8 @@
 // CLI for turva-llms-txt-validator. Exit codes: 0 = valid (or valid with
 // warnings), 1 = not valid (or warnings with --strict), 2 = could not fetch
 // or bad input. The two v2 discovery checks and the input-path note carry status
-// "info" and move no exit code, by design. Same checks and JSON shape as the hosted validator:
+// "info" and move no exit code, by design. Same checks as the hosted validator, and the same
+// JSON shape for a completed validation:
 // curl -H "Accept: application/json" "https://turva.dev/llms-txt-validator?url=example.com"
 import { readFileSync } from "node:fs";
 import { fileURLToPath } from "node:url";
@@ -10,7 +11,7 @@ import { validateHost, stripControls, maskLocation } from "../src/index.mjs";
 
 const USAGE = [
   "usage: llms-txt-validate <domain-or-url> [--json] [--strict]",
-  "  --json     print the result as JSON (same shape as the hosted validator)",
+  "  --json     print the result as JSON (same shape as the hosted validator for a completed validation)",
   "  --strict   exit 1 on warnings too, for CI gates",
   "  --help     print this help",
   "  --version  print the installed version and exit"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "turva-llms-txt-validator",
-  "version": "0.3.13",
+  "version": "0.3.15",
   "description": "Validate a site's llms.txt structure: eight checks as pass/warn/fail plus two llms.txt v2 discovery checks that read pass or info and never move the summary, JSON output. Same checks as the hosted validator at turva.dev/llms-txt-validator.",
   "type": "module",
   "main": "./src/index.mjs",

package/src/index.mjs

@@ -63,11 +63,20 @@ export function maskLocation(href, base) {
   // path ("path@2x.png" stays unmasked below). But when an @ still appears before the first
   // /, the text ahead of it reads as userinfo even when the parser here does not recognise it
   // as a scheme (an underscore, a leading digit or space, a stray control character), so that
-  // shape is masked too instead of let through (VN1, round 3 Tek-542).
+  // shape is masked too instead of let through (VN1, round 3 Tek-542). With no slash at all
+  // the same holds when a ":" comes before the last @: "user_name:secret@example.com" is
+  // user:password and is masked, while "path@2x.png" has no ":" ahead of its @ and stays as
+  // given (outside review V13, D5-1: the slashless shape used to return early and print the
+  // password).
   if (!scheme && from - start < 2) {
     const firstSlash = s.indexOf("/");
     const firstAt = s.indexOf("@");
-    if (firstAt < 0 || firstSlash < 0 || firstAt > firstSlash) return s;
+    if (firstAt < 0) return s;
+    if (firstSlash < 0) {
+      if (!s.slice(0, s.lastIndexOf("@")).includes(":")) return s;
+    } else if (firstAt > firstSlash) {
+      return s;
+    }
   }
   // Nothing tells which @ of a refused target ends the user information, so everything up to the
   // last one is shown as ***. The mask is visible on purpose: three review rounds on 2026-09-22
@@ -702,7 +711,7 @@ export function validateLlmsTxt(f) {
   } else {
     add("size", "warn", "Small enough to be cheap to read", f.bytes + " bytes; consider moving detail to llms-full.txt");
   }
-  if (/<[a-z][a-z0-9-]*[\s>]/i.test(text)) {
+  if (/<\/?[a-z][a-z0-9-]*(?=[\s\/>])|<!--/i.test(text)) {
     add("no-html", "warn", "No HTML markup in the file", "HTML tags found; llms.txt should be plain markdown");
   } else {
     add("no-html", "pass", "No HTML markup in the file", "plain markdown");