turva-llms-txt-validator 0.3.6 → 0.3.8

package/README.md

@@ -14,7 +14,7 @@ Requires **Node.js 18.17 or newer**. Run it without a global install:
 npx --yes turva-llms-txt-validator example.com
 ```
 
-Replace `example.com` with the domain you want to check. A domain, HTTP URL or HTTPS URL is accepted. Validation uses the host's `/llms.txt` and home page over HTTPS, regardless of the supplied path.
+Replace `example.com` with the domain you want to check. A domain, HTTP URL or HTTPS URL is accepted. Validation uses the host's `/llms.txt` and home page over HTTPS, regardless of the supplied path. If the address has a path other than `/` or `/llms.txt`, the result names that path first in an `info` line.
 
 For repeated use, install the CLI globally:
 
@@ -54,7 +54,7 @@ Use JSON for automation and strict mode when warnings should fail a CI step:
 npx --yes turva-llms-txt-validator example.com --json --strict
 ```
 
-Completed validation returns `{ target, summary, checks }`. Each check contains `{ id, status, label, detail }`. With `--json`, input and fetch errors return `{ "error": "..." }` and exit with code `2`.
+Completed validation returns `{ target, summary, checks }`. Each check contains `{ id, status, label, detail }`. With `--json`, input and fetch errors return `{ "error": "..." }` and exit with code `2`. An unknown option, a second domain and a missing domain are input errors too, and `--help` exits with code `0`.
 
 ## What it checks
 
@@ -63,14 +63,16 @@ Completed validation returns `{ target, summary, checks }`. Each check contains
 | # | Check | Failure | Warning |
 | --- | --- | --- | --- |
 | 1 | `/llms.txt` returns HTTP 200 | Non-200 response, rejected or excessive redirects | none |
-| 2 | Response is plain text | Body looks like an HTML page | Content type is neither `text/plain` nor `text/markdown` |
-| 3 | First non-empty line is a Markdown H1 | Missing H1, including a title indented as a code block | none |
-| 4 | Blockquote summary follows the title | none | Missing summary |
-| 5 | H2 sections group the content | none | No H2 sections, or no section contains a Markdown link list |
-| 6 | Markdown links have names and absolute HTTP or HTTPS targets | none | Missing links, empty names, relative targets or unsupported targets |
+| 2 | Response is plain text | Body looks like an HTML page | The media type, read without its parameters, is neither `text/plain` nor `text/markdown` |
+| 3 | First non-empty line is a Markdown H1 | Missing H1, a title indented as a code block, or an H1 with no text | none |
+| 4 | Blockquote summary follows the title | none | Missing summary, or a blockquote after other text but before the first H2, named by its line |
+| 5 | H2 sections group the content | none | No H2 sections, no section with a Markdown link list, a heading between the title and the first H2, or a second H1 |
+| 6 | Markdown links have names and absolute HTTP or HTTPS targets | none | Missing links, empty names, relative targets or targets that do not parse as HTTP or HTTPS URLs |
 | 7 | File is small enough to read cheaply | none | Over 50 KB, or truncated at the 256 KB read limit |
 | 8 | File contains no HTML markup | none | HTML tags found |
 
+Links are read the way CommonMark reads them. A link inside inline code, after an escaped bracket or in an image does not count, and a link in an ordered list, a target in angle brackets and a target with a title do.
+
 Some failures stop the file checks early. For example, an HTML response is reported as a failed plain-text check rather than parsed as Markdown.
 
 ### Home-page discovery
@@ -82,7 +84,7 @@ These checks are labelled **v2** in the output. They look for link relations in
 | 9 | Home page points to its llms.txt | `rel="describedby"` has a non-empty target | `info` |
 | 10 | Home page points to a Markdown version | `rel="alternate"` has media type `text/markdown` and a non-empty target | `info` |
 
-The validator detects these declarations without fetching their targets. It also does not follow the links inside `llms.txt`, crawl the site or test whether an AI agent can complete a task there.
+The validator detects these declarations without fetching their targets. It reads the first 65,536 characters of the head, and when a relation is missing from a longer head the result says that only that part was read. A target is shown with its credentials and fragment removed and its query values masked. It also does not follow the links inside `llms.txt`, crawl the site or test whether an AI agent can complete a task there.
 
 ## Use from Node
 
@@ -124,7 +126,7 @@ The default validation requests two documents: `https://<host>/llms.txt` and `ht
 - Timeout: eight seconds per document, shared across its redirect chain.
 - Read limit: 256 KB per response.
 - Redirects: up to four hops, over HTTPS, to the same host or its `www`/apex equivalent. Off-site redirects, embedded credentials and unsupported ports are rejected. A redirect target is reported with its credentials and fragment removed and its query values masked, whether the redirect was followed or rejected.
-- Host checks: IP literals, bracketed IPv6 addresses, localhost and the internal-use TLDs `local`, `internal`, `home`, `lan`, `corp`, `test`, `invalid`, `arpa` and `onion` are rejected before a request is sent. One trailing dot on the host you give, as in `example.com.`, is removed first. Redirect targets are checked too.
+- Host checks: IP literals, bracketed IPv6 addresses, localhost and the internal-use TLDs `localdomain`, `local`, `internal`, `home`, `lan`, `corp`, `test`, `invalid`, `arpa` and `onion` are rejected before a request is sent. An address that holds `@`, `?` or `#` is not repeated in the error message, because it can carry a user name, a password or a query value. One trailing dot on the host you give, as in `example.com.`, is removed first. Redirect targets are checked too.
 
 **DNS resolution is not checked for private addresses.** A syntactically public domain can resolve to a private IP, so if you expose this package through a service that accepts untrusted domains, enforce private-address restrictions at the network layer. Local and CI runs use their own network policy. The hosted validator runs on the Cloudflare edge.
 

package/bin/cli.mjs

@@ -1,41 +1,67 @@
 #!/usr/bin/env node
 // CLI for turva-llms-txt-validator. Exit codes: 0 = valid (or valid with
 // warnings), 1 = not valid (or warnings with --strict), 2 = could not fetch
-// or bad input. The two v2 discovery checks carry status "info" and move no
-// exit code, by design. Same checks and JSON shape as the hosted validator:
+// or bad input. The two v2 discovery checks and the input-path note carry status
+// "info" and move no exit code, by design. Same checks and JSON shape as the hosted validator:
 // curl -H "Accept: application/json" "https://turva.dev/llms-txt-validator?url=example.com"
-import { validateHost } from "../src/index.mjs";
+import { validateHost, stripControls } from "../src/index.mjs";
 
+const USAGE = [
+  "usage: llms-txt-validate <domain-or-url> [--json] [--strict]",
+  "  --json    print the result as JSON (same shape as the hosted validator)",
+  "  --strict  exit 1 on warnings too, for CI gates",
+  "  --help    print this help"
+];
+const KNOWN = new Set(["--json", "--strict", "--help", "-h"]);
+
+// The arguments are read strictly since 0.3.8 (Tek-496). An unknown flag such as a misspelled
+// --strcit used to be ignored, so a CI step ran without the gate it asked for, and a second
+// target was dropped without a word. Both now stop with exit 2, and every input error is JSON
+// when --json was given, the missing target included.
 const args = process.argv.slice(2);
-const flags = new Set(args.filter((a) => a.startsWith("--")));
-const target = args.find((a) => !a.startsWith("--"));
+const json = args.includes("--json");
+const flags = args.filter((a) => a.startsWith("-") && a !== "-");
+const targets = args.filter((a) => !(a.startsWith("-") && a !== "-"));
+
+function fail(message) {
+  if (json) console.log(JSON.stringify({ error: message }, null, 2));
+  else {
+    console.error("error: " + message);
+    console.error(USAGE[0]);
+  }
+  process.exit(2);
+}
 
-if (!target || flags.has("--help")) {
-  console.log("usage: llms-txt-validate <domain-or-url> [--json] [--strict]");
-  console.log("  --json    print the result as JSON (same shape as the hosted validator)");
-  console.log("  --strict  exit 1 on warnings too, for CI gates");
-  process.exit(target ? 0 : 2);
+if (flags.includes("--help") || flags.includes("-h")) {
+  for (const line of USAGE) console.log(line);
+  process.exit(0);
 }
+const unknown = flags.find((f) => !KNOWN.has(f));
+if (unknown) fail("unknown option " + JSON.stringify(stripControls(unknown).slice(0, 40)));
+if (targets.length === 0 || targets[0].trim() === "") fail("no domain or URL given");
+if (targets.length > 1) fail("one domain or URL at a time, got " + targets.length);
 
 const mark = { pass: "ok  ", warn: "warn", fail: "FAIL", info: "info" };
 try {
-  const result = await validateHost(target);
-  if (flags.has("--json")) {
+  const result = await validateHost(targets[0]);
+  if (json) {
     console.log(JSON.stringify(result, null, 2));
   } else {
+    // Details reach this point without control characters (validateHost strips them), so a
+    // published link cannot write an escape sequence to the terminal.
     console.log(result.target);
     for (const c of result.checks) console.log("  " + mark[c.status] + "  " + c.label + " (" + c.detail + ")");
     console.log("result: " + result.summary);
   }
   if (result.summary === "not valid") process.exit(1);
-  if (result.summary === "valid with warnings" && flags.has("--strict")) process.exit(1);
+  if (result.summary === "valid with warnings" && flags.includes("--strict")) process.exit(1);
   process.exit(0);
 } catch (err) {
-  const message = err && err.message ? err.message : String(err);
+  const message = stripControls(err && err.message ? err.message : String(err));
   // The hosted validator answers every failure as JSON when JSON was asked for, so a CI
   // step that parses --json output gets {"error": ...} here too instead of an empty stdout
   // and a plain-text stderr (round 16 S3-3). The exit code is unchanged.
-  if (flags.has("--json")) console.log(JSON.stringify({ error: message }, null, 2));
+  if (json) console.log(JSON.stringify({ error: message }, null, 2));
   else console.error("error: " + message);
   process.exit(2);
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "turva-llms-txt-validator",
-  "version": "0.3.6",
+  "version": "0.3.8",
   "description": "Validate a site's llms.txt structure: eight checks as pass/warn/fail plus two llms.txt v2 discovery checks that read pass or info and never move the summary, JSON output. Same checks as the hosted validator at turva.dev/llms-txt-validator.",
   "type": "module",
   "main": "./src/index.mjs",

package/src/index.mjs

@@ -71,6 +71,14 @@ export function stripBidi(s) {
   return String(s).replace(/[\u202A-\u202E\u2066-\u2069]/g, "");
 }
 
+// C0 and C1 control characters leave a check's detail as well. JSON escapes them, but a program
+// that prints a detail to a terminal passes ESC through, so a published link could carry a
+// sequence that clears the screen (Tek-496).
+// Mirrors worker.js stripControls().
+export function stripControls(s) {
+  return String(s).replace(/[\u0000-\u001F\u007F-\u009F]/g, "");
+}
+
 export function normalizeHostInput(raw) {
   let s = String(raw || "").trim().toLowerCase();
   if (!s) return null;
@@ -86,6 +94,21 @@ export function normalizeHostInput(raw) {
   return typedHost.length > 1 && typedHost.endsWith(".") ? typedHost.slice(0, -1) : typedHost;
 }
 
+// The path of the address as typed, when it is neither the root nor /llms.txt. validateHost keeps its
+// reads pinned to /llms.txt and the home page, because a fixed path is part of the guard against use
+// as a fetch proxy, so a typed /docs/llms.txt used to be dropped without a word while the result
+// described the root file (outside retest 2026-09-26). The result now names the unused path in an
+// info check. The case of the path is kept, and the query and fragment are never returned.
+// Mirrors worker.js enteredPath() (v3.176.0).
+export function enteredPath(raw) {
+  let s = String(raw || "").trim();
+  if (!/^[a-z][a-z0-9+.-]*:\/\//i.test(s)) s = "https://" + s;
+  let u;
+  try { u = new URL(s); } catch { return ""; }
+  const path = u.pathname;
+  return path === "/" || path === "" || path === "/llms.txt" ? "" : path;
+}
+
 export function isValidPublicHost(host) {
   if (!host || host.length > 253) return false;
   if (host.startsWith("[") || /^\d+\.\d+\.\d+\.\d+$/.test(host)) return false;
@@ -93,7 +116,9 @@ export function isValidPublicHost(host) {
   const tld = host.split(".").pop();
   // arpa and onion since round 19 (P3): home.arpa is the home-network name of RFC 8375,
   // in-addr.arpa and ip6.arpa name addresses, and an onion name never resolves on the public web.
-  if (["localhost", "local", "internal", "home", "lan", "corp", "test", "invalid", "arpa", "onion"].includes(tld)) return false;
+  // localdomain since 2026-09-27 (Tek-496): localhost.localdomain is the loopback alias that many
+  // hosts files carry, and it passed this list although the name never resolves on the public web.
+  if (["localhost", "localdomain", "local", "internal", "home", "lan", "corp", "test", "invalid", "arpa", "onion"].includes(tld)) return false;
   return true;
 }
 
@@ -170,28 +195,125 @@ export async function fetchLlmsTxt(host, opts = {}) {
   }
 }
 
-// Every markdown link in the file, scanned once from left to right instead of collected with
-// matchAll(/\[([^\][]*)\]\(([^)\s]{1,2048})\)/g). That bound meant a target longer than 2048
-// characters was not counted as a link at all, and dropping the bound from the pattern would
-// make it quadratic on a file that repeats "[a](" (Erik 2026-08-29). The scan carries no bound
-// and no backtracking: every character is read once and the furthest failed target scan is
-// remembered. matchAll resumes after a whole match, and so does this.
+// A backslash escapes the character after it when an odd number of backslashes precede it.
+function isEscaped(s, i) {
+  let n = 0;
+  for (let b = i - 1; b >= 0 && s[b] === "\\"; b--) n++;
+  return n % 2 === 1;
+}
+
+// Inline code spans of one line, marked per character, so a link written inside backticks is
+// read as the code it is (outside audit 2026-09-26, Tek-496). A span opens with a run of
+// backticks and closes with the next run of the same length on the same line. An escaped first
+// backtick is literal and the rest of its run still opens, while a closing run is never escaped,
+// because a backslash inside a code span is literal. Runs are grouped by length and every group
+// is read with one forward pointer, so the scan stays linear on a line of unmatched runs.
+function codeSpanMask(line) {
+  const mask = new Uint8Array(line.length);
+  const runs = [];
+  for (let i = 0; i < line.length; ) {
+    if (line[i] !== "`") { i++; continue; }
+    let j = i;
+    while (j < line.length && line[j] === "`") j++;
+    runs.push({ at: i, len: j - i, escaped: isEscaped(line, i) });
+    i = j;
+  }
+  const byLen = new Map();
+  runs.forEach((r, k) => { if (!byLen.has(r.len)) byLen.set(r.len, []); byLen.get(r.len).push(k); });
+  const ptr = new Map();
+  for (let k = 0; k < runs.length; k++) {
+    const r = runs[k];
+    const len = r.escaped ? r.len - 1 : r.len;
+    const list = len > 0 ? byLen.get(len) : undefined;
+    if (!list) continue;
+    let p = ptr.get(len) || 0;
+    while (p < list.length && list[p] <= k) p++;
+    ptr.set(len, p);
+    if (p === list.length) continue;
+    const close = runs[list[p]];
+    mask.fill(1, r.escaped ? r.at + 1 : r.at, close.at + close.len);
+    k = list[p];
+  }
+  return mask;
+}
+
+// Every markdown link in the prose, scanned once from left to right. A "[" inside a code span,
+// an escaped "\[" and the "![" of an image open no link, because none of them is a hyperlink
+// (all three counted until the outside audit of 2026-09-26, Tek-496). A target may be written in angle
+// brackets and may carry a title in quotes, as CommonMark allows; both were missed or misread
+// before. The scan carries no bound and no backtracking: the furthest failed plain target is
+// remembered as before, and the closing ">", quote and line end are looked up through a cache
+// that only moves forward, so a file that repeats "[a](" or "[a](x \"" stays linear.
 function collectLinks(text) {
   const out = [];
+  const code = new Uint8Array(text.length);
+  {
+    let at = 0;
+    for (const line of text.split("\n")) {
+      code.set(codeSpanMask(line), at);
+      at += line.length + 1;
+    }
+  }
+  const cache = Object.create(null);
+  const nextIdx = (c, from) => {
+    const h = cache[c];
+    if (h && h.from <= from && (h.at === -1 || h.at >= from)) return h.at;
+    const at = text.indexOf(c, from);
+    cache[c] = { from, at };
+    return at;
+  };
+  const isBlank = (c) => c === " " || c === "\t";
+  // Spaces, tabs and at most one line ending may stand around the destination and the title, as
+  // CommonMark allows; "[a](  b  )" and a ")" on the next line were missed until the review of
+  // 2026-09-27. A blank line still ends the link.
+  const skipSpace = (p) => {
+    while (p < text.length && isBlank(text[p])) p++;
+    if (text[p] === "\n") { p++; while (p < text.length && isBlank(text[p])) p++; }
+    return p;
+  };
+  const afterTarget = (target, p) => {
+    const q = skipSpace(p);
+    if (text[q] === ")") return { target, end: q };
+    const quote = text[q];
+    if (q > p && (quote === '"' || quote === "'")) {
+      const close = nextIdx(quote, q + 1);
+      const nl = nextIdx("\n", q + 1);
+      if (close === -1 || (nl !== -1 && nl < close)) return null;
+      const r = skipSpace(close + 1);
+      if (text[r] === ")") return { target, end: r };
+    }
+    return null;
+  };
   let failEnd = -1;
+  const readTarget = (k) => {
+    k = skipSpace(k);
+    if (text[k] === "<") {
+      const gt = nextIdx(">", k + 1);
+      if (gt === -1 || gt === k + 1) return null;
+      const lt = nextIdx("<", k + 1);
+      const nl = nextIdx("\n", k + 1);
+      if ((lt !== -1 && lt < gt) || (nl !== -1 && nl < gt)) return null;
+      return afterTarget(text.slice(k + 1, gt), gt + 1);
+    }
+    if (k <= failEnd) return null;
+    let e = k;
+    while (e < text.length && text[e] !== ")" && !/\s/.test(text[e])) e++;
+    if (e === k) return null;
+    if (text[e] === ")") return { target: text.slice(k, e), end: e };
+    const found = e < text.length ? afterTarget(text.slice(k, e), e) : null;
+    if (!found) failEnd = e;
+    return found;
+  };
   for (let i = 0; i < text.length; i++) {
-    if (text[i] !== "[") continue;
+    if (text[i] !== "[" || code[i] || isEscaped(text, i)) continue;
+    if (i > 0 && text[i - 1] === "!" && !isEscaped(text, i - 1)) continue;
     let j = i + 1;
-    while (j < text.length && text[j] !== "]" && text[j] !== "[") j++;
+    while (j < text.length && text[j] !== "]" && text[j] !== "[") j += text[j] === "\\" ? 2 : 1;
     if (j >= text.length) break;
     if (text[j] === "[") { i = j - 1; continue; }
     if (text[j + 1] !== "(") { i = j; continue; }
-    const k = j + 2;
-    if (k <= failEnd) { i = j + 1; continue; }
-    let e = k;
-    while (e < text.length && text[e] !== ")" && !/\s/.test(text[e])) e++;
-    if (e > k && text[e] === ")") { out.push({ name: text.slice(i + 1, j), target: text.slice(k, e) }); i = e; continue; }
-    if (text[e] !== ")") failEnd = e;
+    const t = readTarget(j + 2);
+    if (t) { out.push({ name: text.slice(i + 1, j), target: t.target }); i = t.end; continue; }
     i = j + 1;
   }
   return out;
@@ -232,39 +354,51 @@ function fenceMask(lines) {
   }
   return mask;
 }
-// A markdown list item that carries a link, scanned once from left to right instead of
-// matched with /^ {0,3}[-*+] .*\[[^\][]*\]\([^)\s]+\)/. That pattern is quadratic on a line
-// such as "- " followed by "[a](" repeated, because every candidate rescans the target to the
-// end of the line, and the line comes from the audited site (CodeQL js/polynomial-redos,
-// 2026-08-29). Bounding the quantifier would trade the speed bug for a silent accuracy bug,
-// so the scan is by index: every character is read once and the furthest failed target scan
-// is remembered.
+// A markdown list item that carries a link. A bullet or an ordered marker (1. or 1)) followed by
+// a space or a tab opens the item; an ordered list and a tab after the marker were not read as
+// list items until 2026-09-27 (Tek-496). The link is found by collectLinks, so code, escapes
+// and images count here exactly as they do in the links check. A bare CR, U+2028 and U+2029 stay
+// inside a line after split(/\r?\n/), and a link behind one has never counted, so the item is read
+// up to the first of them.
 function listItemHasLink(l) {
-  const m = /^ {0,3}[-*+] /.exec(l);
+  const m = /^ {0,3}(?:[-*+]|\d{1,9}[.)])[ \t]/.exec(l);
   if (!m) return false;
-  const isSep = (c) => c === "\r" || c === "\n" || c === "\u2028" || c === "\u2029";
-  let failEnd = -1;
-  for (let i = m[0].length; i < l.length; i++) {
-    // A "." in the old pattern never crosses a line terminator, and split(/\r?\n/) leaves
-    // a bare CR, U+2028 and U+2029 inside a line, so a link behind one was not a match then
-    // and is not one now.
-    if (isSep(l[i])) return false;
-    if (l[i] !== "[") continue;
-    let j = i + 1, sep = false;
-    while (j < l.length && l[j] !== "]" && l[j] !== "[") { if (isSep(l[j])) sep = true; j++; }
-    if (j >= l.length) return false;
-    if (l[j] === "[") { if (sep) return false; i = j - 1; continue; }
-    if (l[j + 1] !== "(") { if (sep) return false; i = j; continue; }
-    const k = j + 2;
-    if (k <= failEnd) { if (sep) return false; i = j + 1; continue; }
-    let e = k;
-    while (e < l.length && l[e] !== ")" && !/\s/.test(l[e])) e++;
-    if (e > k && l[e] === ")") return true;
-    if (sep) return false;
-    if (l[e] !== ")") failEnd = e;
-    i = j + 1;
-  }
-  return false;
+  const sep = l.slice(m[0].length).search(/[\r\u2028\u2029]/);
+  const body = sep === -1 ? l.slice(m[0].length) : l.slice(m[0].length, m[0].length + sep);
+  return collectLinks(body).length > 0;
+}
+
+// The text of an ATX H1, or null when the line is not one. One to three leading spaces, a
+// single "#", then a space, a tab or the end of the line. A closing run of "#" preceded by a
+// space or a tab is not part of the title, and the title is what remains once both ends are
+// trimmed. Until 2026-09-27 the check demanded exactly one space and a visible character after
+// it, so "#  Example" and "#\tExample" failed and "# ###", a heading with no text, passed
+// (Tek-496). The closing run is read by index, not by a pattern, so a long line of
+// spaces and "#" stays linear.
+function h1Title(line) {
+  const m = /^ {0,3}#(?=[ \t]|$)/.exec(line);
+  if (!m) return null;
+  const s = line.slice(m[0].length);
+  let end = s.length;
+  while (end > 0 && (s[end - 1] === " " || s[end - 1] === "\t")) end--;
+  let h = end;
+  while (h > 0 && s[h - 1] === "#") h--;
+  if (h < end && (h === 0 || s[h - 1] === " " || s[h - 1] === "\t")) end = h;
+  let start = 0;
+  while (start < end && (s[start] === " " || s[start] === "\t")) start++;
+  while (end > start && (s[end - 1] === " " || s[end - 1] === "\t")) end--;
+  return s.slice(start, end);
+}
+
+// An absolute link target an agent can follow: it starts with http:// or https:// and the URL
+// parser reads it with a host. Until 2026-09-27 a pattern accepted anything after the scheme,
+// so https://% and https://example.com:99999/ passed as absolute URLs although no client can
+// open them (Tek-496).
+function isAbsoluteHttpUrl(t) {
+  if (!/^https?:\/\//i.test(t)) return false;
+  let u;
+  try { u = new URL(t); } catch { return false; }
+  return (u.protocol === "http:" || u.protocol === "https:") && u.hostname !== "";
 }
 
 function redirectFailDetail(f) {
@@ -295,8 +429,11 @@ export function validateLlmsTxt(f) {
     add("content-type", "fail", "Response is plain text", "the body looks like an HTML page, not an llms.txt file");
     return checks;
   }
-  if (ct.includes("text/plain") || ct.includes("text/markdown")) {
-    add("content-type", "pass", "Response is plain text", ct.split(";")[0]);
+  // The media type is compared whole, without its parameters. A substring test passed
+  // application/x-text/plain and text/markdownish until 2026-09-27 (Tek-496).
+  const mediaType = ct.split(";")[0].trim();
+  if (mediaType === "text/plain" || mediaType === "text/markdown") {
+    add("content-type", "pass", "Response is plain text", mediaType);
   } else {
     add("content-type", "warn", "Response is plain text", "content-type is " + (ct.split(";")[0] || "missing") + ", text/plain or text/markdown is the convention");
   }
@@ -307,8 +444,12 @@ export function validateLlmsTxt(f) {
   // The line is read as markdown and not trimmed first. Four spaces or a tab make it an
   // indented code block rather than a heading, and trimming erased that difference, so
   // "    # Site" passed as the H1 until 2026-08-29. CommonMark allows three spaces.
-  if (/^ {0,3}# \S/.test(firstRaw)) {
+  // h1Title reads the heading the way CommonMark does, its closing run included (Tek-496).
+  const title = h1Title(firstRaw);
+  if (title) {
     add("h1-title", "pass", "Starts with an H1 title", JSON.stringify(cut(first, 80)));
+  } else if (title === "") {
+    add("h1-title", "fail", "Starts with an H1 title", "the H1 has no text; the format requires the project name there");
   } else {
     add("h1-title", "fail", "Starts with an H1 title", "the first non-empty line should be a markdown H1 (# Site name)");
   }
@@ -316,7 +457,20 @@ export function validateLlmsTxt(f) {
   if (afterH1.trim().startsWith("> ")) {
     add("summary", "pass", "Blockquote summary after the title", JSON.stringify(cut(afterH1.trim(), 80)));
   } else {
-    add("summary", "warn", "Blockquote summary after the title", "recommended by the format (> one-line summary), not required");
+    // A blockquote further down, before the first H2, is a summary in the wrong place and not a
+    // missing one, and the detail says which (outside retest 2026-09-26, mirrored from worker.js
+    // v3.176.0). The status is warn either way, because the format places the summary directly
+    // after the title.
+    const fence = fenceMask(lines);
+    let late = -1;
+    for (let i = firstIdx + 1; i < lines.length; i++) {
+      if (fence[i]) continue;
+      if (/^ {0,3}## /.test(lines[i])) break;
+      if (/^ {0,3}> /.test(lines[i])) { late = i; break; }
+    }
+    add("summary", "warn", "Blockquote summary after the title", late === -1
+      ? "recommended by the format (> one-line summary), not required"
+      : "the blockquote at line " + (late + 1) + " comes after other text, and the format places the summary directly after the title");
   }
   // Headings are read outside fences only, and with the same indentation the H1 check and
   // listItemHasLink have allowed since 2026-08-29. Until 2026-09-10 this one line still
@@ -326,14 +480,19 @@ export function validateLlmsTxt(f) {
   const h2Count = lines.filter((l, i) => !fenced[i] && /^ {0,3}## /.test(l)).length;
   // A section counts when it carries a file list. An H2 followed by a paragraph satisfied
   // this check until 2026-08-29, and the format puts each section's links in a list.
-  let sectionsWithList = 0;
+  // A heading between the title and the first H2, or a second H1 anywhere, is out of place,
+  // because the format has one H1 and allows only paragraphs and lists before the file lists.
+  // Both passed until 2026-09-27 (Tek-496). The first one is named in the detail.
+  let sectionsWithList = 0, misplaced = -1;
   {
-    let inSection = false, counted = false;
+    let inSection = false, counted = false, seenH2 = false;
     for (let i = 0; i < lines.length; i++) {
       const l = lines[i];
       if (fenced[i]) continue;
-      if (/^ {0,3}## /.test(l)) { inSection = true; counted = false; continue; }
-      if (/^ {0,3}# /.test(l)) { inSection = false; continue; }
+      if (/^ {0,3}## /.test(l)) { inSection = true; counted = false; seenH2 = true; continue; }
+      const h1 = /^ {0,3}#(?:[ \t]|$)/.test(l);
+      if (misplaced === -1 && i > firstIdx && (h1 || (!seenH2 && /^ {0,3}#{3,6}(?:[ \t]|$)/.test(l)))) misplaced = i;
+      if (h1) { inSection = false; continue; }
       if (inSection && !counted && listItemHasLink(l)) { sectionsWithList++; counted = true; }
     }
   }
@@ -344,6 +503,11 @@ export function validateLlmsTxt(f) {
   } else {
     add("sections", "warn", "H2 sections group the content", "no H2 sections found; sections are the convention for grouping links");
   }
+  if (misplaced !== -1) {
+    const s = checks[checks.length - 1];
+    s.status = "warn";
+    s.detail += "; the heading at line " + (misplaced + 1) + " is out of place, because the format has one H1 and no other heading before the first H2";
+  }
   // Links are collected from the prose only, for the same reason the headings are: a link
   // shown inside a code fence is an example of a link, not one an agent can follow.
   const links = collectLinks(lines.filter((l, i) => !fenced[i]).join("\n"));
@@ -351,7 +515,7 @@ export function validateLlmsTxt(f) {
   // bare "https://" both counted as valid absolute links until 2026-08-29.
   const named = links.filter((m) => m.name.trim() !== "");
   const unnamed = links.length - named.length;
-  const absolute = named.filter((m) => /^https?:\/\/[^/\s?#]+/.test(m.target)).length;
+  const absolute = named.filter((m) => isAbsoluteHttpUrl(m.target)).length;
   if (links.length === 0) {
     add("links", "warn", "Markdown links an agent can follow", "no markdown links found");
   } else if (unnamed > 0) {
@@ -361,7 +525,7 @@ export function validateLlmsTxt(f) {
   } else {
     const relativeCount = named.filter((m) => !/^[a-z][a-z0-9+.-]*:/i.test(m.target)).length;
     const hostless = named.length - absolute - relativeCount;
-    add("links", "warn", "Markdown links an agent can follow", named.length + " links, " + relativeCount + " relative" + (hostless > 0 ? " and " + hostless + " with a scheme but no host" : "") + "; absolute URLs travel better when the file is read out of context");
+    add("links", "warn", "Markdown links an agent can follow", named.length + " links, " + relativeCount + " relative" + (hostless > 0 ? " and " + hostless + " that are not valid http or https URLs" : "") + "; absolute URLs travel better when the file is read out of context");
   }
   if (f.truncated) {
     add("size", "warn", "Small enough to be cheap to read", "over 256 KB, read truncated");
@@ -1195,7 +1359,8 @@ export function findLinkRelations(html, linkHeader) {
   // The cap applies after the head is collected, so it bounds this tag scan and not the head
   // parse, and a relation declared past it is not found here. The Link response header is
   // read separately below.
-  const head = headOfDocument(html).slice(0, 65536);
+  const whole = headOfDocument(html);
+  const head = whole.slice(0, 65536);
   for (const tag of htmlTags(head)) {
     // The name has to END at "link": a real parser reads "<link<link" as ONE tag whose
     // NAME is "link<link", not as a link element, so \b would count a relation the site
@@ -1225,6 +1390,9 @@ export function findLinkRelations(html, linkHeader) {
     if (!found.describedby && href && rel.includes("describedby")) found.describedby = href;
     if (!found.markdown && href && rel.includes("alternate") && isMarkdownHeader) found.markdown = href;
   }
+  // A head longer than the cap was read in part, so a relation missing from that part is
+  // reported as missing from the part read, not from the page (Tek-496).
+  if (whole.length > 65536) found.headCut = true;
   return found;
 }
 
@@ -1236,10 +1404,14 @@ export function validateV2Discovery(found, unreadReason) {
     add("v2-markdown-alternate", "info", "Home page points to a markdown version (v2)", unreadReason);
     return checks;
   }
+  // A target is shown through maskLocation, as a redirect target is, so a user name, a password,
+  // a query value or a fragment in a published link stays out of the report; until 2026-09-27 it
+  // was only cut (Tek-496). A relative target stays relative.
+  const where = found.headCut ? "in the first 65,536 characters of the head, the part that was read, or in the Link header" : "in the head or the Link header";
   add("v2-describedby", found.describedby ? "pass" : "info", "Home page points to its llms.txt (v2)",
-    found.describedby ? 'rel="describedby" to ' + cut(found.describedby, 120) : 'no rel="describedby" in the head or the Link header; v2 recommends it so an agent finds the file without guessing');
+    found.describedby ? 'rel="describedby" to ' + cut(maskLocation(found.describedby), 120) : 'no rel="describedby" ' + where + '; v2 recommends it so an agent finds the file without guessing');
   add("v2-markdown-alternate", found.markdown ? "pass" : "info", "Home page points to a markdown version (v2)",
-    found.markdown ? 'rel="alternate" type="text/markdown" to ' + cut(found.markdown, 120) : 'no rel="alternate" type="text/markdown" in the head or the Link header; v2 recommends it so an agent finds the markdown form without guessing');
+    found.markdown ? 'rel="alternate" type="text/markdown" to ' + cut(maskLocation(found.markdown), 120) : 'no rel="alternate" type="text/markdown" ' + where + '; v2 recommends it so an agent finds the markdown form without guessing');
   return checks;
 }
 
@@ -1252,7 +1424,13 @@ export function summarizeChecks(checks) {
 export async function validateHost(input, opts = {}) {
   const host = normalizeHostInput(input);
   if (!host || !isValidPublicHost(host)) {
-    throw new Error("not a public domain name: " + cut(input, 120));
+    // The input is repeated only when it cannot carry a secret. An entry with an @, a ? or a #
+    // can hold a user name, a password or a query value, and until 0.3.8 the message copied
+    // its first 120 characters into stderr and the --json error (Tek-496). Such an entry is
+    // named by the host the parser reads from it, or not at all, as the hosted form does.
+    const raw = String(input ?? "");
+    const shown = /[@?#]/.test(raw) ? normalizeHostInput(raw) || "" : stripControls(cut(raw, 120));
+    throw new Error(shown ? "not a public domain name: " + shown : "not a public domain name");
   }
   // opts comes from the caller, and path and accept exist for this function's own
   // second read. Forwarding them to the first read would let a caller point the
@@ -1270,9 +1448,13 @@ export async function validateHost(input, opts = {}) {
     discovery = validateV2Discovery(null, "the home page could not be read, so this was not measured");
   }
   const checks = validateLlmsTxt(fetched).concat(discovery);
+  // A path in the typed address is not used, and the result says so first (mirrored from worker.js
+  // v3.176.0). Information moves no summary and no exit code.
+  const unusedPath = enteredPath(input);
+  if (unusedPath) checks.unshift({ id: "input-path", status: "info", label: "Path in the address you entered", detail: cut(unusedPath, 120) + " is not used, because the validator always reads /llms.txt at the root of the host" });
   // Text quoted from the fetched file leaves without bidirectional controls (round 19,
   // V6-U1, mirrored from worker.js v3.164.0): a site's own H1 or summary could otherwise
   // read in another order here than in the file itself.
-  const stripped = checks.map((c) => (typeof c.detail === "string" ? { ...c, detail: stripBidi(c.detail) } : c));
+  const stripped = checks.map((c) => (typeof c.detail === "string" ? { ...c, detail: stripControls(stripBidi(c.detail)) } : c));
   return { target: "https://" + host + "/llms.txt", summary: summarizeChecks(stripped), checks: stripped };
 }