turva-llms-txt-validator 0.3.17 → 0.3.19

package/README.md

@@ -71,7 +71,7 @@ Completed validation returns `{ target, summary, checks }`. Each check contains
 | 7 | File is small enough to read cheaply | none | Over 50 KB, or truncated at the 256 KB read limit |
 | 8 | File contains no HTML markup | none | An opening tag with its closing `>`, a closing tag or an HTML comment is found anywhere in the file, including inside code spans and fenced code. An attribute name is any run of characters other than white space, quotes, `<`, `>`, `/` and `=` that holds a Unicode letter, a Unicode digit or the character `_`, so `<div @click="x">` warns. Autolinks such as `<https://example.com>` and prose such as `a <b + c` are not tags. A tag still open at the end of the file, such as `<div class="x"`, has no closing `>` and does not warn |
 
-Links are read the way CommonMark reads them for the forms listed here. A link inside inline code, after an escaped bracket or in an image does not count, and a link in an ordered list, a target in angle brackets and a target with a title in double or single quotes do. A title in parentheses is not read, so `[Guide](https://example.org/guide (Overview))` is not collected. An unbracketed target ends at its first closing parenthesis, so `https://example.org/docs_(v2)` is read only up to `https://example.org/docs_(v2`. Write a target that contains parentheses in angle brackets. A link reference definition or label that spans several lines, and an image used as a reference, are not handled. A quote escaped with a backslash inside a link title, as in `[A](https://example.org/a "Ti\"tle\"")`, is not recognised, so that link is dropped. A title that spans a line break is dropped the same way. Nested brackets in a link label, as in `[A [x] B](https://example.org/a)`, drop the link. A heading is an H2 only when `##` is followed by a space, so `##` followed by a tab is not read as one. A link in a four-space indented code block counts as a link, but such a block does not count as a file list under an H2. A backslash escape in a target is resolved before the URL test, so `https\://example.org/guide` is read as `https://example.org/guide`. A link label cannot span a blank line. A link on a line that continues a list item, such as the line after a bare `-`, counts as that item's link. After a bare marker the next line counts only when it is indented to the item's content column, which is 2 for `-` and 3 for `1.`, and a blank line right after a bare marker ends the item. After a blank line inside an item, a line counts only when it is indented to the content column, where the item's text starts, so `-   item` needs 4 columns. A code fence indented less than that column ends the item. A fence indented to it keeps the item open, and the line after the fence then counts only when it is indented to that column too. CR and CRLF line endings are read as LF. A setext title, which is a text line underlined with `=`, is read only when it is a single line.
+Links are read the way CommonMark reads them for the forms listed here. A link inside inline code, after an escaped bracket or in an image does not count, and a link in an ordered list, a target in angle brackets and a target with a title in double or single quotes do. A title in parentheses is not read, so `[Guide](https://example.org/guide (Overview))` is not collected. An unbracketed target ends at its first closing parenthesis, so `https://example.org/docs_(v2)` is read only up to `https://example.org/docs_(v2`. Write a target that contains parentheses in angle brackets. A link reference definition or label that spans several lines, and an image used as a reference, are not handled. A quote escaped with a backslash inside a link title, as in `[A](https://example.org/a "Ti\"tle\"")`, is not recognised, so that link is dropped. A title that spans a line break is dropped the same way. Nested brackets in a link label, as in `[A [x] B](https://example.org/a)`, drop the link. A heading is an H2 when `##` is followed by a space or a tab, or when it is one line of text underlined with hyphens. The text line must follow a blank line or the start of the file. A list item above a line of hyphens is not an H2, and neither is a line of hyphens after a blank line. A link in a four-space indented code block counts as a link, but such a block does not count as a file list under an H2. A backslash escape in a target is resolved before the URL test, so `https\://example.org/guide` is read as `https://example.org/guide`. A link label cannot span a blank line. A link on a line that continues a list item, such as the line after a bare `-`, counts as that item's link. After a bare marker the next line counts only when it is indented to the item's content column, which is 2 for `-` and 3 for `1.`, and a blank line right after a bare marker ends the item. After a blank line inside an item, a line counts only when it is indented to the content column, where the item's text starts, so `-   item` needs 4 columns. A code fence indented less than that column ends the item. A fence indented to it keeps the item open, and the line after the fence then counts only when it is indented to that column too. CR and CRLF line endings are read as LF. A setext title, which is a text line underlined with `=`, is read only when it is a single line.
 
 Some failures stop the file checks early. For example, an HTML response is reported as a failed plain-text check rather than parsed as Markdown.
 
@@ -137,10 +137,10 @@ For vulnerability reporting and supported versions, see [SECURITY.md](https://gi
 GitHub Actions publishes every version from 0.1.3 on with npm trusted publishing, and each one carries a provenance attestation. To check one, install it in an empty directory and ask npm to verify the signatures. Replace the version with the one you want to check.
 
 ```sh
-mkdir verify-llms && cd verify-llms && npm init -y && npm install turva-llms-txt-validator@0.3.17 --ignore-scripts && npm audit signatures
+mkdir verify-llms && cd verify-llms && npm init -y && npm install turva-llms-txt-validator@0.3.19 --ignore-scripts && npm audit signatures
 ```
 
-`npm audit signatures` checks the registry signature and the provenance attestation of each installed package that has one. The attestation of this package names three things to compare with what you expect: the repository `github.com/erekola/llms-txt-validator`, the workflow file `.github/workflows/publish.yml` and the commit that produced the tarball. The npm version page shows them under Provenance. Tag `v0.3.17` must point at that same commit, and `gh api repos/erekola/llms-txt-validator/commits/v0.3.17 --jq .sha` prints it. A mismatch is a reason not to use that version. Provenance proves where a release was built and from which commit. It does not prove that the code is safe, so reading the source and the dependencies stays your job.
+`npm audit signatures` checks the registry signature and the provenance attestation of each installed package that has one. The attestation of this package names three things to compare with what you expect: the repository `github.com/erekola/llms-txt-validator`, the workflow file `.github/workflows/publish.yml` and the commit that produced the tarball. The npm version page shows them under Provenance. Tag `v0.3.19` must point at that same commit, and `gh api repos/erekola/llms-txt-validator/commits/v0.3.19 --jq .sha` prints it. A mismatch is a reason not to use that version. Provenance proves where a release was built and from which commit. It does not prove that the code is safe, so reading the source and the dependencies stays your job.
 
 ## Hosted version and source
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "turva-llms-txt-validator",
-  "version": "0.3.17",
+  "version": "0.3.19",
   "description": "Validate a site's llms.txt structure: eight checks as pass/warn/fail plus two llms.txt v2 discovery checks that read pass or info and never move the summary, JSON output. Same checks as the hosted validator at turva.dev/llms-txt-validator.",
   "type": "module",
   "main": "./src/index.mjs",

package/src/index.mjs

@@ -530,11 +530,70 @@ function decodeDestination(s) {
 // starts a block of its own (heading, blockquote, list item, fence) is not paragraph text, so it
 // never becomes a setext title. Only a one-line title is read; a title over several lines is not
 // (Tek-560).
-function isSetextH1(line, next) {
-  if (typeof next !== "string" || !/^ {0,3}=+[ \t]*$/.test(next)) return false;
+function isSetextText(line) {
   if (!/^ {0,3}\S/.test(line)) return false;
   return !/^ {0,3}(?:#|>|[-*+](?:[ \t]|$)|\d{1,9}[.)](?:[ \t]|$)|`{3,}|~{3,})/.test(line);
 }
+function isSetextH1(line, next) {
+  if (typeof next !== "string" || !/^ {0,3}=+[ \t]*$/.test(next)) return false;
+  return isSetextText(line);
+}
+
+// Lines that open a block of their own in CommonMark, so they are never setext text after the
+// title: a thematic break, the start of an HTML block (an autolink such as <https://...> is not
+// one, it stays paragraph text) and a link reference definition. Only the later-H1 reading skips
+// them; the title reading keeps its Tek-560 predicate. QA 2 of Tek-564 measured "---", "<div>" and
+// "[a]: url" over a run of "=" warning as a second H1 although 0.3.18 passed them.
+function startsOtherBlock(line) {
+  return /^ {0,3}([-*_])(?:[ \t]*\1){2,}[ \t]*$/.test(line)
+    || /^ {0,3}<(?:[A-Za-z][A-Za-z0-9-]*(?:[\s\/>]|$)|\/[A-Za-z][A-Za-z0-9-]*(?:[\s>]|$)|!--|\?|![A-Za-z]|!\[CDATA\[)/.test(line)
+    || /^ {0,3}\[[^\]]+\]:/.test(line);
+}
+
+// Setext H1 lines after the title (Tek-564). A second H1 written as one text line over a run of "="
+// is out of place exactly like "# Second title" is, and ends the section before it the same way.
+// The text line is read by isSetextH1, the predicate the first-title reading uses. It must start
+// a paragraph: a line that follows other paragraph text, a list item or a blockquote without a
+// blank line is a continuation line, so a title over several lines is not read as a heading, and
+// text that starts with "#" is not an H1 (isSetextText). The underline is returned separately
+// because it is not a heading and not content.
+function laterSetextH1(lines, fenced) {
+  const text = new Array(lines.length).fill(false);
+  const under = new Array(lines.length).fill(false);
+  let open = false;
+  for (let i = 0; i < lines.length; i++) {
+    const l = lines[i];
+    if (fenced[i] || l.trim() === "") { open = false; continue; }
+    if (!open && !fenced[i + 1] && !startsOtherBlock(l) && isSetextH1(l, lines[i + 1])) { text[i] = true; under[i + 1] = true; i++; continue; }
+    if (open && /^ {0,3}=+[ \t]*$/.test(l)) { open = false; continue; }
+    open = !(/^ {0,3}#{1,6}(?:[ \t]|$)/.test(l) || /^ {0,3}(?:-+|([-*_])(?: *\1){2,})[ \t]*$/.test(l));
+  }
+  return { text, under };
+}
+
+// Which lines are H2 headings, outside fences. One reader for every place that asks "is this an
+// H2": the section count, the section membership and the place where the late-summary scan stops.
+// An ATX H2 is "##" followed by a space or a tab (a bare "##" is not one, and "###" is deeper). A
+// setext H2 is one line of paragraph text, read by the same predicate as the setext H1 above,
+// that has a blank line or the start of the file before it and a run of hyphens directly after it.
+// A list item above "---", a thematic break and a line inside a fence are not H2 text. The
+// underline is returned separately because it is not content (Tek-562, outside review W30).
+function h2Lines(lines, fenced) {
+  const h2 = new Array(lines.length).fill(false);
+  const under = new Array(lines.length).fill(false);
+  for (let i = 0; i < lines.length; i++) {
+    if (fenced[i]) continue;
+    const l = lines[i];
+    if (/^ {0,3}##[ \t]/.test(l)) { h2[i] = true; continue; }
+    const next = lines[i + 1];
+    if (typeof next !== "string" || fenced[i + 1] || !/^ {0,3}-+[ \t]*$/.test(next)) continue;
+    if (!isSetextText(l) || /^ {0,3}([-*_])(?: *\1){2,} *$/.test(l)) continue;
+    if (i > 0 && lines[i - 1].trim() !== "") continue;
+    h2[i] = true;
+    under[i + 1] = true;
+  }
+  return { h2, under };
+}
 
 // The text of an ATX H1, or null when the line is not one. One to three leading spaces, a
 // single "#", then a space, a tab or the end of the line. A closing run of "#" preceded by a
@@ -662,10 +721,11 @@ export function validateLlmsTxt(f) {
     // v3.176.0). The status is warn either way, because the format places the summary directly
     // after the title.
     const fence = fenceMask(lines);
+    const lateH2 = h2Lines(lines, fence).h2;
     let late = -1;
     for (let i = h1End + 1; i < lines.length; i++) {
       if (fence[i]) continue;
-      if (/^ {0,3}## /.test(lines[i])) break;
+      if (lateH2[i]) break;
       if (/^ {0,3}>[ \t]*\S/.test(lines[i])) { late = i; break; }
     }
     add("summary", "warn", "Blockquote summary after the title", late === -1
@@ -706,7 +766,9 @@ export function validateLlmsTxt(f) {
       linkedLineIdx.add(idx);
     }
   }
-  const h2Count = lines.filter((l, i) => !fenced[i] && /^ {0,3}## /.test(l)).length;
+  const hm = h2Lines(lines, fenced);
+  const sx = laterSetextH1(lines, fenced);
+  const h2Count = hm.h2.filter(Boolean).length;
   // A section counts when it carries a file list. An H2 followed by a paragraph satisfied
   // this check until 2026-08-29, and the format puts each section's links in a list.
   // A heading between the title and the first H2, or a second H1 anywhere, is out of place,
@@ -729,8 +791,9 @@ export function validateLlmsTxt(f) {
         if (itemOpen) afterFence = true;
         continue;
       }
-      if (/^ {0,3}## /.test(l)) { inSection = true; counted = false; seenH2 = true; itemOpen = false; continue; }
-      const h1 = /^ {0,3}#(?:[ \t]|$)/.test(l);
+      if (hm.under[i] || sx.under[i]) continue;
+      if (hm.h2[i]) { inSection = true; counted = false; seenH2 = true; itemOpen = false; continue; }
+      const h1 = /^ {0,3}#(?:[ \t]|$)/.test(l) || sx.text[i];
       if (misplaced === -1 && i > firstIdx && (h1 || (!seenH2 && /^ {0,3}#{3,6}(?:[ \t]|$)/.test(l)))) misplaced = i;
       if (h1) { inSection = false; itemOpen = false; continue; }
       // A link on a line that continues an open list item counts as that item's link (Tek-560):