turva-llms-txt-validator 0.1.8 → 0.2.0

package/README.md

@@ -1,6 +1,6 @@
 # turva-llms-txt-validator
 
-Validate a site's llms.txt structure from the command line, from Node or in CI. Eight checks, each reported as pass, warn or fail, with a one-line detail. No score on purpose: a structure check can say what is there and what is missing, and a number on top of eight checks would look like an agent-readiness score without measuring one.
+Validate a site's llms.txt structure from the command line, from Node or in CI. Eight structure checks, each reported as pass, warn or fail with a one-line detail, and two v2 discovery checks reported as information. No score on purpose: a structure check can say what is there and what is missing, and a number on top of ten checks would look like an agent-readiness score without measuring one.
 
 This is the open-source form of the hosted validator at [turva.dev/llms-txt-validator](https://turva.dev/llms-txt-validator), which runs the same logic inside the open [turva.dev Cloudflare Worker](https://github.com/erekola/turva-worker). The hosted validator stays canonical: if the two ever disagree, the hosted one wins and this package gets the fix.
 
@@ -18,11 +18,11 @@ Or run it without installing:
     llms-txt-validate example.com --json
     llms-txt-validate example.com --strict
 
-Exit codes: 0 valid (warnings allowed), 1 not valid (with --strict, warnings also exit 1), 2 fetch or input error. Only the target site's /llms.txt is fetched over https, following a redirect only to the same host or its www/apex twin (an off-site or unsafe redirect fails the first check), the fetch times out after 8 seconds and the read is capped at 256 KB. Nothing is stored.
+Exit codes: 0 valid (warnings allowed), 1 not valid (with --strict, warnings also exit 1), 2 fetch or input error. The two v2 checks are reported as information and move no exit code. Two documents are fetched from the target site over https, its /llms.txt and its home page, each following a redirect only to the same host or its www/apex twin (an off-site or unsafe redirect fails the first check), each fetch times out after 8 seconds and each read is capped at 256 KB. Nothing else is requested, the site is never crawled and nothing is stored.
 
 The target has to be a public domain name: an IP literal, a bracketed IPv6 address, localhost and the internal-use TLDs local, internal, home, lan, corp, test and invalid are all refused before any request goes out, and every redirect hop is checked by the same rule. Names are not resolved here, so a public name that points at a private address is stopped by the network the fetch runs on rather than by this code. On the hosted validator that network is the Cloudflare edge, which does not route to private address space.
 
-## The eight checks
+## The checks
 
 | # | Check | fail | warn |
 |---|-------|------|------|
@@ -35,6 +35,13 @@ The target has to be a public domain name: an IP literal, a bracketed IPv6 addre
 | 7 | Small enough to be cheap to read | | over 50 KB, or read truncated at 256 KB |
 | 8 | No HTML markup in the file | | HTML tags found |
 
+Two more checks read the site's home page for the link relations v2 of the format recommends. They describe the site rather than the file, so they carry the status `pass` when the relation is there and `info` when it is not, never `warn` and never `fail`, and they leave both the summary line and the `--strict` exit code alone. v2 is two weeks old at the time of writing, so warning about these would turn valid files into files with warnings for following the version of the format they were written against.
+
+| # | Check | pass | info |
+|---|-------|------|------|
+| 9 | Home page points to its llms.txt (v2) | `rel="describedby"` found in the head or the Link header | not found, or the home page could not be read |
+| 10 | Home page points to a markdown version (v2) | `rel="alternate" type="text/markdown"` found in the head or the Link header | not found, or the home page could not be read |
+
 ## Node API
 
     import { validateHost } from "turva-llms-txt-validator";
@@ -43,7 +50,7 @@ The target has to be a public domain name: an IP literal, a bracketed IPv6 addre
     console.log(result.summary);
     for (const c of result.checks) console.log(c.status, c.label, c.detail);
 
-The summary is one of valid, valid with warnings, or not valid. The result object is { target, summary, checks }, the same shape the hosted validator returns as JSON:
+The summary is one of valid, valid with warnings, or not valid, and the two v2 checks never change it. The result object is { target, summary, checks }, the same shape the hosted validator returns as JSON:
 
     curl -H "Accept: application/json" "https://turva.dev/llms-txt-validator?url=example.com"
 
@@ -58,7 +65,7 @@ Woodpecker or any other runner: the same npx line works anywhere Node 18.17 or n
 
 ## Why these checks
 
-The llms.txt format is a plain text map of a site for AI agents: an H1 title, a one-line blockquote summary and H2 sections of markdown links. The checks test exactly that structure and nothing else. What the format is and why it matters is written out at [llms.txt explained](https://turva.dev/guides/llms-txt), and the free tools this package belongs to are collected at [turva.dev/tools](https://turva.dev/tools).
+The llms.txt format is a plain text map of a site for AI agents: an H1 title, a one-line blockquote summary and H2 sections of markdown links. v2 of the proposal, published in August 2026, left that format alone and added two link relations so an agent can find a page's markdown version and its llms.txt without guessing. The first eight checks test the structure and nothing else, and the last two report those two relations. What the format is and why it matters is written out at [llms.txt explained](https://turva.dev/guides/llms-txt), and the free tools this package belongs to are collected at [turva.dev/tools](https://turva.dev/tools).
 
 ## License
 

package/bin/cli.mjs

@@ -1,7 +1,8 @@
 #!/usr/bin/env node
 // CLI for turva-llms-txt-validator. Exit codes: 0 = valid (or valid with
 // warnings), 1 = not valid (or warnings with --strict), 2 = could not fetch
-// or bad input. Same checks and JSON shape as the hosted validator:
+// or bad input. The two v2 discovery checks carry status "info" and move no
+// exit code, by design. Same checks and JSON shape as the hosted validator:
 // curl -H "Accept: application/json" "https://turva.dev/llms-txt-validator?url=example.com"
 import { validateHost } from "../src/index.mjs";
 
@@ -16,7 +17,7 @@ if (!target || flags.has("--help")) {
   process.exit(target ? 0 : 2);
 }
 
-const mark = { pass: "ok  ", warn: "warn", fail: "FAIL" };
+const mark = { pass: "ok  ", warn: "warn", fail: "FAIL", info: "info" };
 try {
   const result = await validateHost(target);
   if (flags.has("--json")) {

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "turva-llms-txt-validator",
-  "version": "0.1.8",
-  "description": "Validate a site's llms.txt structure: eight checks, pass/warn/fail, JSON output. Same checks as the hosted validator at turva.dev/llms-txt-validator.",
+  "version": "0.2.0",
+  "description": "Validate a site's llms.txt structure: eight checks as pass/warn/fail plus two llms.txt v2 discovery checks as information, JSON output. Same checks as the hosted validator at turva.dev/llms-txt-validator.",
   "type": "module",
   "main": "./src/index.mjs",
   "exports": "./src/index.mjs",

package/src/index.mjs

@@ -35,11 +35,15 @@ export function isValidPublicHost(host) {
   return true;
 }
 
+// opts.path and opts.accept were added for the v2 discovery checks, which need the
+// site's home page as well as its llms.txt. Nothing else moved: the redirect budget,
+// the same-host rule, the credential and port rejections and the 256 KB cap are the
+// guards this function was measured against.
 export async function fetchLlmsTxt(host, opts = {}) {
   const timeoutMs = opts.timeoutMs ?? 8000;
   const cap = opts.cap ?? 262144;
   const reqApex = host.startsWith("www.") ? host.slice(4) : host;
-  let url = "https://" + host + "/llms.txt";
+  let url = "https://" + host + (opts.path ?? "/llms.txt");
   let redirectedFrom = null;
   // One budget for the whole redirect chain, not one per hop. Mirrors the hosted validator
   // (turva.dev/llms-txt-validator), which is canonical: with five hops a per-hop signal made
@@ -49,7 +53,7 @@ export async function fetchLlmsTxt(host, opts = {}) {
     const res = await fetch(url, {
       redirect: "manual",
       signal: AbortSignal.timeout(Math.max(1, deadline - Date.now())),
-      headers: { "user-agent": opts.userAgent ?? UA, "accept": "text/plain, text/markdown;q=0.9, */*;q=0.1" }
+      headers: { "user-agent": opts.userAgent ?? UA, "accept": opts.accept ?? "text/plain, text/markdown;q=0.9, */*;q=0.1" }
     });
     if (res.status >= 300 && res.status < 400) {
       const loc = res.headers.get("location") || "";
@@ -89,6 +93,7 @@ export async function fetchLlmsTxt(host, opts = {}) {
     return {
       status: res.status,
       contentType: res.headers.get("content-type") || "",
+      linkHeader: res.headers.get("link") || "",
       text: new TextDecoder("utf-8").decode(buf),
       bytes,
       truncated,
@@ -172,6 +177,53 @@ export function validateLlmsTxt(f) {
   return checks;
 }
 
+// v2 of the llms.txt proposal (August 2026) left the file format alone and added one
+// thing: a page should say where its markdown version and its llms.txt are, using
+// rel="alternate" type="text/markdown" and rel="describedby", as HTML link elements or
+// as a Link response header. That is a property of the site, not of the file, so these
+// two carry their own status, "info". They are never a warn and never a fail, the
+// summary line is unchanged, and --strict keeps exiting on warnings only.
+export function findLinkRelations(html, linkHeader) {
+  const found = { describedby: null, markdown: null };
+  // Comments are stripped first: a commented-out link element is not published,
+  // and counting one would report a relation the site does not serve. With no
+  // </head> the first 64 KB are scanned, body included, so a malformed document
+  // does not silently report nothing found.
+  const text = String(html || "").replace(/<!--[\s\S]*?-->/g, "");
+  const cut = text.toLowerCase().indexOf("</head>");
+  const head = cut === -1 ? text.slice(0, 65536) : text.slice(0, cut);
+  for (const tag of head.match(/<link\b[^>]*>/gi) || []) {
+    const rel = ((tag.match(/\brel\s*=\s*["']?([^"'>]+)/i) || [])[1] || "").toLowerCase().trim().split(/\s+/);
+    const type = ((tag.match(/\btype\s*=\s*["']?([^"'>\s]+)/i) || [])[1] || "").toLowerCase();
+    const href = ((tag.match(/\bhref\s*=\s*"([^"]*)"|\bhref\s*=\s*'([^']*)'|\bhref\s*=\s*([^\s"'>]+)/i) || []).slice(1).find((x) => x !== undefined) || "").trim();
+    if (!found.describedby && rel.includes("describedby")) found.describedby = href || "(link element without href)";
+    if (!found.markdown && rel.includes("alternate") && type.startsWith("text/markdown")) found.markdown = href || "(link element without href)";
+  }
+  for (const part of String(linkHeader || "").split(/,(?=\s*<)/)) {
+    const href = ((part.match(/<([^>]*)>/) || [])[1] || "").trim();
+    const rel = ((part.match(/rel\s*=\s*"?([^";,]+)"?/i) || [])[1] || "").toLowerCase().trim().split(/\s+/);
+    const type = ((part.match(/type\s*=\s*"?([^";,]+)"?/i) || [])[1] || "").toLowerCase().trim();
+    if (!found.describedby && rel.includes("describedby")) found.describedby = href || "(Link header without a target)";
+    if (!found.markdown && rel.includes("alternate") && type.startsWith("text/markdown")) found.markdown = href || "(Link header without a target)";
+  }
+  return found;
+}
+
+export function validateV2Discovery(found, unreadReason) {
+  const checks = [];
+  const add = (id, status, label, detail) => checks.push({ id, status, label, detail });
+  if (!found) {
+    add("v2-describedby", "info", "Home page points to its llms.txt (v2)", unreadReason);
+    add("v2-markdown-alternate", "info", "Home page points to a markdown version (v2)", unreadReason);
+    return checks;
+  }
+  add("v2-describedby", found.describedby ? "pass" : "info", "Home page points to its llms.txt (v2)",
+    found.describedby ? 'rel="describedby" to ' + found.describedby.slice(0, 120) : 'no rel="describedby" in the head or the Link header; v2 recommends it so an agent finds the file without guessing');
+  add("v2-markdown-alternate", found.markdown ? "pass" : "info", "Home page points to a markdown version (v2)",
+    found.markdown ? 'rel="alternate" type="text/markdown" to ' + found.markdown.slice(0, 120) : 'no rel="alternate" type="text/markdown" in the head or the Link header; v2 recommends it so an agent finds the markdown form without guessing');
+  return checks;
+}
+
 export function summarizeChecks(checks) {
   if (checks.some((c) => c.status === "fail")) return "not valid";
   if (checks.some((c) => c.status === "warn")) return "valid with warnings";
@@ -183,7 +235,21 @@ export async function validateHost(input, opts = {}) {
   if (!host || !isValidPublicHost(host)) {
     throw new Error("not a public domain name: " + String(input).slice(0, 120));
   }
-  const fetched = await fetchLlmsTxt(host, opts);
-  const checks = validateLlmsTxt(fetched);
+  // opts comes from the caller, and path and accept exist for this function's own
+  // second read. Forwarding them to the first read would let a caller point the
+  // "llms.txt" result at any path on the host, so both are pinned here.
+  const fetched = await fetchLlmsTxt(host, { ...opts, path: "/llms.txt", accept: undefined });
+  let discovery;
+  try {
+    const home = await fetchLlmsTxt(host, { ...opts, path: "/", accept: "text/html, */*;q=0.1" });
+    discovery = home.redirect
+      ? validateV2Discovery(null, "the home page redirects away from this host, so this was not measured")
+      : home.status !== 200
+        ? validateV2Discovery(null, "the home page returned HTTP " + home.status + ", so this was not measured")
+        : validateV2Discovery(findLinkRelations(home.text, home.linkHeader));
+  } catch {
+    discovery = validateV2Discovery(null, "the home page could not be read, so this was not measured");
+  }
+  const checks = validateLlmsTxt(fetched).concat(discovery);
   return { target: "https://" + host + "/llms.txt", summary: summarizeChecks(checks), checks };
 }