turva-llms-txt-validator 0.1.4 → 0.1.6

package/README.md

@@ -1,6 +1,6 @@
 # turva-llms-txt-validator
 
-Validate a site's llms.txt structure from the command line, from Node, or in CI. Eight checks, each reported as pass, warn or fail, with a one-line detail. No score on purpose: a structure check can say what is there and what is missing, and a number on top of eight checks would look like an agent-readiness score without measuring one.
+Validate a site's llms.txt structure from the command line, from Node or in CI. Eight checks, each reported as pass, warn or fail, with a one-line detail. No score on purpose: a structure check can say what is there and what is missing, and a number on top of eight checks would look like an agent-readiness score without measuring one.
 
 This is the open-source form of the hosted validator at [turva.dev/llms-txt-validator](https://turva.dev/llms-txt-validator), which runs the same logic inside the open [turva.dev Cloudflare Worker](https://github.com/erekola/turva-worker). The hosted validator stays canonical: if the two ever disagree, the hosted one wins and this package gets the fix.
 
@@ -20,13 +20,15 @@ Or run it without installing:
 
 Exit codes: 0 valid (warnings allowed), 1 not valid (with --strict, warnings also exit 1), 2 fetch or input error. Only the target site's /llms.txt is fetched over https, following a redirect only to the same host or its www/apex twin (an off-site or unsafe redirect fails the first check), the fetch times out after 8 seconds and the read is capped at 256 KB. Nothing is stored.
 
+The target has to be a public domain name: an IP literal, a bracketed IPv6 address, localhost and the internal-use TLDs local, internal, home, lan, corp, test and invalid are all refused before any request goes out, and every redirect hop is checked by the same rule. Names are not resolved here, so a public name that points at a private address is stopped by the network the fetch runs on rather than by this code. On the hosted validator that network is the Cloudflare edge, which does not route to private address space.
+
 ## The eight checks
 
 | # | Check | fail | warn |
 |---|-------|------|------|
 | 1 | File exists at /llms.txt, HTTP 200 (a same-site www/apex redirect is followed) | non-200, or an off-site or unsafe redirect | |
 | 2 | Response is plain text | body looks like HTML | content-type is not text/plain or text/markdown |
-| 3 | Starts with a single H1 title | first non-empty line is not a markdown H1 | |
+| 3 | Starts with an H1 title | first non-empty line is not a markdown H1 | |
 | 4 | Blockquote summary after the title | | missing one-line summary |
 | 5 | H2 sections group the content | | no H2 sections |
 | 6 | Markdown links an agent can follow | | no links, or relative links |
@@ -56,8 +58,8 @@ Woodpecker or any other runner: the same npx line works anywhere Node 18.17 or n
 
 ## Why these checks
 
-The llms.txt format is a plain text map of a site for AI agents: an H1 title, a one-line blockquote summary, and H2 sections of markdown links. The checks test exactly that structure and nothing else. What the format is and why it matters is written out at [llms.txt explained](https://turva.dev/guides/llms-txt), and the free tools this package belongs to are collected at [turva.dev/tools](https://turva.dev/tools).
+The llms.txt format is a plain text map of a site for AI agents: an H1 title, a one-line blockquote summary and H2 sections of markdown links. The checks test exactly that structure and nothing else. What the format is and why it matters is written out at [llms.txt explained](https://turva.dev/guides/llms-txt), and the free tools this package belongs to are collected at [turva.dev/tools](https://turva.dev/tools).
 
 ## License
 
-MIT. Source at [github.com/erekola/llms-txt-validator](https://github.com/erekola/llms-txt-validator), mirrored at [codeberg.org/erekola/llms-txt-validator](https://codeberg.org/erekola/llms-txt-validator). Published to npm from GitHub Actions with provenance, a signed attestation of where and how the package was built, verifiable on the npm package page.
+MIT. Source at [github.com/erekola/llms-txt-validator](https://github.com/erekola/llms-txt-validator). Published to npm from GitHub Actions with provenance, a signed attestation of where and how the package was built, verifiable on the npm package page.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "turva-llms-txt-validator",
-  "version": "0.1.4",
+  "version": "0.1.6",
   "description": "Validate a site's llms.txt structure: eight checks, pass/warn/fail, JSON output. Same checks as the hosted validator at turva.dev/llms-txt-validator.",
   "type": "module",
   "main": "./src/index.mjs",

package/src/index.mjs

@@ -41,10 +41,14 @@ export async function fetchLlmsTxt(host, opts = {}) {
   const reqApex = host.startsWith("www.") ? host.slice(4) : host;
   let url = "https://" + host + "/llms.txt";
   let redirectedFrom = null;
+  // One budget for the whole redirect chain, not one per hop. Mirrors the hosted validator
+  // (turva.dev/llms-txt-validator), which is canonical: with five hops a per-hop signal made
+  // the caller's timeout five times longer than the value it passed in.
+  const deadline = Date.now() + timeoutMs;
   for (let hop = 0; ; hop++) {
     const res = await fetch(url, {
       redirect: "manual",
-      signal: AbortSignal.timeout(timeoutMs),
+      signal: AbortSignal.timeout(Math.max(1, deadline - Date.now())),
       headers: { "user-agent": opts.userAgent ?? UA, "accept": "text/plain, text/markdown;q=0.9, */*;q=0.1" }
     });
     if (res.status >= 300 && res.status < 400) {