turva-llms-txt-validator 0.3.2 → 0.3.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +130 -43
  2. package/package.json +1 -1
  3. package/src/index.mjs +184 -17
package/README.md CHANGED
@@ -1,72 +1,159 @@
1
1
  # turva-llms-txt-validator
2
2
 
3
- Validate a site's llms.txt structure from the command line, from Node or in CI. Eight structure checks, each reported as pass, warn or fail with a one-line detail, and two v2 discovery checks that read pass or info and never move the summary. No score on purpose: a structure check can say what is there and what is missing, and a number on top of ten checks would look like an agent-readiness score without measuring one.
3
+ Check a website's `llms.txt` from the command line, Node or CI. Get a clear result for each check, with readable output or JSON.
4
4
 
5
- This is the open-source form of the hosted validator at [turva.dev/llms-txt-validator](https://turva.dev/llms-txt-validator), which runs the same logic inside the open [turva.dev Cloudflare Worker](https://github.com/erekola/turva-worker). The hosted validator stays canonical: if the two ever disagree, the hosted one wins and this package gets the fix.
5
+ **Eight structure checks and two informational discovery checks.** The result describes the file's structure and the home page's discovery links. It does not measure overall agent readiness or predict AI-search citations.
6
6
 
7
- ## Install
7
+ [npm package](https://www.npmjs.com/package/turva-llms-txt-validator) · [Try it in your browser](https://turva.dev/llms-txt-validator) · [llms.txt guide](https://turva.dev/guides/llms-txt) · [All turva.dev tools](https://turva.dev/tools)
8
8
 
9
- npm install -g turva-llms-txt-validator
9
+ ## Quick start
10
10
 
11
- Or run it without installing:
11
+ Requires **Node.js 18.17 or newer**. Run it without a global install:
12
12
 
13
- npx turva-llms-txt-validator example.com
13
+ ```sh
14
+ npx --yes turva-llms-txt-validator example.com
15
+ ```
14
16
 
15
- ## CLI
17
+ Replace `example.com` with the domain you want to check. A domain, HTTP URL or HTTPS URL is accepted. Validation uses the host's `/llms.txt` and home page over HTTPS, regardless of the supplied path.
16
18
 
17
- llms-txt-validate example.com
18
- llms-txt-validate example.com --json
19
- llms-txt-validate example.com --strict
19
+ For repeated use, install the CLI globally:
20
20
 
21
- Exit codes: 0 valid (warnings allowed), 1 not valid (with --strict, warnings also exit 1), 2 fetch or input error. The two v2 checks are reported as information and move no exit code. Two documents are fetched from the target site over https, its /llms.txt and its home page, each following a redirect only to the same host or its www/apex twin (an off-site or unsafe redirect fails the first check), each fetch times out after 8 seconds and each read is capped at 256 KB. Nothing else is requested, the site is never crawled and nothing is stored.
21
+ ```sh
22
+ npm install -g turva-llms-txt-validator
23
+ llms-txt-validate example.com
24
+ ```
22
25
 
23
- The target has to be a public domain name: an IP literal, a bracketed IPv6 address, localhost and the internal-use TLDs local, internal, home, lan, corp, test and invalid are all refused before any request goes out, and every redirect hop is checked by the same rule. Names are not resolved here, so a public name that points at a private address is stopped by the network the fetch runs on rather than by this code. On the hosted validator that network is the Cloudflare edge, which does not route to private address space.
26
+ ## Read the result
24
27
 
25
- ## The checks
28
+ The summary comes last. Each preceding line reports one check. This shortened example uses the hosted validator's turva.dev result observed on 2026-09-09, formatted as the CLI prints it. The remaining passing checks are omitted here:
26
29
 
27
- | # | Check | fail | warn |
28
- |---|-------|------|------|
29
- | 1 | File exists at /llms.txt, HTTP 200 (a same-site www/apex redirect is followed) | non-200, or an off-site or unsafe redirect | |
30
- | 2 | Response is plain text | body looks like HTML | content-type is not text/plain or text/markdown |
31
- | 3 | Starts with an H1 title | first non-empty line is not a markdown H1, indentation included, since four spaces make it a code block | |
32
- | 4 | Blockquote summary after the title | | missing one-line summary |
33
- | 5 | H2 sections group the content | | no H2 sections, or no section carrying a markdown file list |
34
- | 6 | Markdown links an agent can follow | | no links, an empty link name, a scheme without a host, or relative links |
35
- | 7 | Small enough to be cheap to read | | over 50 KB, or read truncated at 256 KB |
36
- | 8 | No HTML markup in the file | | HTML tags found |
30
+ ```text
31
+ https://turva.dev/llms.txt
32
+ ok File exists at /llms.txt (HTTP 200)
33
+ ok Response is plain text (text/plain)
34
+ ok Starts with an H1 title ("# turva.dev")
35
+ result: valid
36
+ ```
37
37
 
38
- Two more checks read the site's home page for the link relations v2 of the format recommends. They describe the site rather than the file, so they carry the status `pass` when the relation is there and `info` when it is not, never `warn` and never `fail`, and they leave both the summary line and the `--strict` exit code alone. v2 is two weeks old at the time of writing, so warning about these would turn valid files into files with warnings for following the version of the format they were written against.
38
+ A `warn` identifies something to review, such as a missing blockquote summary. It does not fail the command unless you add `--strict`. A `FAIL` means a structural requirement failed. Each line includes the observed detail.
39
39
 
40
- | # | Check | pass | info |
41
- |---|-------|------|------|
42
- | 9 | Home page points to its llms.txt (v2) | `rel="describedby"` with a target, found in the head or the Link header | not found, no target on the relation, or the home page could not be read |
43
- | 10 | Home page points to a markdown version (v2) | `rel="alternate"` with the media type `text/markdown` and a target, found in the head or the Link header | not found, no target on the relation, a different media type, or the home page could not be read |
40
+ The eight structure checks determine the summary:
44
41
 
45
- ## Node API
42
+ | Summary | Meaning | Default exit code | With `--strict` |
43
+ | --- | --- | --- | --- |
44
+ | `valid` | No structure warnings or failures | `0` | `0` |
45
+ | `valid with warnings` | At least one warning, no failures | `0` | `1` |
46
+ | `not valid` | At least one failure | `1` | `1` |
47
+ | Input or fetch error | Validation could not complete | `2` | `2` |
46
48
 
47
- import { validateHost } from "turva-llms-txt-validator";
49
+ The two discovery checks return `pass` or `info`. They never change the summary or exit code, including in strict mode. If the home page cannot be read, those checks report `info`. The file's structure result still stands.
48
50
 
49
- const result = await validateHost("example.com");
50
- console.log(result.summary);
51
- for (const c of result.checks) console.log(c.status, c.label, c.detail);
51
+ Use JSON for automation and strict mode when warnings should fail a CI step:
52
52
 
53
- The summary is one of valid, valid with warnings, or not valid, and the two v2 checks never change it. The result object is { target, summary, checks }, the same shape the hosted validator returns as JSON:
53
+ ```sh
54
+ npx --yes turva-llms-txt-validator example.com --json --strict
55
+ ```
54
56
 
55
- curl -H "Accept: application/json" "https://turva.dev/llms-txt-validator?url=example.com"
57
+ Completed validation returns `{ target, summary, checks }`. Each check contains `{ id, status, label, detail }`. With `--json`, input and fetch errors return `{ "error": "..." }` and exit with code `2`.
56
58
 
57
- ## CI
59
+ ## What it checks
58
60
 
59
- GitHub Actions:
61
+ ### File structure
60
62
 
61
- - name: Validate llms.txt
62
- run: npx turva-llms-txt-validator your-domain.com --strict
63
+ | # | Check | Failure | Warning |
64
+ | --- | --- | --- | --- |
65
+ | 1 | `/llms.txt` returns HTTP 200 | Non-200 response, rejected or excessive redirects | none |
66
+ | 2 | Response is plain text | Body looks like an HTML page | Content type is neither `text/plain` nor `text/markdown` |
67
+ | 3 | First non-empty line is a Markdown H1 | Missing H1, including a title indented as a code block | none |
68
+ | 4 | Blockquote summary follows the title | none | Missing summary |
69
+ | 5 | H2 sections group the content | none | No H2 sections, or no section contains a Markdown link list |
70
+ | 6 | Markdown links have names and absolute HTTP or HTTPS targets | none | Missing links, empty names, relative targets or unsupported targets |
71
+ | 7 | File is small enough to read cheaply | none | Over 50 KB, or truncated at the 256 KB read limit |
72
+ | 8 | File contains no HTML markup | none | HTML tags found |
63
73
 
64
- Woodpecker or any other runner: the same npx line works anywhere Node 18.17 or newer is present.
74
+ Some failures stop the file checks early. For example, an HTML response is reported as a failed plain-text check rather than parsed as Markdown.
65
75
 
66
- ## Why these checks
76
+ ### Home-page discovery
67
77
 
68
- The llms.txt format is a plain text map of a site for AI agents: an H1 title, a one-line blockquote summary and H2 sections of markdown links. v2 of the proposal, published in August 2026, left that format alone and added two link relations so an agent can find a page's markdown version and its llms.txt without guessing. The first eight checks test the structure and nothing else, and the last two report those two relations. What the format is and why it matters is written out at [llms.txt explained](https://turva.dev/guides/llms-txt), and the free tools this package belongs to are collected at [turva.dev/tools](https://turva.dev/tools).
78
+ These checks are labelled **v2** in the output. They look for link relations in the home page's HTML head or HTTP `Link` header.
79
+
80
+ | # | Check | Pass condition | Otherwise |
81
+ | --- | --- | --- | --- |
82
+ | 9 | Home page points to its llms.txt | `rel="describedby"` has a non-empty target | `info` |
83
+ | 10 | Home page points to a Markdown version | `rel="alternate"` has media type `text/markdown` and a non-empty target | `info` |
84
+
85
+ The validator detects these declarations without fetching their targets. It also does not follow the links inside `llms.txt`, crawl the site or test whether an AI agent can complete a task there.
86
+
87
+ ## Use from Node
88
+
89
+ Install the package in your project:
90
+
91
+ ```sh
92
+ npm install turva-llms-txt-validator
93
+ ```
94
+
95
+ Use an ES module, for example `validate.mjs`:
96
+
97
+ ```js
98
+ import { validateHost } from "turva-llms-txt-validator";
99
+
100
+ const result = await validateHost("example.com");
101
+ console.log(result.summary);
102
+ for (const check of result.checks) {
103
+ console.log(check.status, check.label, check.detail);
104
+ }
105
+ ```
106
+
107
+ `validateHost` rejects on invalid input or a failed `/llms.txt` fetch. Structural failures are returned in `checks` with the summary `not valid`.
108
+
109
+ ## Use in CI
110
+
111
+ Add this step to a GitHub Actions job that already has Node.js 18.17 or newer available. Replace `your-domain.com` with your domain:
112
+
113
+ ```yaml
114
+ - name: Validate llms.txt
115
+ run: npx --yes turva-llms-txt-validator your-domain.com --strict
116
+ ```
117
+
118
+ The same command works in Woodpecker and other runners with Node installed. Omit `--strict` if warnings should remain advisory.
119
+
120
+ ## Fetch limits and security
121
+
122
+ The default validation requests two documents: `https://<host>/llms.txt` and `https://<host>/`. Redirects may add requests, but linked pages and discovery targets are never fetched. The validator stores no results.
123
+
124
+ - Timeout: eight seconds per document, shared across its redirect chain.
125
+ - Read limit: 256 KB per response.
126
+ - Redirects: up to four hops, over HTTPS, to the same host or its `www`/apex equivalent. Off-site redirects, embedded credentials and unsupported ports are rejected.
127
+ - Host checks: IP literals, bracketed IPv6 addresses, localhost and the internal-use TLDs `local`, `internal`, `home`, `lan`, `corp`, `test` and `invalid` are rejected before a request is sent. Redirect targets are checked too.
128
+
129
+ **DNS resolution is not checked for private addresses.** A syntactically public domain can resolve to a private IP, so if you expose this package through a service that accepts untrusted domains, enforce private-address restrictions at the network layer. Local and CI runs use their own network policy. The hosted validator runs on the Cloudflare edge.
130
+
131
+ For vulnerability reporting and supported versions, see [SECURITY.md](SECURITY.md).
132
+
133
+ ## Hosted version and source
134
+
135
+ The [hosted validator](https://turva.dev/llms-txt-validator) accepts any public domain in the browser. It requests the same two documents, `/llms.txt` and the home page, and runs the same checks. Its source is in the [turva.dev Cloudflare Worker](https://github.com/erekola/turva-worker). The hosted validator is the canonical implementation: if results diverge, this package is updated to match it.
136
+
137
+ The hosted version also returns JSON with the same result shape:
138
+
139
+ ```sh
140
+ curl -H "Accept: application/json" "https://turva.dev/llms-txt-validator?url=example.com"
141
+ ```
142
+
143
+ In Windows PowerShell, use `curl.exe` if `curl` resolves to `Invoke-WebRequest`.
144
+
145
+ The package has no runtime dependencies. The repository's release workflow uses npm trusted publishing with provenance. See [SECURITY.md](SECURITY.md) for details and [CHANGELOG.md](CHANGELOG.md) for releases.
146
+
147
+ ## Check HTML and Markdown content
148
+
149
+ [markdown-parity-check](https://github.com/erekola/markdown-parity-check) is a separate tool for a different question. This validator reads the structure of `llms.txt` and the home page's discovery links. The parity check compares the main content of one page's HTML and Markdown versions and reports changed text, numbers, links or blocks. It requires Node.js 22 or newer:
150
+
151
+ ```sh
152
+ npx --yes markdown-parity-check --url https://example.com/page --format json
153
+ ```
154
+
155
+ Replace the URL with your page. If Markdown is served at another address, add `--markdown-url https://example.com/page.md`. Its [browser version](https://turva.dev/markdown-parity-check) on turva.dev checks turva.dev's own pages only. See the [comparison README](https://github.com/erekola/markdown-parity-check) for local-file mode, exit codes and limits.
69
156
 
70
157
  ## License
71
158
 
72
- MIT. Source at [github.com/erekola/llms-txt-validator](https://github.com/erekola/llms-txt-validator). Published to npm from GitHub Actions with provenance, a signed attestation of where and how the package was built, verifiable on the npm package page.
159
+ [MIT](LICENSE). Built by [Erik Rekola](https://github.com/erekola) at [turva.dev](https://turva.dev).
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "turva-llms-txt-validator",
3
- "version": "0.3.2",
3
+ "version": "0.3.3",
4
4
  "description": "Validate a site's llms.txt structure: eight checks as pass/warn/fail plus two llms.txt v2 discovery checks that read pass or info and never move the summary, JSON output. Same checks as the hosted validator at turva.dev/llms-txt-validator.",
5
5
  "type": "module",
6
6
  "main": "./src/index.mjs",
package/src/index.mjs CHANGED
@@ -68,6 +68,11 @@ export async function fetchLlmsTxt(host, opts = {}) {
68
68
  headers: { "user-agent": opts.userAgent ?? UA, "accept": opts.accept ?? "text/plain, text/markdown;q=0.9, */*;q=0.1" }
69
69
  });
70
70
  if (res.status >= 300 && res.status < 400) {
71
+ // Six paths leave this branch, five returns and one continue, and not one of them
72
+ // ever reads the redirect body. undici holds the connection until a body is read or
73
+ // cancelled, so it is released here, once, before the location is even parsed. A
74
+ // failed cancel must not turn a redirect verdict into a throw (2026-09-10).
75
+ try { await res.body?.cancel(); } catch { /* the verdict below is the answer */ }
71
76
  const loc = res.headers.get("location") || "";
72
77
  if (!loc) return { redirect: true, reason: "no-location", status: res.status, location: "" };
73
78
  if (hop >= 4) return { redirect: true, reason: "too-many", status: res.status, location: cut(loc, 120) };
@@ -149,6 +154,41 @@ function collectLinks(text) {
149
154
  // 2026-08-29). Bounding the quantifier would trade the speed bug for a silent accuracy bug,
150
155
  // so the scan is by index: every character is read once and the furthest failed target scan
151
156
  // is remembered.
157
+ // CommonMark fenced code blocks, marked line by line. A "## " or a link inside a fence is
158
+ // example text and not the file's own structure, but until 2026-09-10 both counted, so a
159
+ // file whose only section and only link lived inside ``` or ~~~ was reported valid. The
160
+ // scan is one pass over the lines with no backtracking pattern, because the ReDoS repair
161
+ // of 2026-08-24 bought a worse accuracy defect with a bounded quantifier and the rule out
162
+ // of it was to scan by index instead (mds/gotchas.md 2026-08-24 (jatko 12)).
163
+ // Returns one boolean per line: true for a fence line and for everything inside it.
164
+ function fenceMask(lines) {
165
+ const mask = new Array(lines.length).fill(false);
166
+ let fenceChar = "", fenceLen = 0;
167
+ for (let i = 0; i < lines.length; i++) {
168
+ const l = lines[i];
169
+ let p = 0;
170
+ while (p < 4 && l[p] === " ") p++;
171
+ if (p > 3) { mask[i] = fenceChar !== ""; continue; }
172
+ const c = l[p];
173
+ let run = 0;
174
+ if (c === "`" || c === "~") { while (l[p + run] === c) run++; }
175
+ if (fenceChar === "") {
176
+ // An opening backtick fence may not carry a backtick in its info string; a tilde
177
+ // fence may. Anything shorter than three markers is not a fence at all. A later
178
+ // round will read the consequence as a bug and it is not: in ```js `x` the info
179
+ // string holds a backtick, so that line is prose, and a bare ``` after it OPENS a
180
+ // block instead of closing one. CommonMark reads the same input the same way,
181
+ // measured against the spec 2026-09-10. Leave it.
182
+ if (run >= 3 && (c !== "`" || l.indexOf("`", p + run) === -1)) {
183
+ fenceChar = c; fenceLen = run; mask[i] = true;
184
+ }
185
+ continue;
186
+ }
187
+ mask[i] = true;
188
+ if (c === fenceChar && run >= fenceLen && l.slice(p + run).trim() === "") { fenceChar = ""; fenceLen = 0; }
189
+ }
190
+ return mask;
191
+ }
152
192
  function listItemHasLink(l) {
153
193
  const m = /^ {0,3}[-*+] /.exec(l);
154
194
  if (!m) return false;
@@ -225,15 +265,22 @@ export function validateLlmsTxt(f) {
225
265
  } else {
226
266
  add("summary", "warn", "Blockquote summary after the title", "recommended by the format (> one-line summary), not required");
227
267
  }
228
- const h2Count = (f.text.match(/^## /gm) || []).length;
268
+ // Headings are read outside fences only, and with the same indentation the H1 check and
269
+ // listItemHasLink have allowed since 2026-08-29. Until 2026-09-10 this one line still
270
+ // demanded column zero, so a correct file indented by one to three spaces was reported as
271
+ // having no sections at all while its list under the same indentation counted fine.
272
+ const fenced = fenceMask(lines);
273
+ const h2Count = lines.filter((l, i) => !fenced[i] && /^ {0,3}## /.test(l)).length;
229
274
  // A section counts when it carries a file list. An H2 followed by a paragraph satisfied
230
275
  // this check until 2026-08-29, and the format puts each section's links in a list.
231
276
  let sectionsWithList = 0;
232
277
  {
233
278
  let inSection = false, counted = false;
234
- for (const l of lines) {
235
- if (/^## /.test(l)) { inSection = true; counted = false; continue; }
236
- if (/^# /.test(l)) { inSection = false; continue; }
279
+ for (let i = 0; i < lines.length; i++) {
280
+ const l = lines[i];
281
+ if (fenced[i]) continue;
282
+ if (/^ {0,3}## /.test(l)) { inSection = true; counted = false; continue; }
283
+ if (/^ {0,3}# /.test(l)) { inSection = false; continue; }
237
284
  if (inSection && !counted && listItemHasLink(l)) { sectionsWithList++; counted = true; }
238
285
  }
239
286
  }
@@ -244,7 +291,9 @@ export function validateLlmsTxt(f) {
244
291
  } else {
245
292
  add("sections", "warn", "H2 sections group the content", "no H2 sections found; sections are the convention for grouping links");
246
293
  }
247
- const links = collectLinks(f.text);
294
+ // Links are collected from the prose only, for the same reason the headings are: a link
295
+ // shown inside a code fence is an example of a link, not one an agent can follow.
296
+ const links = collectLinks(lines.filter((l, i) => !fenced[i]).join("\n"));
248
297
  // An entry an agent can use has a name and a target with a host. An empty name and a
249
298
  // bare "https://" both counted as valid absolute links until 2026-08-29.
250
299
  const named = links.filter((m) => m.name.trim() !== "");
@@ -494,6 +543,122 @@ function* htmlTags(text) {
494
543
  }
495
544
  }
496
545
 
546
+ // One forward scan over a tag's attributes, quote aware, first declaration wins, which is
547
+ // what the HTML tokenizer does. A quoted value is opaque: nothing inside it is an
548
+ // attribute name, which is the whole point of reading the tag this way. The quote opens
549
+ // only directly after "=", the same rule tagEnd uses to find the end of the tag, so the
550
+ // two agree on where a value starts and ends.
551
+ function tagAttributes(tag) {
552
+ const out = {};
553
+ const space = " \t\n\r\f";
554
+ const n = tag.length;
555
+ const open = /^<[a-z0-9]*/i.exec(tag);
556
+ let i = open ? open[0].length : 0;
557
+ while (i < n) {
558
+ const c = tag[i];
559
+ if (c === ">") break;
560
+ if (space.includes(c) || c === "/") { i++; continue; }
561
+ const nameStart = i;
562
+ while (i < n && !space.includes(tag[i]) && tag[i] !== "/" && tag[i] !== "=" && tag[i] !== ">") i++;
563
+ const name = tag.slice(nameStart, i).toLowerCase();
564
+ while (i < n && space.includes(tag[i])) i++;
565
+ let value = "";
566
+ if (tag[i] === "=") {
567
+ i++;
568
+ while (i < n && space.includes(tag[i])) i++;
569
+ const q = tag[i];
570
+ if (q === '"' || q === "'") {
571
+ i++;
572
+ const valueStart = i;
573
+ while (i < n && tag[i] !== q) i++;
574
+ value = tag.slice(valueStart, i);
575
+ if (i < n) i++;
576
+ } else {
577
+ const valueStart = i;
578
+ while (i < n && !space.includes(tag[i]) && tag[i] !== ">") i++;
579
+ value = tag.slice(valueStart, i);
580
+ }
581
+ }
582
+ if (name && !(name in out)) out[name] = value;
583
+ }
584
+ return out;
585
+ }
586
+
587
+ // RFC 8288 section 3: a Link header is a comma separated list, each value an angle
588
+ // bracketed URI reference followed by semicolon separated parameters, and a parameter
589
+ // value may be a quoted string that contains commas, semicolons and backslash escapes.
590
+ // Reading rel and type with one regex over the whole value read the CONTENT of
591
+ // title="note; rel=alternate; type=text/markdown" as parameters of the link.
592
+ function parseLinkHeader(header) {
593
+ const s = String(header || "");
594
+ const space = " \t";
595
+ const n = s.length;
596
+ const out = [];
597
+ let i = 0;
598
+ while (i < n) {
599
+ while (i < n && (s[i] === "," || space.includes(s[i]))) i++;
600
+ if (i >= n) break;
601
+ if (s[i] !== "<") { i = nextLinkValue(s, i); continue; }
602
+ const gt = s.indexOf(">", i + 1);
603
+ if (gt === -1) break;
604
+ const href = s.slice(i + 1, gt).trim();
605
+ i = gt + 1;
606
+ const params = {};
607
+ while (i < n) {
608
+ while (i < n && space.includes(s[i])) i++;
609
+ if (i >= n) break;
610
+ if (s[i] === ",") { i++; break; }
611
+ if (s[i] !== ";") { i = nextLinkValue(s, i); break; }
612
+ i++;
613
+ while (i < n && space.includes(s[i])) i++;
614
+ const nameStart = i;
615
+ while (i < n && s[i] !== "=" && s[i] !== ";" && s[i] !== "," && !space.includes(s[i])) i++;
616
+ const name = s.slice(nameStart, i).toLowerCase();
617
+ while (i < n && space.includes(s[i])) i++;
618
+ let value = "";
619
+ if (s[i] === "=") {
620
+ i++;
621
+ while (i < n && space.includes(s[i])) i++;
622
+ if (s[i] === '"') {
623
+ i++;
624
+ let buf = "";
625
+ while (i < n && s[i] !== '"') {
626
+ if (s[i] === "\\" && i + 1 < n) { buf += s[i + 1]; i += 2; continue; }
627
+ buf += s[i];
628
+ i++;
629
+ }
630
+ if (i < n) i++;
631
+ value = buf;
632
+ } else {
633
+ const valueStart = i;
634
+ while (i < n && s[i] !== ";" && s[i] !== "," && !space.includes(s[i])) i++;
635
+ value = s.slice(valueStart, i);
636
+ }
637
+ }
638
+ if (name && !(name in params)) params[name] = value;
639
+ }
640
+ out.push({ href: href, params: params });
641
+ }
642
+ return out;
643
+ }
644
+
645
+ // Move to the start of the next comma separated value without stopping inside a quoted
646
+ // string, so a comma in title="a, b" does not split one value into two.
647
+ function nextLinkValue(s, from) {
648
+ let quote = "";
649
+ for (let j = from; j < s.length; j++) {
650
+ const c = s[j];
651
+ if (quote) {
652
+ if (c === "\\") { j++; continue; }
653
+ if (c === quote) quote = "";
654
+ continue;
655
+ }
656
+ if (c === '"') { quote = c; continue; }
657
+ if (c === ",") return j + 1;
658
+ }
659
+ return s.length;
660
+ }
661
+
497
662
  // v2 of the llms.txt proposal (August 2026) left the file format alone and added one
498
663
  // thing: a page should say where its markdown version and its llms.txt are, using
499
664
  // rel="alternate" type="text/markdown" and rel="describedby", as HTML link elements or
@@ -513,24 +678,26 @@ export function findLinkRelations(html, linkHeader) {
513
678
  // NAME is "link<link", not as a link element, so \b would count a relation the site
514
679
  // does not publish. Measured against parse5, 2026-08-24.
515
680
  if (!/^<link(?=[\s/>])/i.test(tag)) continue;
516
- // The attribute name has to start the token. \b sits between the hyphen and the name,
517
- // so data-rel, data-type and data-href were read as the real attributes until
518
- // 2026-08-29, and a page could claim a relation it does not publish.
519
- const rel = ((tag.match(/(?:^|[\s/])rel\s*=\s*["']?([^"'>]+)/i) || [])[1] || "").toLowerCase().trim().split(/\s+/);
520
- const type = ((tag.match(/(?:^|[\s/])type\s*=\s*["']?([^"'>\s]+)/i) || [])[1] || "").toLowerCase();
521
- const href = ((tag.match(/(?:^|[\s/])href\s*=\s*"([^"]*)"|(?:^|[\s/])href\s*=\s*'([^']*)'|(?:^|[\s/])href\s*=\s*([^\s"'>]+)/i) || []).slice(1).find((x) => x !== undefined) || "").trim();
681
+ // Attributes come from ONE quote aware scan, not from three independent regex matches
682
+ // over the whole tag. Those three could not tell an attribute from the quoted CONTENT
683
+ // of another attribute, so data-note=" rel='alternate' type='text/markdown'
684
+ // href='/fake.md'" reported a relation the page does not publish. The earlier
685
+ // data-rel fix, 2026-08-29, moved the START of the name and does not reach this.
686
+ // Measured and fixed 2026-09-09.
687
+ const attrs = tagAttributes(tag);
688
+ const rel = (attrs.rel || "").toLowerCase().trim().split(/\s+/);
689
+ const type = (attrs.type || "").toLowerCase().trim();
690
+ const href = (attrs.href || "").trim();
522
691
  // text/markdown, not anything that starts with it, and a relation without a target is
523
692
  // not a relation: both passed until 2026-08-29.
524
693
  const isMarkdown = type.split(";")[0].trim() === "text/markdown";
525
694
  if (!found.describedby && href && rel.includes("describedby")) found.describedby = href;
526
695
  if (!found.markdown && href && rel.includes("alternate") && isMarkdown) found.markdown = href;
527
696
  }
528
- for (const part of String(linkHeader || "").split(/,(?=\s*<)/)) {
529
- const lt = part.indexOf("<");
530
- const gt = lt === -1 ? -1 : part.indexOf(">", lt + 1);
531
- const href = (gt === -1 ? "" : part.slice(lt + 1, gt)).trim();
532
- const rel = ((part.match(/(?:^|[;\s])rel\s*=\s*"?([^";,]+)"?/i) || [])[1] || "").toLowerCase().trim().split(/\s+/);
533
- const type = ((part.match(/(?:^|[;\s])type\s*=\s*"?([^";,]+)"?/i) || [])[1] || "").toLowerCase().trim();
697
+ for (const link of parseLinkHeader(linkHeader)) {
698
+ const href = link.href;
699
+ const rel = (link.params.rel || "").toLowerCase().trim().split(/\s+/);
700
+ const type = (link.params.type || "").toLowerCase().trim();
534
701
  const isMarkdownHeader = type.split(";")[0].trim() === "text/markdown";
535
702
  if (!found.describedby && href && rel.includes("describedby")) found.describedby = href;
536
703
  if (!found.markdown && href && rel.includes("alternate") && isMarkdownHeader) found.markdown = href;