turva-llms-txt-validator 0.3.2 → 0.3.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +130 -43
- package/package.json +1 -1
- package/src/index.mjs +184 -17
package/README.md
CHANGED
|
@@ -1,72 +1,159 @@
|
|
|
1
1
|
# turva-llms-txt-validator
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Check a website's `llms.txt` from the command line, Node or CI. Get a clear result for each check, with readable output or JSON.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
**Eight structure checks and two informational discovery checks.** The result describes the file's structure and the home page's discovery links. It does not measure overall agent readiness or predict AI-search citations.
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
[npm package](https://www.npmjs.com/package/turva-llms-txt-validator) · [Try it in your browser](https://turva.dev/llms-txt-validator) · [llms.txt guide](https://turva.dev/guides/llms-txt) · [All turva.dev tools](https://turva.dev/tools)
|
|
8
8
|
|
|
9
|
-
|
|
9
|
+
## Quick start
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
Requires **Node.js 18.17 or newer**. Run it without a global install:
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
```sh
|
|
14
|
+
npx --yes turva-llms-txt-validator example.com
|
|
15
|
+
```
|
|
14
16
|
|
|
15
|
-
|
|
17
|
+
Replace `example.com` with the domain you want to check. A domain, HTTP URL or HTTPS URL is accepted. Validation uses the host's `/llms.txt` and home page over HTTPS, regardless of the supplied path.
|
|
16
18
|
|
|
17
|
-
|
|
18
|
-
llms-txt-validate example.com --json
|
|
19
|
-
llms-txt-validate example.com --strict
|
|
19
|
+
For repeated use, install the CLI globally:
|
|
20
20
|
|
|
21
|
-
|
|
21
|
+
```sh
|
|
22
|
+
npm install -g turva-llms-txt-validator
|
|
23
|
+
llms-txt-validate example.com
|
|
24
|
+
```
|
|
22
25
|
|
|
23
|
-
|
|
26
|
+
## Read the result
|
|
24
27
|
|
|
25
|
-
|
|
28
|
+
The summary comes last. Each preceding line reports one check. This shortened example uses the hosted validator's turva.dev result observed on 2026-09-09, formatted as the CLI prints it. The remaining passing checks are omitted here:
|
|
26
29
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
| 6 | Markdown links an agent can follow | | no links, an empty link name, a scheme without a host, or relative links |
|
|
35
|
-
| 7 | Small enough to be cheap to read | | over 50 KB, or read truncated at 256 KB |
|
|
36
|
-
| 8 | No HTML markup in the file | | HTML tags found |
|
|
30
|
+
```text
|
|
31
|
+
https://turva.dev/llms.txt
|
|
32
|
+
ok File exists at /llms.txt (HTTP 200)
|
|
33
|
+
ok Response is plain text (text/plain)
|
|
34
|
+
ok Starts with an H1 title ("# turva.dev")
|
|
35
|
+
result: valid
|
|
36
|
+
```
|
|
37
37
|
|
|
38
|
-
|
|
38
|
+
A `warn` identifies something to review, such as a missing blockquote summary. It does not fail the command unless you add `--strict`. A `FAIL` means a structural requirement failed. Each line includes the observed detail.
|
|
39
39
|
|
|
40
|
-
|
|
41
|
-
|---|-------|------|------|
|
|
42
|
-
| 9 | Home page points to its llms.txt (v2) | `rel="describedby"` with a target, found in the head or the Link header | not found, no target on the relation, or the home page could not be read |
|
|
43
|
-
| 10 | Home page points to a markdown version (v2) | `rel="alternate"` with the media type `text/markdown` and a target, found in the head or the Link header | not found, no target on the relation, a different media type, or the home page could not be read |
|
|
40
|
+
The eight structure checks determine the summary:
|
|
44
41
|
|
|
45
|
-
|
|
42
|
+
| Summary | Meaning | Default exit code | With `--strict` |
|
|
43
|
+
| --- | --- | --- | --- |
|
|
44
|
+
| `valid` | No structure warnings or failures | `0` | `0` |
|
|
45
|
+
| `valid with warnings` | At least one warning, no failures | `0` | `1` |
|
|
46
|
+
| `not valid` | At least one failure | `1` | `1` |
|
|
47
|
+
| Input or fetch error | Validation could not complete | `2` | `2` |
|
|
46
48
|
|
|
47
|
-
|
|
49
|
+
The two discovery checks return `pass` or `info`. They never change the summary or exit code, including in strict mode. If the home page cannot be read, those checks report `info`. The file's structure result still stands.
|
|
48
50
|
|
|
49
|
-
|
|
50
|
-
console.log(result.summary);
|
|
51
|
-
for (const c of result.checks) console.log(c.status, c.label, c.detail);
|
|
51
|
+
Use JSON for automation and strict mode when warnings should fail a CI step:
|
|
52
52
|
|
|
53
|
-
|
|
53
|
+
```sh
|
|
54
|
+
npx --yes turva-llms-txt-validator example.com --json --strict
|
|
55
|
+
```
|
|
54
56
|
|
|
55
|
-
|
|
57
|
+
Completed validation returns `{ target, summary, checks }`. Each check contains `{ id, status, label, detail }`. With `--json`, input and fetch errors return `{ "error": "..." }` and exit with code `2`.
|
|
56
58
|
|
|
57
|
-
##
|
|
59
|
+
## What it checks
|
|
58
60
|
|
|
59
|
-
|
|
61
|
+
### File structure
|
|
60
62
|
|
|
61
|
-
|
|
62
|
-
|
|
63
|
+
| # | Check | Failure | Warning |
|
|
64
|
+
| --- | --- | --- | --- |
|
|
65
|
+
| 1 | `/llms.txt` returns HTTP 200 | Non-200 response, rejected or excessive redirects | none |
|
|
66
|
+
| 2 | Response is plain text | Body looks like an HTML page | Content type is neither `text/plain` nor `text/markdown` |
|
|
67
|
+
| 3 | First non-empty line is a Markdown H1 | Missing H1, including a title indented as a code block | none |
|
|
68
|
+
| 4 | Blockquote summary follows the title | none | Missing summary |
|
|
69
|
+
| 5 | H2 sections group the content | none | No H2 sections, or no section contains a Markdown link list |
|
|
70
|
+
| 6 | Markdown links have names and absolute HTTP or HTTPS targets | none | Missing links, empty names, relative targets or unsupported targets |
|
|
71
|
+
| 7 | File is small enough to read cheaply | none | Over 50 KB, or truncated at the 256 KB read limit |
|
|
72
|
+
| 8 | File contains no HTML markup | none | HTML tags found |
|
|
63
73
|
|
|
64
|
-
|
|
74
|
+
Some failures stop the file checks early. For example, an HTML response is reported as a failed plain-text check rather than parsed as Markdown.
|
|
65
75
|
|
|
66
|
-
|
|
76
|
+
### Home-page discovery
|
|
67
77
|
|
|
68
|
-
|
|
78
|
+
These checks are labelled **v2** in the output. They look for link relations in the home page's HTML head or HTTP `Link` header.
|
|
79
|
+
|
|
80
|
+
| # | Check | Pass condition | Otherwise |
|
|
81
|
+
| --- | --- | --- | --- |
|
|
82
|
+
| 9 | Home page points to its llms.txt | `rel="describedby"` has a non-empty target | `info` |
|
|
83
|
+
| 10 | Home page points to a Markdown version | `rel="alternate"` has media type `text/markdown` and a non-empty target | `info` |
|
|
84
|
+
|
|
85
|
+
The validator detects these declarations without fetching their targets. It also does not follow the links inside `llms.txt`, crawl the site or test whether an AI agent can complete a task there.
|
|
86
|
+
|
|
87
|
+
## Use from Node
|
|
88
|
+
|
|
89
|
+
Install the package in your project:
|
|
90
|
+
|
|
91
|
+
```sh
|
|
92
|
+
npm install turva-llms-txt-validator
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
Use an ES module, for example `validate.mjs`:
|
|
96
|
+
|
|
97
|
+
```js
|
|
98
|
+
import { validateHost } from "turva-llms-txt-validator";
|
|
99
|
+
|
|
100
|
+
const result = await validateHost("example.com");
|
|
101
|
+
console.log(result.summary);
|
|
102
|
+
for (const check of result.checks) {
|
|
103
|
+
console.log(check.status, check.label, check.detail);
|
|
104
|
+
}
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
`validateHost` rejects on invalid input or a failed `/llms.txt` fetch. Structural failures are returned in `checks` with the summary `not valid`.
|
|
108
|
+
|
|
109
|
+
## Use in CI
|
|
110
|
+
|
|
111
|
+
Add this step to a GitHub Actions job that already has Node.js 18.17 or newer available. Replace `your-domain.com` with your domain:
|
|
112
|
+
|
|
113
|
+
```yaml
|
|
114
|
+
- name: Validate llms.txt
|
|
115
|
+
run: npx --yes turva-llms-txt-validator your-domain.com --strict
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
The same command works in Woodpecker and other runners with Node installed. Omit `--strict` if warnings should remain advisory.
|
|
119
|
+
|
|
120
|
+
## Fetch limits and security
|
|
121
|
+
|
|
122
|
+
The default validation requests two documents: `https://<host>/llms.txt` and `https://<host>/`. Redirects may add requests, but linked pages and discovery targets are never fetched. The validator stores no results.
|
|
123
|
+
|
|
124
|
+
- Timeout: eight seconds per document, shared across its redirect chain.
|
|
125
|
+
- Read limit: 256 KB per response.
|
|
126
|
+
- Redirects: up to four hops, over HTTPS, to the same host or its `www`/apex equivalent. Off-site redirects, embedded credentials and unsupported ports are rejected.
|
|
127
|
+
- Host checks: IP literals, bracketed IPv6 addresses, localhost and the internal-use TLDs `local`, `internal`, `home`, `lan`, `corp`, `test` and `invalid` are rejected before a request is sent. Redirect targets are checked too.
|
|
128
|
+
|
|
129
|
+
**DNS resolution is not checked for private addresses.** A syntactically public domain can resolve to a private IP, so if you expose this package through a service that accepts untrusted domains, enforce private-address restrictions at the network layer. Local and CI runs use their own network policy. The hosted validator runs on the Cloudflare edge.
|
|
130
|
+
|
|
131
|
+
For vulnerability reporting and supported versions, see [SECURITY.md](SECURITY.md).
|
|
132
|
+
|
|
133
|
+
## Hosted version and source
|
|
134
|
+
|
|
135
|
+
The [hosted validator](https://turva.dev/llms-txt-validator) accepts any public domain in the browser. It requests the same two documents, `/llms.txt` and the home page, and runs the same checks. Its source is in the [turva.dev Cloudflare Worker](https://github.com/erekola/turva-worker). The hosted validator is the canonical implementation: if results diverge, this package is updated to match it.
|
|
136
|
+
|
|
137
|
+
The hosted version also returns JSON with the same result shape:
|
|
138
|
+
|
|
139
|
+
```sh
|
|
140
|
+
curl -H "Accept: application/json" "https://turva.dev/llms-txt-validator?url=example.com"
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
In Windows PowerShell, use `curl.exe` if `curl` resolves to `Invoke-WebRequest`.
|
|
144
|
+
|
|
145
|
+
The package has no runtime dependencies. The repository's release workflow uses npm trusted publishing with provenance. See [SECURITY.md](SECURITY.md) for details and [CHANGELOG.md](CHANGELOG.md) for releases.
|
|
146
|
+
|
|
147
|
+
## Check HTML and Markdown content
|
|
148
|
+
|
|
149
|
+
[markdown-parity-check](https://github.com/erekola/markdown-parity-check) is a separate tool for a different question. This validator reads the structure of `llms.txt` and the home page's discovery links. The parity check compares the main content of one page's HTML and Markdown versions and reports changed text, numbers, links or blocks. It requires Node.js 22 or newer:
|
|
150
|
+
|
|
151
|
+
```sh
|
|
152
|
+
npx --yes markdown-parity-check --url https://example.com/page --format json
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Replace the URL with your page. If Markdown is served at another address, add `--markdown-url https://example.com/page.md`. Its [browser version](https://turva.dev/markdown-parity-check) on turva.dev checks turva.dev's own pages only. See the [comparison README](https://github.com/erekola/markdown-parity-check) for local-file mode, exit codes and limits.
|
|
69
156
|
|
|
70
157
|
## License
|
|
71
158
|
|
|
72
|
-
MIT.
|
|
159
|
+
[MIT](LICENSE). Built by [Erik Rekola](https://github.com/erekola) at [turva.dev](https://turva.dev).
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "turva-llms-txt-validator",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.3",
|
|
4
4
|
"description": "Validate a site's llms.txt structure: eight checks as pass/warn/fail plus two llms.txt v2 discovery checks that read pass or info and never move the summary, JSON output. Same checks as the hosted validator at turva.dev/llms-txt-validator.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./src/index.mjs",
|
package/src/index.mjs
CHANGED
|
@@ -68,6 +68,11 @@ export async function fetchLlmsTxt(host, opts = {}) {
|
|
|
68
68
|
headers: { "user-agent": opts.userAgent ?? UA, "accept": opts.accept ?? "text/plain, text/markdown;q=0.9, */*;q=0.1" }
|
|
69
69
|
});
|
|
70
70
|
if (res.status >= 300 && res.status < 400) {
|
|
71
|
+
// Six paths leave this branch, five returns and one continue, and not one of them
|
|
72
|
+
// ever reads the redirect body. undici holds the connection until a body is read or
|
|
73
|
+
// cancelled, so it is released here, once, before the location is even parsed. A
|
|
74
|
+
// failed cancel must not turn a redirect verdict into a throw (2026-09-10).
|
|
75
|
+
try { await res.body?.cancel(); } catch { /* the verdict below is the answer */ }
|
|
71
76
|
const loc = res.headers.get("location") || "";
|
|
72
77
|
if (!loc) return { redirect: true, reason: "no-location", status: res.status, location: "" };
|
|
73
78
|
if (hop >= 4) return { redirect: true, reason: "too-many", status: res.status, location: cut(loc, 120) };
|
|
@@ -149,6 +154,41 @@ function collectLinks(text) {
|
|
|
149
154
|
// 2026-08-29). Bounding the quantifier would trade the speed bug for a silent accuracy bug,
|
|
150
155
|
// so the scan is by index: every character is read once and the furthest failed target scan
|
|
151
156
|
// is remembered.
|
|
157
|
+
// CommonMark fenced code blocks, marked line by line. A "## " or a link inside a fence is
|
|
158
|
+
// example text and not the file's own structure, but until 2026-09-10 both counted, so a
|
|
159
|
+
// file whose only section and only link lived inside ``` or ~~~ was reported valid. The
|
|
160
|
+
// scan is one pass over the lines with no backtracking pattern, because the ReDoS repair
|
|
161
|
+
// of 2026-08-24 bought a worse accuracy defect with a bounded quantifier and the rule out
|
|
162
|
+
// of it was to scan by index instead (mds/gotchas.md 2026-08-24 (jatko 12)).
|
|
163
|
+
// Returns one boolean per line: true for a fence line and for everything inside it.
|
|
164
|
+
function fenceMask(lines) {
|
|
165
|
+
const mask = new Array(lines.length).fill(false);
|
|
166
|
+
let fenceChar = "", fenceLen = 0;
|
|
167
|
+
for (let i = 0; i < lines.length; i++) {
|
|
168
|
+
const l = lines[i];
|
|
169
|
+
let p = 0;
|
|
170
|
+
while (p < 4 && l[p] === " ") p++;
|
|
171
|
+
if (p > 3) { mask[i] = fenceChar !== ""; continue; }
|
|
172
|
+
const c = l[p];
|
|
173
|
+
let run = 0;
|
|
174
|
+
if (c === "`" || c === "~") { while (l[p + run] === c) run++; }
|
|
175
|
+
if (fenceChar === "") {
|
|
176
|
+
// An opening backtick fence may not carry a backtick in its info string; a tilde
|
|
177
|
+
// fence may. Anything shorter than three markers is not a fence at all. A later
|
|
178
|
+
// round will read the consequence as a bug and it is not: in ```js `x` the info
|
|
179
|
+
// string holds a backtick, so that line is prose, and a bare ``` after it OPENS a
|
|
180
|
+
// block instead of closing one. CommonMark reads the same input the same way,
|
|
181
|
+
// measured against the spec 2026-09-10. Leave it.
|
|
182
|
+
if (run >= 3 && (c !== "`" || l.indexOf("`", p + run) === -1)) {
|
|
183
|
+
fenceChar = c; fenceLen = run; mask[i] = true;
|
|
184
|
+
}
|
|
185
|
+
continue;
|
|
186
|
+
}
|
|
187
|
+
mask[i] = true;
|
|
188
|
+
if (c === fenceChar && run >= fenceLen && l.slice(p + run).trim() === "") { fenceChar = ""; fenceLen = 0; }
|
|
189
|
+
}
|
|
190
|
+
return mask;
|
|
191
|
+
}
|
|
152
192
|
function listItemHasLink(l) {
|
|
153
193
|
const m = /^ {0,3}[-*+] /.exec(l);
|
|
154
194
|
if (!m) return false;
|
|
@@ -225,15 +265,22 @@ export function validateLlmsTxt(f) {
|
|
|
225
265
|
} else {
|
|
226
266
|
add("summary", "warn", "Blockquote summary after the title", "recommended by the format (> one-line summary), not required");
|
|
227
267
|
}
|
|
228
|
-
|
|
268
|
+
// Headings are read outside fences only, and with the same indentation the H1 check and
|
|
269
|
+
// listItemHasLink have allowed since 2026-08-29. Until 2026-09-10 this one line still
|
|
270
|
+
// demanded column zero, so a correct file indented by one to three spaces was reported as
|
|
271
|
+
// having no sections at all while its list under the same indentation counted fine.
|
|
272
|
+
const fenced = fenceMask(lines);
|
|
273
|
+
const h2Count = lines.filter((l, i) => !fenced[i] && /^ {0,3}## /.test(l)).length;
|
|
229
274
|
// A section counts when it carries a file list. An H2 followed by a paragraph satisfied
|
|
230
275
|
// this check until 2026-08-29, and the format puts each section's links in a list.
|
|
231
276
|
let sectionsWithList = 0;
|
|
232
277
|
{
|
|
233
278
|
let inSection = false, counted = false;
|
|
234
|
-
for (
|
|
235
|
-
|
|
236
|
-
if (
|
|
279
|
+
for (let i = 0; i < lines.length; i++) {
|
|
280
|
+
const l = lines[i];
|
|
281
|
+
if (fenced[i]) continue;
|
|
282
|
+
if (/^ {0,3}## /.test(l)) { inSection = true; counted = false; continue; }
|
|
283
|
+
if (/^ {0,3}# /.test(l)) { inSection = false; continue; }
|
|
237
284
|
if (inSection && !counted && listItemHasLink(l)) { sectionsWithList++; counted = true; }
|
|
238
285
|
}
|
|
239
286
|
}
|
|
@@ -244,7 +291,9 @@ export function validateLlmsTxt(f) {
|
|
|
244
291
|
} else {
|
|
245
292
|
add("sections", "warn", "H2 sections group the content", "no H2 sections found; sections are the convention for grouping links");
|
|
246
293
|
}
|
|
247
|
-
|
|
294
|
+
// Links are collected from the prose only, for the same reason the headings are: a link
|
|
295
|
+
// shown inside a code fence is an example of a link, not one an agent can follow.
|
|
296
|
+
const links = collectLinks(lines.filter((l, i) => !fenced[i]).join("\n"));
|
|
248
297
|
// An entry an agent can use has a name and a target with a host. An empty name and a
|
|
249
298
|
// bare "https://" both counted as valid absolute links until 2026-08-29.
|
|
250
299
|
const named = links.filter((m) => m.name.trim() !== "");
|
|
@@ -494,6 +543,122 @@ function* htmlTags(text) {
|
|
|
494
543
|
}
|
|
495
544
|
}
|
|
496
545
|
|
|
546
|
+
// One forward scan over a tag's attributes, quote aware, first declaration wins, which is
|
|
547
|
+
// what the HTML tokenizer does. A quoted value is opaque: nothing inside it is an
|
|
548
|
+
// attribute name, which is the whole point of reading the tag this way. The quote opens
|
|
549
|
+
// only directly after "=", the same rule tagEnd uses to find the end of the tag, so the
|
|
550
|
+
// two agree on where a value starts and ends.
|
|
551
|
+
function tagAttributes(tag) {
|
|
552
|
+
const out = {};
|
|
553
|
+
const space = " \t\n\r\f";
|
|
554
|
+
const n = tag.length;
|
|
555
|
+
const open = /^<[a-z0-9]*/i.exec(tag);
|
|
556
|
+
let i = open ? open[0].length : 0;
|
|
557
|
+
while (i < n) {
|
|
558
|
+
const c = tag[i];
|
|
559
|
+
if (c === ">") break;
|
|
560
|
+
if (space.includes(c) || c === "/") { i++; continue; }
|
|
561
|
+
const nameStart = i;
|
|
562
|
+
while (i < n && !space.includes(tag[i]) && tag[i] !== "/" && tag[i] !== "=" && tag[i] !== ">") i++;
|
|
563
|
+
const name = tag.slice(nameStart, i).toLowerCase();
|
|
564
|
+
while (i < n && space.includes(tag[i])) i++;
|
|
565
|
+
let value = "";
|
|
566
|
+
if (tag[i] === "=") {
|
|
567
|
+
i++;
|
|
568
|
+
while (i < n && space.includes(tag[i])) i++;
|
|
569
|
+
const q = tag[i];
|
|
570
|
+
if (q === '"' || q === "'") {
|
|
571
|
+
i++;
|
|
572
|
+
const valueStart = i;
|
|
573
|
+
while (i < n && tag[i] !== q) i++;
|
|
574
|
+
value = tag.slice(valueStart, i);
|
|
575
|
+
if (i < n) i++;
|
|
576
|
+
} else {
|
|
577
|
+
const valueStart = i;
|
|
578
|
+
while (i < n && !space.includes(tag[i]) && tag[i] !== ">") i++;
|
|
579
|
+
value = tag.slice(valueStart, i);
|
|
580
|
+
}
|
|
581
|
+
}
|
|
582
|
+
if (name && !(name in out)) out[name] = value;
|
|
583
|
+
}
|
|
584
|
+
return out;
|
|
585
|
+
}
|
|
586
|
+
|
|
587
|
+
// RFC 8288 section 3: a Link header is a comma separated list, each value an angle
|
|
588
|
+
// bracketed URI reference followed by semicolon separated parameters, and a parameter
|
|
589
|
+
// value may be a quoted string that contains commas, semicolons and backslash escapes.
|
|
590
|
+
// Reading rel and type with one regex over the whole value read the CONTENT of
|
|
591
|
+
// title="note; rel=alternate; type=text/markdown" as parameters of the link.
|
|
592
|
+
function parseLinkHeader(header) {
|
|
593
|
+
const s = String(header || "");
|
|
594
|
+
const space = " \t";
|
|
595
|
+
const n = s.length;
|
|
596
|
+
const out = [];
|
|
597
|
+
let i = 0;
|
|
598
|
+
while (i < n) {
|
|
599
|
+
while (i < n && (s[i] === "," || space.includes(s[i]))) i++;
|
|
600
|
+
if (i >= n) break;
|
|
601
|
+
if (s[i] !== "<") { i = nextLinkValue(s, i); continue; }
|
|
602
|
+
const gt = s.indexOf(">", i + 1);
|
|
603
|
+
if (gt === -1) break;
|
|
604
|
+
const href = s.slice(i + 1, gt).trim();
|
|
605
|
+
i = gt + 1;
|
|
606
|
+
const params = {};
|
|
607
|
+
while (i < n) {
|
|
608
|
+
while (i < n && space.includes(s[i])) i++;
|
|
609
|
+
if (i >= n) break;
|
|
610
|
+
if (s[i] === ",") { i++; break; }
|
|
611
|
+
if (s[i] !== ";") { i = nextLinkValue(s, i); break; }
|
|
612
|
+
i++;
|
|
613
|
+
while (i < n && space.includes(s[i])) i++;
|
|
614
|
+
const nameStart = i;
|
|
615
|
+
while (i < n && s[i] !== "=" && s[i] !== ";" && s[i] !== "," && !space.includes(s[i])) i++;
|
|
616
|
+
const name = s.slice(nameStart, i).toLowerCase();
|
|
617
|
+
while (i < n && space.includes(s[i])) i++;
|
|
618
|
+
let value = "";
|
|
619
|
+
if (s[i] === "=") {
|
|
620
|
+
i++;
|
|
621
|
+
while (i < n && space.includes(s[i])) i++;
|
|
622
|
+
if (s[i] === '"') {
|
|
623
|
+
i++;
|
|
624
|
+
let buf = "";
|
|
625
|
+
while (i < n && s[i] !== '"') {
|
|
626
|
+
if (s[i] === "\\" && i + 1 < n) { buf += s[i + 1]; i += 2; continue; }
|
|
627
|
+
buf += s[i];
|
|
628
|
+
i++;
|
|
629
|
+
}
|
|
630
|
+
if (i < n) i++;
|
|
631
|
+
value = buf;
|
|
632
|
+
} else {
|
|
633
|
+
const valueStart = i;
|
|
634
|
+
while (i < n && s[i] !== ";" && s[i] !== "," && !space.includes(s[i])) i++;
|
|
635
|
+
value = s.slice(valueStart, i);
|
|
636
|
+
}
|
|
637
|
+
}
|
|
638
|
+
if (name && !(name in params)) params[name] = value;
|
|
639
|
+
}
|
|
640
|
+
out.push({ href: href, params: params });
|
|
641
|
+
}
|
|
642
|
+
return out;
|
|
643
|
+
}
|
|
644
|
+
|
|
645
|
+
// Move to the start of the next comma separated value without stopping inside a quoted
|
|
646
|
+
// string, so a comma in title="a, b" does not split one value into two.
|
|
647
|
+
function nextLinkValue(s, from) {
|
|
648
|
+
let quote = "";
|
|
649
|
+
for (let j = from; j < s.length; j++) {
|
|
650
|
+
const c = s[j];
|
|
651
|
+
if (quote) {
|
|
652
|
+
if (c === "\\") { j++; continue; }
|
|
653
|
+
if (c === quote) quote = "";
|
|
654
|
+
continue;
|
|
655
|
+
}
|
|
656
|
+
if (c === '"') { quote = c; continue; }
|
|
657
|
+
if (c === ",") return j + 1;
|
|
658
|
+
}
|
|
659
|
+
return s.length;
|
|
660
|
+
}
|
|
661
|
+
|
|
497
662
|
// v2 of the llms.txt proposal (August 2026) left the file format alone and added one
|
|
498
663
|
// thing: a page should say where its markdown version and its llms.txt are, using
|
|
499
664
|
// rel="alternate" type="text/markdown" and rel="describedby", as HTML link elements or
|
|
@@ -513,24 +678,26 @@ export function findLinkRelations(html, linkHeader) {
|
|
|
513
678
|
// NAME is "link<link", not as a link element, so \b would count a relation the site
|
|
514
679
|
// does not publish. Measured against parse5, 2026-08-24.
|
|
515
680
|
if (!/^<link(?=[\s/>])/i.test(tag)) continue;
|
|
516
|
-
//
|
|
517
|
-
//
|
|
518
|
-
//
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
681
|
+
// Attributes come from ONE quote aware scan, not from three independent regex matches
|
|
682
|
+
// over the whole tag. Those three could not tell an attribute from the quoted CONTENT
|
|
683
|
+
// of another attribute, so data-note=" rel='alternate' type='text/markdown'
|
|
684
|
+
// href='/fake.md'" reported a relation the page does not publish. The earlier
|
|
685
|
+
// data-rel fix, 2026-08-29, moved the START of the name and does not reach this.
|
|
686
|
+
// Measured and fixed 2026-09-09.
|
|
687
|
+
const attrs = tagAttributes(tag);
|
|
688
|
+
const rel = (attrs.rel || "").toLowerCase().trim().split(/\s+/);
|
|
689
|
+
const type = (attrs.type || "").toLowerCase().trim();
|
|
690
|
+
const href = (attrs.href || "").trim();
|
|
522
691
|
// text/markdown, not anything that starts with it, and a relation without a target is
|
|
523
692
|
// not a relation: both passed until 2026-08-29.
|
|
524
693
|
const isMarkdown = type.split(";")[0].trim() === "text/markdown";
|
|
525
694
|
if (!found.describedby && href && rel.includes("describedby")) found.describedby = href;
|
|
526
695
|
if (!found.markdown && href && rel.includes("alternate") && isMarkdown) found.markdown = href;
|
|
527
696
|
}
|
|
528
|
-
for (const
|
|
529
|
-
const
|
|
530
|
-
const
|
|
531
|
-
const
|
|
532
|
-
const rel = ((part.match(/(?:^|[;\s])rel\s*=\s*"?([^";,]+)"?/i) || [])[1] || "").toLowerCase().trim().split(/\s+/);
|
|
533
|
-
const type = ((part.match(/(?:^|[;\s])type\s*=\s*"?([^";,]+)"?/i) || [])[1] || "").toLowerCase().trim();
|
|
697
|
+
for (const link of parseLinkHeader(linkHeader)) {
|
|
698
|
+
const href = link.href;
|
|
699
|
+
const rel = (link.params.rel || "").toLowerCase().trim().split(/\s+/);
|
|
700
|
+
const type = (link.params.type || "").toLowerCase().trim();
|
|
534
701
|
const isMarkdownHeader = type.split(";")[0].trim() === "text/markdown";
|
|
535
702
|
if (!found.describedby && href && rel.includes("describedby")) found.describedby = href;
|
|
536
703
|
if (!found.markdown && href && rel.includes("alternate") && isMarkdownHeader) found.markdown = href;
|