ssqch 0.13.0 → 0.13.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bundled/docs/netlify.md +2 -2
- package/bundled/docs/plugins.md +1 -1
- package/bundled/docs/robots.md +15 -2
- package/bundled/ssqch.d.ts +9 -3
- package/lib/{chunk-2FKYWYAB.js → chunk-GM5SHCTU.js} +9 -5
- package/lib/cli.js +1 -1
- package/lib/llms.js +105 -14
- package/package.json +1 -1
- package/types/llms.d.ts +8 -2
package/bundled/docs/netlify.md
CHANGED
|
@@ -18,11 +18,11 @@ Netlify can build the site itself, with no deploy token to manage. With `ssqch`
|
|
|
18
18
|
paths = [".cache/ssqch"]
|
|
19
19
|
```
|
|
20
20
|
|
|
21
|
-
- **The `ssqch` dev dependency** pins the version, so a new SiteSquatch release can't surprise a deploy. Pin it exactly (`pnpm add -D ssqch@0.13.
|
|
21
|
+
- **The `ssqch` dev dependency** pins the version, so a new SiteSquatch release can't surprise a deploy. Pin it exactly (`pnpm add -D ssqch@0.13.1`) and bump it when you're ready.
|
|
22
22
|
- **`NODE_VERSION`** is the Node that runs the build: anything from 22.18 on. ssqch builds the same bytes on every supported Node, so pick the one your other build tools like.
|
|
23
23
|
- **[`netlify-plugin-cache`](https://github.com/jakejarvis/netlify-plugin-cache)** carries `.cache/ssqch` (the encoded images and their hashes, highlighted code, and any `fetch` copies) over to the next build. Add it to your `devDependencies`.
|
|
24
24
|
|
|
25
|
-
A site without a `package.json` can run a pinned release straight from npm in the build command, with `SSQCH_VERSION = "0.13.
|
|
25
|
+
A site without a `package.json` can run a pinned release straight from npm in the build command, with `SSQCH_VERSION = "0.13.1"` under `[build.environment]` to pin it (and `NODE_VERSION` there as well):
|
|
26
26
|
|
|
27
27
|
```toml
|
|
28
28
|
[build]
|
package/bundled/docs/plugins.md
CHANGED
|
@@ -9,7 +9,7 @@ title: "The gear shed: built-in plugins"
|
|
|
9
9
|
- **`ssqch/sass`:** `sassPlugin({ files: { "src/scss": "/css" } })` compiles Sass with [grass](https://github.com/connorskees/grass), built into the package as WebAssembly, so there's no Dart Sass to install. grass writes a few things differently from Dart Sass; a site that needs Dart Sass's exact output can run the `sass` package from `beforeBuild` instead (`{ run: "sass src/scss:dist/css" }`). A folder compiles every `.scss` and `.sass` file in it that doesn't start with `_`, keeping their paths; a file (`{ "src/app.scss": "/app.css" }`) compiles to the path given. The CSS is written at the start of each build, so `ssqch/assets` can fingerprint it and the link checker sees it, and `ssqch serve` rebuilds when a source changes. Options: `files` (required), `style` (`"compressed"` for production builds and `"expanded"` otherwise, by default), `loadPaths`, and `charset`: `false` leaves out what grass (as Dart Sass) starts CSS with when it has a character outside ASCII, a byte order mark when compressed and `@charset "UTF-8";` when expanded, so the file starts as a Jekyll site's did (jekyll-sass-converter takes the mark out). An import that names its extension (`@import "primer-core/index.scss"`) is found in the load paths too, as Dart Sass finds it, and a stylesheet that starts with Jekyll's front matter (`---` lines) compiles, the lines left blank. (Yes, that makes it Sassquatch.)
|
|
10
10
|
- **`ssqch/sitemap`:** `sitemapPlugin({ url: "https://example.com" })` writes `sitemap.xml` listing every HTML page with its absolute URL and its date as `<lastmod>`. Left out: pages with `sitemap: false` in their data, 404 pages, pages 2 and on of a list split into pages (pagination with a `size` over 1; a page per tag is still listed), and URLs matching `exclude` globs (`["/tags/**"]`). Options: `url` (the site's address, required: a sitemap lists whole addresses, and search engines read only those, so a build without one stops and asks for it; jekyll-sitemap wrote paths from the root instead), `exclude`, `lastmod` (default `true`), `output` (default `"/sitemap.xml"`), `paginated` (list every page of a split list) and `alternates`, a page's translations as hreflang links: a function that gets each page (`url`, `inputPath`, `outputPath`, `date`, `data`) and returns `[{ hreflang: "fr", href: "/fr/guide/" }]`, the page itself included (`x-default` too, if you have one), or nothing. An `href` from the site's root gets the address and `base` in front, and the sitemap gets the `xhtml` namespace only when a page has some: `alternates: (page) => page.data.translations?.map((t) => ({ hreflang: t.lang, href: t.url }))`. It's called at once, so work the translations out in data or a collection first.
|
|
11
11
|
- **`ssqch/elements`:** `elementsPlugin()` is interactivity without a framework. A custom element is a tag with a hyphen (`<like-button>`) and a script that defines it. Put `like-button.js` in `src/elements/` (and `like-button.css` next to it, if it has styles), write the tag in any page or layout, and that page gets the script and the stylesheet; pages without the tag get neither, so a page with no elements loads no JavaScript. What's inside the tag is the fallback a visitor sees before the script runs, or without JavaScript: `<like-button><a href="/likes/">3 likes</a></like-button>`. `ssqch:load` on a tag says when its script loads: `eager` (with the page, the default), `idle` (when the browser has a quiet moment) or `visible` (when the element scrolls into view), which is the "islands" idea without a framework to run it. The whole folder is copied to `/elements/`, so elements can import shared modules (a file without a hyphen in its name isn't an element). Options: `dir` (default: `elements` in the input folder, which has to exist), `url` (default `/elements/`) and `load` (`eager`, `idle` or `visible`: the default for tags without `ssqch:load`; any other value is an error that says so). The [`islands` example](https://github.com/reverentgeek/sitesquatch/tree/main/examples/sites/islands/) has three small elements.
|
|
12
|
-
- **`ssqch/llms`:** `llmsPlugin({ url: "https://example.com", description: "Hikes and the gear for them." })` writes the site for language models, as [llmstxt.org](https://llmstxt.org) describes it: `/llms.txt` lists the site's pages (each one's title, a link to its Markdown and its `description`), under a `##` heading for each of the site's top folders (`/guide/...` is "Guide"; the top's pages are "Pages"), and each of those pages gets a Markdown copy beside its URL as llms.txt's v2 proposal has it (`/guide/intro.html` is `/guide/intro.md`; a URL without a file name gets `index.md`, so `/guide/intro/` is `/guide/intro/index.md` and `/` is `/index.md`). A copy is what the page says: a Markdown page's copy is its Markdown after its template engine ran (`{{ site.title }}` is the title, a `{% for %}` is the list it wrote), without its front matter, its `<script>` and `<style>` blocks, or pictures with `alt=""` (decoration), and with its title on top when only the front matter had it, and its pictures (and links to them) at the addresses the page shows, so a picture `imagePlugin()` published under another name links the published file; any other page's copy (a home page in Jinja) is its `<article>`, or else its `<main>` or `<body>`, as Markdown (headings, paragraphs, lists, links, code, quotes, tables and described pictures; navigation, forms, scripts and anything `hidden` are left out). Links in a copy are whole addresses (with `url`), and a link to a page that has a copy goes to the copy, a relative `.md` link to another page's file included (`[Install](install.md)`). `/llms-full.txt` has the heading, then every copy in one file, each after a `url:` line. A page whose front matter says `llms: false` is left out. The plugin adds one template function, `markdownCopy(page.url)`: the URL of the page's copy, or `""` when it has none, for the layout to link it (a build adds no markup of its own): `{% if markdownCopy(page.url) %}<link rel="alternate" type="text/markdown" href="{{ markdownCopy(page.url) }}">{% endif %}`. Options: `title` (the `#` heading; default `site.title` in the site's data, or the home page's title), `description` (a `>` line under it), `details` (Markdown after that, in both files: what a model should know first), `url` (the site's address, for whole links; without it, links are from the site's root), `exclude` (URL globs, like `["/drafts/**"]`; 404 pages are always left out), `copies` (default `true`; `false` links `llms.txt` to the pages instead), `full` (default `true`), `section` (a function that gets each page and returns its section's name; sections come in the order of their first page) and `sort` (a function that compares two pages, as `Array.prototype.sort` takes it; default by URL). vitepress-plugin-llms does this for VitePress sites.
|
|
12
|
+
- **`ssqch/llms`:** `llmsPlugin({ url: "https://example.com", description: "Hikes and the gear for them." })` writes the site for language models, as [llmstxt.org](https://llmstxt.org) describes it: `/llms.txt` lists the site's pages (each one's title, a link to its Markdown and its `description`), under a `##` heading for each of the site's top folders (`/guide/...` is "Guide"; the top's pages are "Pages"), and each of those pages gets a Markdown copy beside its URL as llms.txt's v2 proposal has it (`/guide/intro.html` is `/guide/intro.md`; a URL without a file name gets `index.md`, so `/guide/intro/` is `/guide/intro/index.md` and `/` is `/index.md`). A copy is what the page says: a Markdown page's copy is its Markdown after its template engine ran (`{{ site.title }}` is the title, a `{% for %}` is the list it wrote), without its front matter, its `<script>` and `<style>` blocks, or pictures with `alt=""` (decoration), and with its title on top when only the front matter had it, and its pictures (and links to them) at the addresses the page shows, so a picture `imagePlugin()` published under another name links the published file; any other page's copy (a home page in Jinja) is its `<article>`, or else its `<main>` or `<body>`, as Markdown (headings, paragraphs, lists, links, code, quotes, tables and described pictures; navigation, forms, scripts and anything `hidden` are left out). Links in a copy are whole addresses (with `url`), and a link to a page that has a copy goes to the copy, a relative `.md` link to another page's file included (`[Install](install.md)`). `/llms-full.txt` has the heading, then every copy in one file, each after a `url:` line. A page whose front matter says `llms: false` is left out. The plugin adds one template function, `markdownCopy(page.url)`: the URL of the page's copy, or `""` when it has none, for the layout to link it (a build adds no markup of its own): `{% if markdownCopy(page.url) %}<link rel="alternate" type="text/markdown" href="{{ markdownCopy(page.url) }}">{% endif %}`. Options: `title` (the `#` heading; default `site.title` in the site's data, or the home page's title), `description` (a `>` line under it), `details` (Markdown after that, in both files: what a model should know first), `url` (the site's address, for whole links; without it, links are from the site's root), `exclude` (URL globs, like `["/drafts/**"]`; 404 pages are always left out), `copies` (default `true`; `false` links `llms.txt` to the pages instead), `full` (default `true`), `section` (a function that gets each page and returns its section's name; sections come in the order of their first page, with "Optional" last) and `sort` (a function that compares two pages, as `Array.prototype.sort` takes it; default by URL). `paths` (parts of the site with an `llms.txt` of their own, like `["/docs/"]`), `budget` (the most characters an `llms.txt` has before pages move into "Optional"; default 50,000) and the template function `llmsTxt(page.url)`, in [A big site's llms.txt](robots.md#a-big-sites-llmstxt). vitepress-plugin-llms does this for VitePress sites.
|
|
13
13
|
- **`ssqch/robots`:** `robotsPlugin({ ai: { training: false } })` writes `/robots.txt`: everyone's group (`disallow`, URL paths no crawler should read), a group with `Disallow: /` for each kind of AI crawler that's turned off (`ai`: `training`, `search` and `fetch`, each `true` by default), and the `Sitemap:` line with `sitemapPlugin()`'s address (`sitemap: false` leaves it out, a string is another address). [Posted signs](robots.md) lists the crawlers of each kind and the links a layout writes for agents.
|
|
14
14
|
- **`ssqch/vue`:** `vuePlugin(options)` renders the Vue components a site's pages use, at build time, with the site's own Vite, `@vitejs/plugin-vue` and Vue (`npm install -D vite @vitejs/plugin-vue vue`; ssqch doesn't bundle them, and says so if they're missing). With pnpm, which keeps a VitePress site's vite and @vitejs/plugin-vue beside `vitepress` rather than in the site's `node_modules`, they're found there. Vue renders with its production build in a production build and its development build while serving (`NODE_ENV` follows ssqch's env unless you set it). A page's `<script setup>` imports name them, as VitePress's pages do (`import ApiIndex from './ApiIndex.vue'`, then `<ApiIndex />` in the page), and `components` names the ones every page can use (`{ VueSchoolLink: ".vitepress/theme/components/VueSchoolLink.vue" }`, VitePress's `app.component()`). Each tag a page has (and no template in `_includes/components/`, which comes first) becomes the HTML Vue's server renderer writes for it: its attributes are its props, and a binding is read as Vue reads it in the page, with the page's front matter (`frontmatter`, `$frontmatter`), `params` (a dynamic route's, from its URL: `[partnerId].md` written as `/partners/vehikl.html`), `page` (VitePress's `useData().page`) and the `data` of each VitePress loader its `<script setup>` imports (`import { data } from './errors.data'`; the loader's `load()` runs once a build, through Vite, with `__dirname`), and what's between its tags is its default slot, with the components inside it rendered first (a layout component that holds the page works). With `fromVitepress()` from `ssqch/from-vitepress` (a site moving from VitePress), components that import from `"vitepress"` get a stand-in (with `current`, its page's state, for a wrapper that renders the site's theme: the VitePress trailhead's "Keeping your VitePress theme") for `useData()` (the page's front matter, title and route, and the config's `themeConfig`), `useRoute()`, `withBase()` and `inBrowser`, and `@theme/` is the theme's folder. `hydrate` runs each component in the browser too, as an island: `"load"`, `"idle"` or `"visible"` (when it scrolls into view) wraps its HTML in `<ssqch-vue>` and adds one script to the pages that have one, `/assets/vue/islands.js`, which loads only the components on the page; default `false`, HTML only. `css` is where the components' styles go (default `"/assets/vue.css"`, for your layout to link; `false` for none), and the fonts and images they point at go where Vite put them (`/assets/inter-x.woff2`, after `base`); the site's `public/` isn't copied (that's `passthrough`'s), `root` is Vite's root (default the input folder, so scope ids are VitePress's), and `vite` adds Vite options (`resolve.alias`, `plugins`). The components build in `.cache/ssqch/vue/` before each build; while serving, an edit that's only to pages keeps the last one. A VitePress loader named like ssqch's data (`api/api.data.ts`) goes in `ignore`, which `ssqch scout` drafts.
|
|
15
15
|
- **`ssqch/redirects`:** `redirectsPlugin(options)` keeps old URLs working. A page lists its old URLs in front matter (`aliases: [/2019/pine-cones/, /pine.html]`), and the `redirects` option adds others (`{ "/feed/": "/feed.xml" }`, where the new URL can be on another site). Each old URL gets a small page that sends visitors on (a meta refresh, a canonical link and `noindex`), and with `netlify: true`, a `/old /new 301` line is appended to `_redirects`, after any lines your own `_redirects` template writes, and [`linksPlugin()`](#) honors those lines. An old URL that's another page's URL is an error. For redirect pages in your own markup, add `_includes/hooks/redirect.*` (any engine): it gets `redirect.from` and `redirect.to` with the site's global data. Options: `pages` (default `true`), `netlify` (default `false`) and `redirects`.
|
package/bundled/docs/robots.md
CHANGED
|
@@ -55,6 +55,16 @@ Not written yet, because nothing reads them yet: the IETF draft's shared group f
|
|
|
55
55
|
|
|
56
56
|
`llmsPlugin()` writes `/llms.txt`, a Markdown copy of each page and `/llms-full.txt` ([Built-in plugins](plugins.md)). A copy is what the page says (a Markdown page's text after its template ran, a template page's `<article>` or `<main>` as Markdown), at the address llms.txt's proposal gives it: `/guide/intro/` has `/guide/intro/index.md`.
|
|
57
57
|
|
|
58
|
+
### A big site's llms.txt
|
|
59
|
+
|
|
60
|
+
`llms.txt` is the map an agent reads before it decides where to go, so it has to stay short enough to read whole (llms.txt's proposal: "small enough to fit in context"). Three options keep it that way:
|
|
61
|
+
|
|
62
|
+
- `paths`: parts of the site that get an `llms.txt` of their own, which covers the pages under it (llms.txt's v2 has agents use the most specific one). `paths: ["/docs/"]` writes `/docs/llms.txt` with the pages under `/docs/`, and `/llms.txt` lists that file once, where its first page would be, in place of them. An item is a URL path that starts and ends with `/`, or `{ path: "/docs/api/", title: "Trail API", description: "Every endpoint.", details: "..." }` for the file's `#` heading (default: the site's title and the folder's name, `Trail Guide: Docs`), its `>` line and its Markdown after that; the title and description are also its line in the file above it. Inside a file, a page's section is its folder under the path (`/docs/guide/start/` is "Guide" in `/docs/llms.txt`), unless `section` names one. A path can be under another (`/docs/` and `/docs/api/`), and a path with no listed pages writes no file and warns. `llms-full.txt` stays one file with every page.
|
|
63
|
+
- A section named "Optional" always comes last: llms.txt's convention for links an agent can skip when its context is short. `section: (page) => (page.date < new Date("2020-01-01") ? "Optional" : undefined)` puts old posts there and leaves the rest in their folder's section.
|
|
64
|
+
- `budget`: the most characters an `llms.txt` may have, 50,000 by default (where `ssqch patrol` warns, `llms-long`). Past it, pages from the end of the file move into "Optional" without their descriptions until it fits. A file that wouldn't fit even with every page moved stays as it is, and the build warns with what to do: give a part of the site its own file with `paths`, or raise `budget`. `budget: false` turns it off.
|
|
65
|
+
|
|
66
|
+
`ssqch patrol` reads `/llms.txt` and every `llms.txt` it links, with the same checks, and counts the pages they all list.
|
|
67
|
+
|
|
58
68
|
## The links in the layout
|
|
59
69
|
|
|
60
70
|
llms.txt's proposal (v2) has agents find both from any page by two links, which a layout writes in its `<head>` (a build adds no markup of its own):
|
|
@@ -62,10 +72,13 @@ llms.txt's proposal (v2) has agents find both from any page by two links, which
|
|
|
62
72
|
```html
|
|
63
73
|
<link rel="canonical" href="{{ site.url }}{{ page.url }}">
|
|
64
74
|
{% if markdownCopy(page.url) %}<link rel="alternate" type="text/markdown" href="{{ markdownCopy(page.url) }}">{% endif %}
|
|
65
|
-
<link rel="describedby" href="
|
|
75
|
+
<link rel="describedby" href="{{ llmsTxt(page.url) }}">
|
|
66
76
|
```
|
|
67
77
|
|
|
68
|
-
`markdownCopy(page.url)` comes with `llmsPlugin()`: the copy's URL, or `""` for a page without one. The starters' layouts have all three.
|
|
78
|
+
`markdownCopy(page.url)` comes with `llmsPlugin()`: the copy's URL, or `""` for a page without one. The starters' layouts have all three, with `/llms.txt` written out.
|
|
79
|
+
`llmsTxt(page.url)` comes with it too: the `llms.txt` that covers the page, `/docs/llms.txt` for a page under `/docs/` when `paths` has it and `/llms.txt` for the rest.
|
|
80
|
+
|
|
81
|
+
A copy is a Markdown file, with no `<head>` to point from: llms.txt's v2 points from it with an HTTP `Link` header, which a host's `_headers` file (Netlify, Cloudflare) sends for a folder, like `/docs/*` with `Link: </docs/llms.txt>; rel="describedby"`. ssqch adds no line to the copy itself, so a copy says what its page says and nothing more.
|
|
69
82
|
|
|
70
83
|
## When an agent asks for Markdown
|
|
71
84
|
|
package/bundled/ssqch.d.ts
CHANGED
|
@@ -644,12 +644,18 @@ declare module "ssqch/llms" {
|
|
|
644
644
|
copies?: boolean;
|
|
645
645
|
/** `/llms-full.txt`, every listed page's Markdown in one file, each after a `url:` line. Default `true`. */
|
|
646
646
|
full?: boolean;
|
|
647
|
-
/** A page's section in `llms.txt`: a function that gets each page and returns a name. Default: its first folder (`/guide/...` is "Guide"; the top's pages are "Pages"). Sections are in the order their first page comes. */
|
|
647
|
+
/** A page's section in `llms.txt`: a function that gets each page and returns a name. Default: its first folder (`/guide/...` is "Guide"; the top's pages are "Pages"). Sections are in the order their first page comes, with "Optional" (links an agent can skip) last. Inside a file of `paths`, the default is the page's folder under that path. */
|
|
648
648
|
section?: (page: LlmsPage) => string | undefined;
|
|
649
649
|
/** The pages' order, as `Array.prototype.sort` takes it: `(a, b) => a.data.order - b.data.order`. Default: by URL. */
|
|
650
650
|
sort?: (a: LlmsPage, b: LlmsPage) => number;
|
|
651
|
-
|
|
652
|
-
|
|
651
|
+
/** Parts of the site with an `llms.txt` of their own, as llms.txt's v2 has them: `["/docs/"]` writes `/docs/llms.txt` with the pages under `/docs/`, and `/llms.txt` lists that file in their place. An item is a URL path that starts and ends with `/`, or an object with its `path` and the file's `title` (its heading; default the site's title and the folder's name), `description` and `details`. A page belongs to the deepest path it's under. A path with no listed pages writes no file, and the build warns. */
|
|
652
|
+
paths?: Array<string | LlmsPath>;
|
|
653
|
+
/** The most characters an `llms.txt` has: past it, pages from the end move into "Optional" without their descriptions until it fits. A file that wouldn't fit even so stays as it is, and the build warns. Default `50000` (where `ssqch patrol` warns); `false` for no limit. */
|
|
654
|
+
budget?: number | false;
|
|
655
|
+
}
|
|
656
|
+
/** A part of the site with an `llms.txt` of its own, in `paths`. */
|
|
657
|
+
export interface LlmsPath { path: string; title?: string; description?: string; details?: string }
|
|
658
|
+
/** Writes `/llms.txt` (the site's pages, with links to their Markdown), a Markdown copy of each, and `/llms-full.txt`, for language models, as llmstxt.org describes; and adds the template functions `markdownCopy(page.url)`, the URL of a page's copy (or `""` when it has none), for a layout's `<link rel="alternate" type="text/markdown">`, and `llmsTxt(page.url)`, the `llms.txt` that covers the page, for its `<link rel="describedby">`. */
|
|
653
659
|
export function llmsPlugin(options?: LlmsPluginOptions): BuiltinPlugin;
|
|
654
660
|
}
|
|
655
661
|
|
|
@@ -155,12 +155,15 @@ function patrol(dir) {
|
|
|
155
155
|
const llms = has.has("llms.txt") ? read("llms.txt") : void 0;
|
|
156
156
|
const listed = /* @__PURE__ */ new Set();
|
|
157
157
|
if (llms === void 0) say("no-llms", "no llms.txt: an agent gets no map of the site (llmsPlugin() writes one, with a Markdown copy of each page)");
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
158
|
+
const maps = llms === void 0 ? [] : ["llms.txt"];
|
|
159
|
+
for (let i = 0; i < maps.length; i++) {
|
|
160
|
+
const name = maps[i];
|
|
161
|
+
const text = i === 0 ? llms : read(name);
|
|
162
|
+
if (!/^\uFEFF?# \S/.test(text)) say("llms-shape", "llms.txt doesn't start with a `# ` heading (the site's name), which llmstxt.org's format requires", name);
|
|
163
|
+
if (text.length > LLMS_LIMIT) say("llms-long", `llms.txt is ${count(text.length)} characters, over ${count(LLMS_LIMIT)}: more than an agent reads before deciding where to go (link section files from it instead)`, name);
|
|
164
|
+
for (const href of linksIn(text)) {
|
|
162
165
|
if (relative(href)) {
|
|
163
|
-
say("relative-links", "links that aren't whole addresses (or from the site's root): they break once the text is read on its own",
|
|
166
|
+
say("relative-links", "links that aren't whole addresses (or from the site's root): they break once the text is read on its own", name);
|
|
164
167
|
continue;
|
|
165
168
|
}
|
|
166
169
|
const file = fileOf(href);
|
|
@@ -169,6 +172,7 @@ function patrol(dir) {
|
|
|
169
172
|
else {
|
|
170
173
|
if (!/\.(md|markdown|txt)$/i.test(file)) say("llms-html", "llms.txt links to pages, not their Markdown: an agent reads Markdown without the page's chrome", href);
|
|
171
174
|
listed.add(file);
|
|
175
|
+
if (/(^|\/)llms\.txt$/.test(file) && !maps.includes(file)) maps.push(file);
|
|
172
176
|
}
|
|
173
177
|
}
|
|
174
178
|
}
|
package/lib/cli.js
CHANGED
|
@@ -558,7 +558,7 @@ async function runDiff(dirs, options) {
|
|
|
558
558
|
async function runPatrol(dir, shown, format) {
|
|
559
559
|
try {
|
|
560
560
|
checkFormat(format);
|
|
561
|
-
const { patrol, patrolJson, patrolText } = await import("./chunk-
|
|
561
|
+
const { patrol, patrolJson, patrolText } = await import("./chunk-GM5SHCTU.js");
|
|
562
562
|
const report = patrol(dir);
|
|
563
563
|
process.stdout.write(format === "json" ? JSON.stringify(patrolJson(report, shown), null, 2) + "\n" : patrolText(report, shown));
|
|
564
564
|
return report.findings.length ? 1 : 0;
|
package/lib/llms.js
CHANGED
|
@@ -26,7 +26,9 @@ import "./chunk-FJML7INB.js";
|
|
|
26
26
|
import "./chunk-4AZFCSGJ.js";
|
|
27
27
|
import "./chunk-CMUWRPSQ.js";
|
|
28
28
|
import {
|
|
29
|
-
bail
|
|
29
|
+
bail,
|
|
30
|
+
warn,
|
|
31
|
+
warning
|
|
30
32
|
} from "./chunk-YNJNMQWU.js";
|
|
31
33
|
import {
|
|
32
34
|
compareText
|
|
@@ -386,13 +388,19 @@ function copyUrl(url) {
|
|
|
386
388
|
if (url.endsWith("/")) return `${url}index.md`;
|
|
387
389
|
return url.replace(/\.html?$/, "") + ".md";
|
|
388
390
|
}
|
|
391
|
+
function folderName(folder) {
|
|
392
|
+
const name = folder.replace(/[-_]+/g, " ");
|
|
393
|
+
return name.charAt(0).toUpperCase() + name.slice(1);
|
|
394
|
+
}
|
|
395
|
+
var BUDGET = 5e4;
|
|
389
396
|
function sectionOf(url) {
|
|
390
397
|
const first = url.split("/").filter(Boolean);
|
|
391
398
|
if (first.length < 2 && !url.endsWith("/")) return "Pages";
|
|
392
399
|
if (first.length === 0) return "Pages";
|
|
393
|
-
|
|
394
|
-
return name.charAt(0).toUpperCase() + name.slice(1);
|
|
400
|
+
return folderName(first[0]);
|
|
395
401
|
}
|
|
402
|
+
var grouped = (n) => String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ",");
|
|
403
|
+
var shown = (v) => typeof v === "string" ? debugString(v) : JSON.stringify(v) ?? typeof v;
|
|
396
404
|
var oneLine = (s) => typeof s === "string" ? s.replace(/\s+/g, " ").trim() : "";
|
|
397
405
|
var isPageUrl = (url) => url.endsWith("/") || url.endsWith(".html");
|
|
398
406
|
var keyOf = (url) => url.replace(/\/index\.html$/, "/");
|
|
@@ -475,9 +483,41 @@ function publishedImages(markdown, outputPath, base, output) {
|
|
|
475
483
|
return became.has(href) ? `${open}"${became.get(href)}"` : whole;
|
|
476
484
|
});
|
|
477
485
|
}
|
|
486
|
+
function listing(head, items, budget, file) {
|
|
487
|
+
const render = (list2) => {
|
|
488
|
+
const sections = /* @__PURE__ */ new Map();
|
|
489
|
+
for (const item of list2) sections.set(item.section, [...sections.get(item.section) ?? [], item.line]);
|
|
490
|
+
const optional = sections.get("Optional");
|
|
491
|
+
if (optional) {
|
|
492
|
+
sections.delete("Optional");
|
|
493
|
+
sections.set("Optional", optional);
|
|
494
|
+
}
|
|
495
|
+
let txt2 = head;
|
|
496
|
+
for (const [name, lines] of sections) txt2 += `
|
|
497
|
+
## ${name}
|
|
498
|
+
|
|
499
|
+
${lines.join("")}`;
|
|
500
|
+
return txt2;
|
|
501
|
+
};
|
|
502
|
+
let txt = render(items);
|
|
503
|
+
if (txt.length <= budget) return txt;
|
|
504
|
+
const moved = items.map((item) => ({ ...item }));
|
|
505
|
+
let over = txt.length - budget + (items.some((i) => i.section === "Optional") ? 0 : "\n## Optional\n\n".length);
|
|
506
|
+
const most = items.reduce((n, i) => n + (i.section === "Optional" ? 0 : i.line.length - i.short.length), 0);
|
|
507
|
+
if (most < over) {
|
|
508
|
+
warn(warning("build", `llmsPlugin: ${file} is ${grouped(txt.length)} characters, over the budget of ${grouped(budget)} even without its pages' notes: give a part of the site an llms.txt of its own (paths: ["/docs/"]), or raise budget`));
|
|
509
|
+
return txt;
|
|
510
|
+
}
|
|
511
|
+
for (let i = moved.length - 1; i >= 0 && over > 0; i--) {
|
|
512
|
+
if (moved[i].section === "Optional") continue;
|
|
513
|
+
over -= moved[i].line.length - moved[i].short.length;
|
|
514
|
+
moved[i] = { section: "Optional", line: moved[i].short, short: moved[i].short };
|
|
515
|
+
}
|
|
516
|
+
return render(moved);
|
|
517
|
+
}
|
|
478
518
|
BUILTINS.set("ssqch/llms", (config, given) => {
|
|
479
519
|
if (given !== null && typeof given === "object" && Object.hasOwn(given, "hint")) bail('`hint` is gone: a build adds no markup of its own. Link the copy from your layout\'s <head> instead: <link rel="alternate" type="text/markdown" href="{{ markdownCopy(page.url) }}">');
|
|
480
|
-
const o = new Options(given, ["title", "description", "details", "url", "exclude", "copies", "full", "section", "sort"]);
|
|
520
|
+
const o = new Options(given, ["title", "description", "details", "url", "exclude", "copies", "full", "section", "sort", "paths", "budget"]);
|
|
481
521
|
const url = o.string("url");
|
|
482
522
|
if (url !== void 0 && url !== "" && !/^https?:\/\//.test(url)) bail(`url must be the site's address, like "https://example.com" (got ${debugString(url)})`);
|
|
483
523
|
const exclude = o.list("exclude") ?? [];
|
|
@@ -493,6 +533,27 @@ BUILTINS.set("ssqch/llms", (config, given) => {
|
|
|
493
533
|
const description = o.string("description");
|
|
494
534
|
const details = o.string("details");
|
|
495
535
|
const address = (url ?? "").replace(/\/+$/, "");
|
|
536
|
+
const givenBudget = o.get("budget");
|
|
537
|
+
if (givenBudget !== void 0 && givenBudget !== false && !(typeof givenBudget === "number" && Number.isInteger(givenBudget) && givenBudget > 0)) bail(`budget must be a number of characters, like 50000, or false for no limit (got ${shown(givenBudget)})`);
|
|
538
|
+
const budget = givenBudget === false ? Infinity : givenBudget ?? BUDGET;
|
|
539
|
+
const paths = [];
|
|
540
|
+
const givenPaths = o.get("paths");
|
|
541
|
+
if (givenPaths !== void 0 && !Array.isArray(givenPaths)) bail(`paths must be a list of URL paths that get an llms.txt of their own, like ["/docs/"] (got ${shown(givenPaths)})`);
|
|
542
|
+
for (const item of givenPaths ?? []) {
|
|
543
|
+
const one = typeof item === "string" ? { path: item } : item;
|
|
544
|
+
if (one === null || typeof one !== "object" || typeof one.path !== "string") bail(`paths: each one is a URL path like "/docs/", or { path: "/docs/", title, description, details } (got ${shown(item)})`);
|
|
545
|
+
const { path: at, ...rest } = one;
|
|
546
|
+
for (const [key, value] of Object.entries(rest)) {
|
|
547
|
+
if (!["title", "description", "details"].includes(key)) bail(`paths: ${shown(at)} has \`${key}\`, which isn't an option; it takes title, description and details`);
|
|
548
|
+
if (typeof value !== "string") bail(`paths: ${shown(at)}'s ${key} must be a string (got ${shown(value)})`);
|
|
549
|
+
}
|
|
550
|
+
const folder = at;
|
|
551
|
+
if (!folder.startsWith("/") || !folder.endsWith("/") || folder === "/") bail(`paths: ${shown(folder)} must be a folder's URL path, starting and ending with /, like "/docs/" (/llms.txt is the site's)`);
|
|
552
|
+
if (paths.some((f) => f.path === folder)) bail(`paths: ${shown(folder)} is listed twice`);
|
|
553
|
+
paths.push({ path: folder, ...rest });
|
|
554
|
+
}
|
|
555
|
+
paths.sort((a, b) => b.path.length - a.path.length || compareText(a.path, b.path));
|
|
556
|
+
const coveringFile = (pageUrl) => paths.find((f) => pageUrl.startsWith(f.path))?.path ?? "/";
|
|
496
557
|
const listedUrl = (pageUrl) => isPageUrl(pageUrl) && !excluded(pageUrl) && pageUrl !== "/404.html" && pageUrl !== "/404/";
|
|
497
558
|
const listed = (p) => p.outputPath.endsWith(".html") && listedUrl(p.url) && p.data.llms !== false;
|
|
498
559
|
const sources = /* @__PURE__ */ new Map();
|
|
@@ -517,6 +578,7 @@ BUILTINS.set("ssqch/llms", (config, given) => {
|
|
|
517
578
|
if (page && page.url === pageUrl && data?.llms === false) return "";
|
|
518
579
|
return copyUrl(pageUrl);
|
|
519
580
|
});
|
|
581
|
+
config.filters.set("llmsTxt", (value) => `${coveringFile(value === void 0 || value === null ? "/" : String(value))}llms.txt`);
|
|
520
582
|
config.generators.push(["ssqch/llms", (config2, pages) => {
|
|
521
583
|
const base = config2.base.replace(/\/$/, "");
|
|
522
584
|
const output = path.resolve(config2.root, config2.output);
|
|
@@ -561,7 +623,7 @@ BUILTINS.set("ssqch/llms", (config, given) => {
|
|
|
561
623
|
${body}`;
|
|
562
624
|
body = publishedImages(body, p.outputPath, base, output);
|
|
563
625
|
body = rewriteLinks(body, (href) => target(href, p));
|
|
564
|
-
let name
|
|
626
|
+
let name;
|
|
565
627
|
if (typeof section === "function") {
|
|
566
628
|
let out;
|
|
567
629
|
try {
|
|
@@ -586,18 +648,47 @@ ${body}`;
|
|
|
586
648
|
if (details?.trim()) head += `
|
|
587
649
|
${details.trim()}
|
|
588
650
|
`;
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
651
|
+
const written = paths.filter((f) => {
|
|
652
|
+
if (entries.some((e) => coveringFile(e.url) === f.path)) return true;
|
|
653
|
+
warn(warning("build", `llmsPlugin: no listed pages under ${f.path}, so ${f.path}llms.txt isn't written (paths in ssqch.config.js)`));
|
|
654
|
+
return false;
|
|
655
|
+
});
|
|
656
|
+
const fileFor = (pageUrl) => written.find((f) => pageUrl.startsWith(f.path))?.path ?? "/";
|
|
657
|
+
const parentOf = (inner) => written.find((f) => f.path !== inner && inner.startsWith(f.path))?.path ?? "/";
|
|
658
|
+
const nameOf = (f) => f.title ?? folderName(f.path.split("/").filter(Boolean).at(-1) ?? "");
|
|
659
|
+
const link = (title2, href, note) => `- [${title2.replace(/([[\]])/g, "\\$1")}](${href})${note ? `: ${note}` : ""}
|
|
596
660
|
`;
|
|
597
|
-
|
|
661
|
+
for (const file of [{ path: "/" }, ...written]) {
|
|
662
|
+
const items = [];
|
|
663
|
+
const listedFiles = /* @__PURE__ */ new Set();
|
|
664
|
+
for (const e of entries) {
|
|
665
|
+
let at = fileFor(e.url);
|
|
666
|
+
if (at === file.path) {
|
|
667
|
+
const name2 = e.section ?? sectionOf(`/${e.url.slice(file.path.length)}`);
|
|
668
|
+
items.push({ section: name2, line: link(e.title, whole(copies ? e.copy : e.url), e.description), short: link(e.title, whole(copies ? e.copy : e.url), "") });
|
|
669
|
+
continue;
|
|
670
|
+
}
|
|
671
|
+
if (file.path !== "/" && !at.startsWith(file.path)) continue;
|
|
672
|
+
while (at !== "/" && parentOf(at) !== file.path) at = parentOf(at);
|
|
673
|
+
if (at === "/" || listedFiles.has(at)) continue;
|
|
674
|
+
listedFiles.add(at);
|
|
675
|
+
const inner = written.find((f) => f.path === at);
|
|
676
|
+
const name = e.section ?? sectionOf(`/${e.url.slice(file.path.length)}`);
|
|
677
|
+
items.push({ section: name, line: link(nameOf(inner), whole(`${inner.path}llms.txt`), oneLine(inner.description)), short: link(nameOf(inner), whole(`${inner.path}llms.txt`), "") });
|
|
678
|
+
}
|
|
679
|
+
let top = head;
|
|
680
|
+
if (file.path !== "/") {
|
|
681
|
+
top = `# ${file.title ?? `${siteTitle}: ${nameOf(file)}`}
|
|
682
|
+
`;
|
|
683
|
+
if (file.description) top += `
|
|
684
|
+
> ${oneLine(file.description)}
|
|
598
685
|
`;
|
|
686
|
+
if (file.details?.trim()) top += `
|
|
687
|
+
${file.details.trim()}
|
|
688
|
+
`;
|
|
689
|
+
}
|
|
690
|
+
files.push({ path: `${file.path}llms.txt`, contents: listing(top, items, budget, `${file.path}llms.txt`) });
|
|
599
691
|
}
|
|
600
|
-
files.push({ path: "/llms.txt", contents: txt });
|
|
601
692
|
if (copies) for (const e of entries) files.push({ path: e.copy, contents: e.markdown });
|
|
602
693
|
if (full) files.push({ path: "/llms-full.txt", contents: `${head}
|
|
603
694
|
${entries.map((e) => `---
|
package/package.json
CHANGED
package/types/llms.d.ts
CHANGED
|
@@ -18,10 +18,16 @@ export interface LlmsPluginOptions {
|
|
|
18
18
|
copies?: boolean;
|
|
19
19
|
/** `/llms-full.txt`, every listed page's Markdown in one file, each after a `url:` line. Default `true`. */
|
|
20
20
|
full?: boolean;
|
|
21
|
-
/** A page's section in `llms.txt`: a function that gets each page and returns a name. Default: its first folder (`/guide/...` is "Guide"; the top's pages are "Pages"). Sections are in the order their first page comes. */
|
|
21
|
+
/** A page's section in `llms.txt`: a function that gets each page and returns a name. Default: its first folder (`/guide/...` is "Guide"; the top's pages are "Pages"). Sections are in the order their first page comes, with "Optional" (links an agent can skip) last. Inside a file of `paths`, the default is the page's folder under that path. */
|
|
22
22
|
section?: (page: LlmsPage) => string | undefined;
|
|
23
23
|
/** The pages' order, as `Array.prototype.sort` takes it: `(a, b) => a.data.order - b.data.order`. Default: by URL. */
|
|
24
24
|
sort?: (a: LlmsPage, b: LlmsPage) => number;
|
|
25
|
+
/** Parts of the site with an `llms.txt` of their own, as llms.txt's v2 has them: `["/docs/"]` writes `/docs/llms.txt` with the pages under `/docs/`, and `/llms.txt` lists that file in their place. An item is a URL path that starts and ends with `/`, or an object with its `path` and the file's `title` (its heading; default the site's title and the folder's name), `description` and `details`. A page belongs to the deepest path it's under. A path with no listed pages writes no file, and the build warns. */
|
|
26
|
+
paths?: Array<string | LlmsPath>;
|
|
27
|
+
/** The most characters an `llms.txt` has: past it, pages from the end move into "Optional" without their descriptions until it fits. A file that wouldn't fit even so stays as it is, and the build warns. Default `50000` (where `ssqch patrol` warns); `false` for no limit. */
|
|
28
|
+
budget?: number | false;
|
|
25
29
|
}
|
|
26
|
-
/**
|
|
30
|
+
/** A part of the site with an `llms.txt` of its own, in `paths`. */
|
|
31
|
+
export interface LlmsPath { path: string; title?: string; description?: string; details?: string }
|
|
32
|
+
/** Writes `/llms.txt` (the site's pages, with links to their Markdown), a Markdown copy of each, and `/llms-full.txt`, for language models, as llmstxt.org describes; and adds the template functions `markdownCopy(page.url)`, the URL of a page's copy (or `""` when it has none), for a layout's `<link rel="alternate" type="text/markdown">`, and `llmsTxt(page.url)`, the `llms.txt` that covers the page, for its `<link rel="describedby">`. */
|
|
27
33
|
export function llmsPlugin(options?: LlmsPluginOptions): BuiltinPlugin;
|