ssqch 0.13.0 → 0.13.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -18,11 +18,11 @@ Netlify can build the site itself, with no deploy token to manage. With `ssqch`
18
18
  paths = [".cache/ssqch"]
19
19
  ```
20
20
 
21
- - **The `ssqch` dev dependency** pins the version, so a new SiteSquatch release can't surprise a deploy. Pin it exactly (`pnpm add -D ssqch@0.13.0`) and bump it when you're ready.
21
+ - **The `ssqch` dev dependency** pins the version, so a new SiteSquatch release can't surprise a deploy. Pin it exactly (`pnpm add -D ssqch@0.13.1`) and bump it when you're ready.
22
22
  - **`NODE_VERSION`** is the Node that runs the build: anything from 22.18 on. ssqch builds the same bytes on every supported Node, so pick the one your other build tools like.
23
23
  - **[`netlify-plugin-cache`](https://github.com/jakejarvis/netlify-plugin-cache)** carries `.cache/ssqch` (the encoded images and their hashes, highlighted code, and any `fetch` copies) over to the next build. Add it to your `devDependencies`.
24
24
 
25
- A site without a `package.json` can run a pinned release straight from npm in the build command, with `SSQCH_VERSION = "0.13.0"` under `[build.environment]` to pin it (and `NODE_VERSION` there as well):
25
+ A site without a `package.json` can run a pinned release straight from npm in the build command, with `SSQCH_VERSION = "0.13.1"` under `[build.environment]` to pin it (and `NODE_VERSION` there as well):
26
26
 
27
27
  ```toml
28
28
  [build]
@@ -9,7 +9,7 @@ title: "The gear shed: built-in plugins"
9
9
  - **`ssqch/sass`:** `sassPlugin({ files: { "src/scss": "/css" } })` compiles Sass with [grass](https://github.com/connorskees/grass), built into the package as WebAssembly, so there's no Dart Sass to install. grass writes a few things differently from Dart Sass; a site that needs Dart Sass's exact output can run the `sass` package from `beforeBuild` instead (`{ run: "sass src/scss:dist/css" }`). A folder compiles every `.scss` and `.sass` file in it that doesn't start with `_`, keeping their paths; a file (`{ "src/app.scss": "/app.css" }`) compiles to the path given. The CSS is written at the start of each build, so `ssqch/assets` can fingerprint it and the link checker sees it, and `ssqch serve` rebuilds when a source changes. Options: `files` (required), `style` (`"compressed"` for production builds and `"expanded"` otherwise, by default), `loadPaths`, and `charset`: `false` leaves out what grass (as Dart Sass) starts CSS with when it has a character outside ASCII, a byte order mark when compressed and `@charset "UTF-8";` when expanded, so the file starts as a Jekyll site's did (jekyll-sass-converter takes the mark out). An import that names its extension (`@import "primer-core/index.scss"`) is found in the load paths too, as Dart Sass finds it, and a stylesheet that starts with Jekyll's front matter (`---` lines) compiles, the lines left blank. (Yes, that makes it Sassquatch.)
10
10
  - **`ssqch/sitemap`:** `sitemapPlugin({ url: "https://example.com" })` writes `sitemap.xml` listing every HTML page with its absolute URL and its date as `<lastmod>`. Left out: pages with `sitemap: false` in their data, 404 pages, pages 2 and on of a list split into pages (pagination with a `size` over 1; a page per tag is still listed), and URLs matching `exclude` globs (`["/tags/**"]`). Options: `url` (the site's address, required: a sitemap lists whole addresses, and search engines read only those, so a build without one stops and asks for it; jekyll-sitemap wrote paths from the root instead), `exclude`, `lastmod` (default `true`), `output` (default `"/sitemap.xml"`), `paginated` (list every page of a split list) and `alternates`, a page's translations as hreflang links: a function that gets each page (`url`, `inputPath`, `outputPath`, `date`, `data`) and returns `[{ hreflang: "fr", href: "/fr/guide/" }]`, the page itself included (`x-default` too, if you have one), or nothing. An `href` from the site's root gets the address and `base` in front, and the sitemap gets the `xhtml` namespace only when a page has some: `alternates: (page) => page.data.translations?.map((t) => ({ hreflang: t.lang, href: t.url }))`. It's called at once, so work the translations out in data or a collection first.
11
11
  - **`ssqch/elements`:** `elementsPlugin()` is interactivity without a framework. A custom element is a tag with a hyphen (`<like-button>`) and a script that defines it. Put `like-button.js` in `src/elements/` (and `like-button.css` next to it, if it has styles), write the tag in any page or layout, and that page gets the script and the stylesheet; pages without the tag get neither, so a page with no elements loads no JavaScript. What's inside the tag is the fallback a visitor sees before the script runs, or without JavaScript: `<like-button><a href="/likes/">3 likes</a></like-button>`. `ssqch:load` on a tag says when its script loads: `eager` (with the page, the default), `idle` (when the browser has a quiet moment) or `visible` (when the element scrolls into view), which is the "islands" idea without a framework to run it. The whole folder is copied to `/elements/`, so elements can import shared modules (a file without a hyphen in its name isn't an element). Options: `dir` (default: `elements` in the input folder, which has to exist), `url` (default `/elements/`) and `load` (`eager`, `idle` or `visible`: the default for tags without `ssqch:load`; any other value is an error that says so). The [`islands` example](https://github.com/reverentgeek/sitesquatch/tree/main/examples/sites/islands/) has three small elements.
12
- - **`ssqch/llms`:** `llmsPlugin({ url: "https://example.com", description: "Hikes and the gear for them." })` writes the site for language models, as [llmstxt.org](https://llmstxt.org) describes it: `/llms.txt` lists the site's pages (each one's title, a link to its Markdown and its `description`), under a `##` heading for each of the site's top folders (`/guide/...` is "Guide"; the top's pages are "Pages"), and each of those pages gets a Markdown copy beside its URL as llms.txt's v2 proposal has it (`/guide/intro.html` is `/guide/intro.md`; a URL without a file name gets `index.md`, so `/guide/intro/` is `/guide/intro/index.md` and `/` is `/index.md`). A copy is what the page says: a Markdown page's copy is its Markdown after its template engine ran (`{{ site.title }}` is the title, a `{% for %}` is the list it wrote), without its front matter, its `<script>` and `<style>` blocks, or pictures with `alt=""` (decoration), and with its title on top when only the front matter had it, and its pictures (and links to them) at the addresses the page shows, so a picture `imagePlugin()` published under another name links the published file; any other page's copy (a home page in Jinja) is its `<article>`, or else its `<main>` or `<body>`, as Markdown (headings, paragraphs, lists, links, code, quotes, tables and described pictures; navigation, forms, scripts and anything `hidden` are left out). Links in a copy are whole addresses (with `url`), and a link to a page that has a copy goes to the copy, a relative `.md` link to another page's file included (`[Install](install.md)`). `/llms-full.txt` has the heading, then every copy in one file, each after a `url:` line. A page whose front matter says `llms: false` is left out. The plugin adds one template function, `markdownCopy(page.url)`: the URL of the page's copy, or `""` when it has none, for the layout to link it (a build adds no markup of its own): `{% if markdownCopy(page.url) %}<link rel="alternate" type="text/markdown" href="{{ markdownCopy(page.url) }}">{% endif %}`. Options: `title` (the `#` heading; default `site.title` in the site's data, or the home page's title), `description` (a `>` line under it), `details` (Markdown after that, in both files: what a model should know first), `url` (the site's address, for whole links; without it, links are from the site's root), `exclude` (URL globs, like `["/drafts/**"]`; 404 pages are always left out), `copies` (default `true`; `false` links `llms.txt` to the pages instead), `full` (default `true`), `section` (a function that gets each page and returns its section's name; sections come in the order of their first page) and `sort` (a function that compares two pages, as `Array.prototype.sort` takes it; default by URL). vitepress-plugin-llms does this for VitePress sites.
12
+ - **`ssqch/llms`:** `llmsPlugin({ url: "https://example.com", description: "Hikes and the gear for them." })` writes the site for language models, as [llmstxt.org](https://llmstxt.org) describes it: `/llms.txt` lists the site's pages (each one's title, a link to its Markdown and its `description`), under a `##` heading for each of the site's top folders (`/guide/...` is "Guide"; the top's pages are "Pages"), and each of those pages gets a Markdown copy beside its URL as llms.txt's v2 proposal has it (`/guide/intro.html` is `/guide/intro.md`; a URL without a file name gets `index.md`, so `/guide/intro/` is `/guide/intro/index.md` and `/` is `/index.md`). A copy is what the page says: a Markdown page's copy is its Markdown after its template engine ran (`{{ site.title }}` is the title, a `{% for %}` is the list it wrote), without its front matter, its `<script>` and `<style>` blocks, or pictures with `alt=""` (decoration), and with its title on top when only the front matter had it, and its pictures (and links to them) at the addresses the page shows, so a picture `imagePlugin()` published under another name links the published file; any other page's copy (a home page in Jinja) is its `<article>`, or else its `<main>` or `<body>`, as Markdown (headings, paragraphs, lists, links, code, quotes, tables and described pictures; navigation, forms, scripts and anything `hidden` are left out). Links in a copy are whole addresses (with `url`), and a link to a page that has a copy goes to the copy, a relative `.md` link to another page's file included (`[Install](install.md)`). `/llms-full.txt` has the heading, then every copy in one file, each after a `url:` line. A page whose front matter says `llms: false` is left out. The plugin adds one template function, `markdownCopy(page.url)`: the URL of the page's copy, or `""` when it has none, for the layout to link it (a build adds no markup of its own): `{% if markdownCopy(page.url) %}<link rel="alternate" type="text/markdown" href="{{ markdownCopy(page.url) }}">{% endif %}`. Options: `title` (the `#` heading; default `site.title` in the site's data, or the home page's title), `description` (a `>` line under it), `details` (Markdown after that, in both files: what a model should know first), `url` (the site's address, for whole links; without it, links are from the site's root), `exclude` (URL globs, like `["/drafts/**"]`; 404 pages are always left out), `copies` (default `true`; `false` links `llms.txt` to the pages instead), `full` (default `true`), `section` (a function that gets each page and returns its section's name; sections come in the order of their first page, with "Optional" last) and `sort` (a function that compares two pages, as `Array.prototype.sort` takes it; default by URL). `paths` (parts of the site with an `llms.txt` of their own, like `["/docs/"]`), `budget` (the most characters an `llms.txt` has before pages move into "Optional"; default 50,000) and the template function `llmsTxt(page.url)`, in [A big site's llms.txt](robots.md#a-big-sites-llmstxt). vitepress-plugin-llms does this for VitePress sites.
13
13
  - **`ssqch/robots`:** `robotsPlugin({ ai: { training: false } })` writes `/robots.txt`: everyone's group (`disallow`, URL paths no crawler should read), a group with `Disallow: /` for each kind of AI crawler that's turned off (`ai`: `training`, `search` and `fetch`, each `true` by default), and the `Sitemap:` line with `sitemapPlugin()`'s address (`sitemap: false` leaves it out, a string is another address). [Posted signs](robots.md) lists the crawlers of each kind and the links a layout writes for agents.
14
14
  - **`ssqch/vue`:** `vuePlugin(options)` renders the Vue components a site's pages use, at build time, with the site's own Vite, `@vitejs/plugin-vue` and Vue (`npm install -D vite @vitejs/plugin-vue vue`; ssqch doesn't bundle them, and says so if they're missing). With pnpm, which keeps a VitePress site's vite and @vitejs/plugin-vue beside `vitepress` rather than in the site's `node_modules`, they're found there. Vue renders with its production build in a production build and its development build while serving (`NODE_ENV` follows ssqch's env unless you set it). A page's `<script setup>` imports name them, as VitePress's pages do (`import ApiIndex from './ApiIndex.vue'`, then `<ApiIndex />` in the page), and `components` names the ones every page can use (`{ VueSchoolLink: ".vitepress/theme/components/VueSchoolLink.vue" }`, VitePress's `app.component()`). Each tag a page has (and no template in `_includes/components/`, which comes first) becomes the HTML Vue's server renderer writes for it: its attributes are its props, and a binding is read as Vue reads it in the page, with the page's front matter (`frontmatter`, `$frontmatter`), `params` (a dynamic route's, from its URL: `[partnerId].md` written as `/partners/vehikl.html`), `page` (VitePress's `useData().page`) and the `data` of each VitePress loader its `<script setup>` imports (`import { data } from './errors.data'`; the loader's `load()` runs once a build, through Vite, with `__dirname`), and what's between its tags is its default slot, with the components inside it rendered first (a layout component that holds the page works). With `fromVitepress()` from `ssqch/from-vitepress` (a site moving from VitePress), components that import from `"vitepress"` get a stand-in (with `current`, its page's state, for a wrapper that renders the site's theme: the VitePress trailhead's "Keeping your VitePress theme") for `useData()` (the page's front matter, title and route, and the config's `themeConfig`), `useRoute()`, `withBase()` and `inBrowser`, and `@theme/` is the theme's folder. `hydrate` runs each component in the browser too, as an island: `"load"`, `"idle"` or `"visible"` (when it scrolls into view) wraps its HTML in `<ssqch-vue>` and adds one script to the pages that have one, `/assets/vue/islands.js`, which loads only the components on the page; default `false`, HTML only. `css` is where the components' styles go (default `"/assets/vue.css"`, for your layout to link; `false` for none), and the fonts and images they point at go where Vite put them (`/assets/inter-x.woff2`, after `base`); the site's `public/` isn't copied (that's `passthrough`'s), `root` is Vite's root (default the input folder, so scope ids are VitePress's), and `vite` adds Vite options (`resolve.alias`, `plugins`). The components build in `.cache/ssqch/vue/` before each build; while serving, an edit that's only to pages keeps the last one. A VitePress loader named like ssqch's data (`api/api.data.ts`) goes in `ignore`, which `ssqch scout` drafts.
15
15
  - **`ssqch/redirects`:** `redirectsPlugin(options)` keeps old URLs working. A page lists its old URLs in front matter (`aliases: [/2019/pine-cones/, /pine.html]`), and the `redirects` option adds others (`{ "/feed/": "/feed.xml" }`, where the new URL can be on another site). Each old URL gets a small page that sends visitors on (a meta refresh, a canonical link and `noindex`), and with `netlify: true`, a `/old /new 301` line is appended to `_redirects`, after any lines your own `_redirects` template writes, and [`linksPlugin()`](#) honors those lines. An old URL that's another page's URL is an error. For redirect pages in your own markup, add `_includes/hooks/redirect.*` (any engine): it gets `redirect.from` and `redirect.to` with the site's global data. Options: `pages` (default `true`), `netlify` (default `false`) and `redirects`.
@@ -55,6 +55,16 @@ Not written yet, because nothing reads them yet: the IETF draft's shared group f
55
55
 
56
56
  `llmsPlugin()` writes `/llms.txt`, a Markdown copy of each page and `/llms-full.txt` ([Built-in plugins](plugins.md)). A copy is what the page says (a Markdown page's text after its template ran, a template page's `<article>` or `<main>` as Markdown), at the address llms.txt's proposal gives it: `/guide/intro/` has `/guide/intro/index.md`.
57
57
 
58
+ ### A big site's llms.txt
59
+
60
+ `llms.txt` is the map an agent reads before it decides where to go, so it has to stay short enough to read whole (llms.txt's proposal: "small enough to fit in context"). Three options keep it that way:
61
+
62
+ - `paths`: parts of the site that get an `llms.txt` of their own, which covers the pages under it (llms.txt's v2 has agents use the most specific one). `paths: ["/docs/"]` writes `/docs/llms.txt` with the pages under `/docs/`, and `/llms.txt` lists that file once, where its first page would be, in place of them. An item is a URL path that starts and ends with `/`, or `{ path: "/docs/api/", title: "Trail API", description: "Every endpoint.", details: "..." }` for the file's `#` heading (default: the site's title and the folder's name, `Trail Guide: Docs`), its `>` line and its Markdown after that; the title and description are also its line in the file above it. Inside a file, a page's section is its folder under the path (`/docs/guide/start/` is "Guide" in `/docs/llms.txt`), unless `section` names one. A path can be under another (`/docs/` and `/docs/api/`), and a path with no listed pages writes no file and warns. `llms-full.txt` stays one file with every page.
63
+ - A section named "Optional" always comes last: llms.txt's convention for links an agent can skip when its context is short. `section: (page) => (page.date < new Date("2020-01-01") ? "Optional" : undefined)` puts old posts there and leaves the rest in their folder's section.
64
+ - `budget`: the most characters an `llms.txt` may have, 50,000 by default (where `ssqch patrol` warns, `llms-long`). Past it, pages from the end of the file move into "Optional" without their descriptions until it fits. A file that wouldn't fit even with every page moved stays as it is, and the build warns with what to do: give a part of the site its own file with `paths`, or raise `budget`. `budget: false` turns it off.
65
+
66
+ `ssqch patrol` reads `/llms.txt` and every `llms.txt` it links, with the same checks, and counts the pages they all list.
67
+
58
68
  ## The links in the layout
59
69
 
60
70
  llms.txt's proposal (v2) has agents find both from any page by two links, which a layout writes in its `<head>` (a build adds no markup of its own):
@@ -62,10 +72,13 @@ llms.txt's proposal (v2) has agents find both from any page by two links, which
62
72
  ```html
63
73
  <link rel="canonical" href="{{ site.url }}{{ page.url }}">
64
74
  {% if markdownCopy(page.url) %}<link rel="alternate" type="text/markdown" href="{{ markdownCopy(page.url) }}">{% endif %}
65
- <link rel="describedby" href="/llms.txt">
75
+ <link rel="describedby" href="{{ llmsTxt(page.url) }}">
66
76
  ```
67
77
 
68
- `markdownCopy(page.url)` comes with `llmsPlugin()`: the copy's URL, or `""` for a page without one. The starters' layouts have all three.
78
+ `markdownCopy(page.url)` comes with `llmsPlugin()`: the copy's URL, or `""` for a page without one. The starters' layouts have all three, with `/llms.txt` written out.
79
+ `llmsTxt(page.url)` comes with it too: the `llms.txt` that covers the page, `/docs/llms.txt` for a page under `/docs/` when `paths` has it and `/llms.txt` for the rest.
80
+
81
+ A copy is a Markdown file, with no `<head>` to point from: llms.txt's v2 points from it with an HTTP `Link` header, which a host's `_headers` file (Netlify, Cloudflare) sends for a folder, like `/docs/*` with `Link: </docs/llms.txt>; rel="describedby"`. ssqch adds no line to the copy itself, so a copy says what its page says and nothing more.
69
82
 
70
83
  ## When an agent asks for Markdown
71
84
 
@@ -644,12 +644,18 @@ declare module "ssqch/llms" {
644
644
  copies?: boolean;
645
645
  /** `/llms-full.txt`, every listed page's Markdown in one file, each after a `url:` line. Default `true`. */
646
646
  full?: boolean;
647
- /** A page's section in `llms.txt`: a function that gets each page and returns a name. Default: its first folder (`/guide/...` is "Guide"; the top's pages are "Pages"). Sections are in the order their first page comes. */
647
+ /** A page's section in `llms.txt`: a function that gets each page and returns a name. Default: its first folder (`/guide/...` is "Guide"; the top's pages are "Pages"). Sections are in the order their first page comes, with "Optional" (links an agent can skip) last. Inside a file of `paths`, the default is the page's folder under that path. */
648
648
  section?: (page: LlmsPage) => string | undefined;
649
649
  /** The pages' order, as `Array.prototype.sort` takes it: `(a, b) => a.data.order - b.data.order`. Default: by URL. */
650
650
  sort?: (a: LlmsPage, b: LlmsPage) => number;
651
- }
652
- /** Writes `/llms.txt` (the site's pages, with links to their Markdown), a Markdown copy of each, and `/llms-full.txt`, for language models, as llmstxt.org describes; and adds the template function `markdownCopy(page.url)`, the URL of a page's copy (or `""` when it has none), for a layout's `<link rel="alternate" type="text/markdown">`. */
651
+ /** Parts of the site with an `llms.txt` of their own, as llms.txt's v2 has them: `["/docs/"]` writes `/docs/llms.txt` with the pages under `/docs/`, and `/llms.txt` lists that file in their place. An item is a URL path that starts and ends with `/`, or an object with its `path` and the file's `title` (its heading; default the site's title and the folder's name), `description` and `details`. A page belongs to the deepest path it's under. A path with no listed pages writes no file, and the build warns. */
652
+ paths?: Array<string | LlmsPath>;
653
+ /** The most characters an `llms.txt` has: past it, pages from the end move into "Optional" without their descriptions until it fits. A file that wouldn't fit even so stays as it is, and the build warns. Default `50000` (where `ssqch patrol` warns); `false` for no limit. */
654
+ budget?: number | false;
655
+ }
656
+ /** A part of the site with an `llms.txt` of its own, in `paths`. */
657
+ export interface LlmsPath { path: string; title?: string; description?: string; details?: string }
658
+ /** Writes `/llms.txt` (the site's pages, with links to their Markdown), a Markdown copy of each, and `/llms-full.txt`, for language models, as llmstxt.org describes; and adds the template functions `markdownCopy(page.url)`, the URL of a page's copy (or `""` when it has none), for a layout's `<link rel="alternate" type="text/markdown">`, and `llmsTxt(page.url)`, the `llms.txt` that covers the page, for its `<link rel="describedby">`. */
653
659
  export function llmsPlugin(options?: LlmsPluginOptions): BuiltinPlugin;
654
660
  }
655
661
 
@@ -155,12 +155,15 @@ function patrol(dir) {
155
155
  const llms = has.has("llms.txt") ? read("llms.txt") : void 0;
156
156
  const listed = /* @__PURE__ */ new Set();
157
157
  if (llms === void 0) say("no-llms", "no llms.txt: an agent gets no map of the site (llmsPlugin() writes one, with a Markdown copy of each page)");
158
- else {
159
- if (!/^?# \S/.test(llms)) say("llms-shape", "llms.txt doesn't start with a `# ` heading (the site's name), which llmstxt.org's format requires", "llms.txt");
160
- if (llms.length > LLMS_LIMIT) say("llms-long", `llms.txt is ${count(llms.length)} characters, over ${count(LLMS_LIMIT)}: more than an agent reads before deciding where to go (link section files from it instead)`, "llms.txt");
161
- for (const href of linksIn(llms)) {
158
+ const maps = llms === void 0 ? [] : ["llms.txt"];
159
+ for (let i = 0; i < maps.length; i++) {
160
+ const name = maps[i];
161
+ const text = i === 0 ? llms : read(name);
162
+ if (!/^\uFEFF?# \S/.test(text)) say("llms-shape", "llms.txt doesn't start with a `# ` heading (the site's name), which llmstxt.org's format requires", name);
163
+ if (text.length > LLMS_LIMIT) say("llms-long", `llms.txt is ${count(text.length)} characters, over ${count(LLMS_LIMIT)}: more than an agent reads before deciding where to go (link section files from it instead)`, name);
164
+ for (const href of linksIn(text)) {
162
165
  if (relative(href)) {
163
- say("relative-links", "links that aren't whole addresses (or from the site's root): they break once the text is read on its own", "llms.txt");
166
+ say("relative-links", "links that aren't whole addresses (or from the site's root): they break once the text is read on its own", name);
164
167
  continue;
165
168
  }
166
169
  const file = fileOf(href);
@@ -169,6 +172,7 @@ function patrol(dir) {
169
172
  else {
170
173
  if (!/\.(md|markdown|txt)$/i.test(file)) say("llms-html", "llms.txt links to pages, not their Markdown: an agent reads Markdown without the page's chrome", href);
171
174
  listed.add(file);
175
+ if (/(^|\/)llms\.txt$/.test(file) && !maps.includes(file)) maps.push(file);
172
176
  }
173
177
  }
174
178
  }
package/lib/cli.js CHANGED
@@ -558,7 +558,7 @@ async function runDiff(dirs, options) {
558
558
  async function runPatrol(dir, shown, format) {
559
559
  try {
560
560
  checkFormat(format);
561
- const { patrol, patrolJson, patrolText } = await import("./chunk-2FKYWYAB.js");
561
+ const { patrol, patrolJson, patrolText } = await import("./chunk-GM5SHCTU.js");
562
562
  const report = patrol(dir);
563
563
  process.stdout.write(format === "json" ? JSON.stringify(patrolJson(report, shown), null, 2) + "\n" : patrolText(report, shown));
564
564
  return report.findings.length ? 1 : 0;
package/lib/llms.js CHANGED
@@ -26,7 +26,9 @@ import "./chunk-FJML7INB.js";
26
26
  import "./chunk-4AZFCSGJ.js";
27
27
  import "./chunk-CMUWRPSQ.js";
28
28
  import {
29
- bail
29
+ bail,
30
+ warn,
31
+ warning
30
32
  } from "./chunk-YNJNMQWU.js";
31
33
  import {
32
34
  compareText
@@ -386,13 +388,19 @@ function copyUrl(url) {
386
388
  if (url.endsWith("/")) return `${url}index.md`;
387
389
  return url.replace(/\.html?$/, "") + ".md";
388
390
  }
391
+ function folderName(folder) {
392
+ const name = folder.replace(/[-_]+/g, " ");
393
+ return name.charAt(0).toUpperCase() + name.slice(1);
394
+ }
395
+ var BUDGET = 5e4;
389
396
  function sectionOf(url) {
390
397
  const first = url.split("/").filter(Boolean);
391
398
  if (first.length < 2 && !url.endsWith("/")) return "Pages";
392
399
  if (first.length === 0) return "Pages";
393
- const name = first[0].replace(/[-_]+/g, " ");
394
- return name.charAt(0).toUpperCase() + name.slice(1);
400
+ return folderName(first[0]);
395
401
  }
402
+ var grouped = (n) => String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ",");
403
+ var shown = (v) => typeof v === "string" ? debugString(v) : JSON.stringify(v) ?? typeof v;
396
404
  var oneLine = (s) => typeof s === "string" ? s.replace(/\s+/g, " ").trim() : "";
397
405
  var isPageUrl = (url) => url.endsWith("/") || url.endsWith(".html");
398
406
  var keyOf = (url) => url.replace(/\/index\.html$/, "/");
@@ -475,9 +483,41 @@ function publishedImages(markdown, outputPath, base, output) {
475
483
  return became.has(href) ? `${open}"${became.get(href)}"` : whole;
476
484
  });
477
485
  }
486
+ function listing(head, items, budget, file) {
487
+ const render = (list2) => {
488
+ const sections = /* @__PURE__ */ new Map();
489
+ for (const item of list2) sections.set(item.section, [...sections.get(item.section) ?? [], item.line]);
490
+ const optional = sections.get("Optional");
491
+ if (optional) {
492
+ sections.delete("Optional");
493
+ sections.set("Optional", optional);
494
+ }
495
+ let txt2 = head;
496
+ for (const [name, lines] of sections) txt2 += `
497
+ ## ${name}
498
+
499
+ ${lines.join("")}`;
500
+ return txt2;
501
+ };
502
+ let txt = render(items);
503
+ if (txt.length <= budget) return txt;
504
+ const moved = items.map((item) => ({ ...item }));
505
+ let over = txt.length - budget + (items.some((i) => i.section === "Optional") ? 0 : "\n## Optional\n\n".length);
506
+ const most = items.reduce((n, i) => n + (i.section === "Optional" ? 0 : i.line.length - i.short.length), 0);
507
+ if (most < over) {
508
+ warn(warning("build", `llmsPlugin: ${file} is ${grouped(txt.length)} characters, over the budget of ${grouped(budget)} even without its pages' notes: give a part of the site an llms.txt of its own (paths: ["/docs/"]), or raise budget`));
509
+ return txt;
510
+ }
511
+ for (let i = moved.length - 1; i >= 0 && over > 0; i--) {
512
+ if (moved[i].section === "Optional") continue;
513
+ over -= moved[i].line.length - moved[i].short.length;
514
+ moved[i] = { section: "Optional", line: moved[i].short, short: moved[i].short };
515
+ }
516
+ return render(moved);
517
+ }
478
518
  BUILTINS.set("ssqch/llms", (config, given) => {
479
519
  if (given !== null && typeof given === "object" && Object.hasOwn(given, "hint")) bail('`hint` is gone: a build adds no markup of its own. Link the copy from your layout\'s <head> instead: <link rel="alternate" type="text/markdown" href="{{ markdownCopy(page.url) }}">');
480
- const o = new Options(given, ["title", "description", "details", "url", "exclude", "copies", "full", "section", "sort"]);
520
+ const o = new Options(given, ["title", "description", "details", "url", "exclude", "copies", "full", "section", "sort", "paths", "budget"]);
481
521
  const url = o.string("url");
482
522
  if (url !== void 0 && url !== "" && !/^https?:\/\//.test(url)) bail(`url must be the site's address, like "https://example.com" (got ${debugString(url)})`);
483
523
  const exclude = o.list("exclude") ?? [];
@@ -493,6 +533,27 @@ BUILTINS.set("ssqch/llms", (config, given) => {
493
533
  const description = o.string("description");
494
534
  const details = o.string("details");
495
535
  const address = (url ?? "").replace(/\/+$/, "");
536
+ const givenBudget = o.get("budget");
537
+ if (givenBudget !== void 0 && givenBudget !== false && !(typeof givenBudget === "number" && Number.isInteger(givenBudget) && givenBudget > 0)) bail(`budget must be a number of characters, like 50000, or false for no limit (got ${shown(givenBudget)})`);
538
+ const budget = givenBudget === false ? Infinity : givenBudget ?? BUDGET;
539
+ const paths = [];
540
+ const givenPaths = o.get("paths");
541
+ if (givenPaths !== void 0 && !Array.isArray(givenPaths)) bail(`paths must be a list of URL paths that get an llms.txt of their own, like ["/docs/"] (got ${shown(givenPaths)})`);
542
+ for (const item of givenPaths ?? []) {
543
+ const one = typeof item === "string" ? { path: item } : item;
544
+ if (one === null || typeof one !== "object" || typeof one.path !== "string") bail(`paths: each one is a URL path like "/docs/", or { path: "/docs/", title, description, details } (got ${shown(item)})`);
545
+ const { path: at, ...rest } = one;
546
+ for (const [key, value] of Object.entries(rest)) {
547
+ if (!["title", "description", "details"].includes(key)) bail(`paths: ${shown(at)} has \`${key}\`, which isn't an option; it takes title, description and details`);
548
+ if (typeof value !== "string") bail(`paths: ${shown(at)}'s ${key} must be a string (got ${shown(value)})`);
549
+ }
550
+ const folder = at;
551
+ if (!folder.startsWith("/") || !folder.endsWith("/") || folder === "/") bail(`paths: ${shown(folder)} must be a folder's URL path, starting and ending with /, like "/docs/" (/llms.txt is the site's)`);
552
+ if (paths.some((f) => f.path === folder)) bail(`paths: ${shown(folder)} is listed twice`);
553
+ paths.push({ path: folder, ...rest });
554
+ }
555
+ paths.sort((a, b) => b.path.length - a.path.length || compareText(a.path, b.path));
556
+ const coveringFile = (pageUrl) => paths.find((f) => pageUrl.startsWith(f.path))?.path ?? "/";
496
557
  const listedUrl = (pageUrl) => isPageUrl(pageUrl) && !excluded(pageUrl) && pageUrl !== "/404.html" && pageUrl !== "/404/";
497
558
  const listed = (p) => p.outputPath.endsWith(".html") && listedUrl(p.url) && p.data.llms !== false;
498
559
  const sources = /* @__PURE__ */ new Map();
@@ -517,6 +578,7 @@ BUILTINS.set("ssqch/llms", (config, given) => {
517
578
  if (page && page.url === pageUrl && data?.llms === false) return "";
518
579
  return copyUrl(pageUrl);
519
580
  });
581
+ config.filters.set("llmsTxt", (value) => `${coveringFile(value === void 0 || value === null ? "/" : String(value))}llms.txt`);
520
582
  config.generators.push(["ssqch/llms", (config2, pages) => {
521
583
  const base = config2.base.replace(/\/$/, "");
522
584
  const output = path.resolve(config2.root, config2.output);
@@ -561,7 +623,7 @@ BUILTINS.set("ssqch/llms", (config, given) => {
561
623
  ${body}`;
562
624
  body = publishedImages(body, p.outputPath, base, output);
563
625
  body = rewriteLinks(body, (href) => target(href, p));
564
- let name = sectionOf(p.url);
626
+ let name;
565
627
  if (typeof section === "function") {
566
628
  let out;
567
629
  try {
@@ -586,18 +648,47 @@ ${body}`;
586
648
  if (details?.trim()) head += `
587
649
  ${details.trim()}
588
650
  `;
589
- let txt = head;
590
- const sections = /* @__PURE__ */ new Map();
591
- for (const e of entries) sections.set(e.section, [...sections.get(e.section) ?? [], e]);
592
- for (const [name, list2] of sections) {
593
- txt += `
594
- ## ${name}
595
-
651
+ const written = paths.filter((f) => {
652
+ if (entries.some((e) => coveringFile(e.url) === f.path)) return true;
653
+ warn(warning("build", `llmsPlugin: no listed pages under ${f.path}, so ${f.path}llms.txt isn't written (paths in ssqch.config.js)`));
654
+ return false;
655
+ });
656
+ const fileFor = (pageUrl) => written.find((f) => pageUrl.startsWith(f.path))?.path ?? "/";
657
+ const parentOf = (inner) => written.find((f) => f.path !== inner && inner.startsWith(f.path))?.path ?? "/";
658
+ const nameOf = (f) => f.title ?? folderName(f.path.split("/").filter(Boolean).at(-1) ?? "");
659
+ const link = (title2, href, note) => `- [${title2.replace(/([[\]])/g, "\\$1")}](${href})${note ? `: ${note}` : ""}
596
660
  `;
597
- for (const e of list2) txt += `- [${e.title.replace(/([[\]])/g, "\\$1")}](${whole(copies ? e.copy : e.url)})${e.description ? `: ${e.description}` : ""}
661
+ for (const file of [{ path: "/" }, ...written]) {
662
+ const items = [];
663
+ const listedFiles = /* @__PURE__ */ new Set();
664
+ for (const e of entries) {
665
+ let at = fileFor(e.url);
666
+ if (at === file.path) {
667
+ const name2 = e.section ?? sectionOf(`/${e.url.slice(file.path.length)}`);
668
+ items.push({ section: name2, line: link(e.title, whole(copies ? e.copy : e.url), e.description), short: link(e.title, whole(copies ? e.copy : e.url), "") });
669
+ continue;
670
+ }
671
+ if (file.path !== "/" && !at.startsWith(file.path)) continue;
672
+ while (at !== "/" && parentOf(at) !== file.path) at = parentOf(at);
673
+ if (at === "/" || listedFiles.has(at)) continue;
674
+ listedFiles.add(at);
675
+ const inner = written.find((f) => f.path === at);
676
+ const name = e.section ?? sectionOf(`/${e.url.slice(file.path.length)}`);
677
+ items.push({ section: name, line: link(nameOf(inner), whole(`${inner.path}llms.txt`), oneLine(inner.description)), short: link(nameOf(inner), whole(`${inner.path}llms.txt`), "") });
678
+ }
679
+ let top = head;
680
+ if (file.path !== "/") {
681
+ top = `# ${file.title ?? `${siteTitle}: ${nameOf(file)}`}
682
+ `;
683
+ if (file.description) top += `
684
+ > ${oneLine(file.description)}
598
685
  `;
686
+ if (file.details?.trim()) top += `
687
+ ${file.details.trim()}
688
+ `;
689
+ }
690
+ files.push({ path: `${file.path}llms.txt`, contents: listing(top, items, budget, `${file.path}llms.txt`) });
599
691
  }
600
- files.push({ path: "/llms.txt", contents: txt });
601
692
  if (copies) for (const e of entries) files.push({ path: e.copy, contents: e.markdown });
602
693
  if (full) files.push({ path: "/llms-full.txt", contents: `${head}
603
694
  ${entries.map((e) => `---
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "ssqch",
3
- "version": "0.13.0",
3
+ "version": "0.13.1",
4
4
  "description": "SiteSquatch (ssqch): a fast static site generator, configured with JavaScript",
5
5
  "keywords": [
6
6
  "static site generator",
package/types/llms.d.ts CHANGED
@@ -18,10 +18,16 @@ export interface LlmsPluginOptions {
18
18
  copies?: boolean;
19
19
  /** `/llms-full.txt`, every listed page's Markdown in one file, each after a `url:` line. Default `true`. */
20
20
  full?: boolean;
21
- /** A page's section in `llms.txt`: a function that gets each page and returns a name. Default: its first folder (`/guide/...` is "Guide"; the top's pages are "Pages"). Sections are in the order their first page comes. */
21
+ /** A page's section in `llms.txt`: a function that gets each page and returns a name. Default: its first folder (`/guide/...` is "Guide"; the top's pages are "Pages"). Sections are in the order their first page comes, with "Optional" (links an agent can skip) last. Inside a file of `paths`, the default is the page's folder under that path. */
22
22
  section?: (page: LlmsPage) => string | undefined;
23
23
  /** The pages' order, as `Array.prototype.sort` takes it: `(a, b) => a.data.order - b.data.order`. Default: by URL. */
24
24
  sort?: (a: LlmsPage, b: LlmsPage) => number;
25
+ /** Parts of the site with an `llms.txt` of their own, as llms.txt's v2 has them: `["/docs/"]` writes `/docs/llms.txt` with the pages under `/docs/`, and `/llms.txt` lists that file in their place. An item is a URL path that starts and ends with `/`, or an object with its `path` and the file's `title` (its heading; default the site's title and the folder's name), `description` and `details`. A page belongs to the deepest path it's under. A path with no listed pages writes no file, and the build warns. */
26
+ paths?: Array<string | LlmsPath>;
27
+ /** The most characters an `llms.txt` has: past it, pages from the end move into "Optional" without their descriptions until it fits. A file that wouldn't fit even so stays as it is, and the build warns. Default `50000` (where `ssqch patrol` warns); `false` for no limit. */
28
+ budget?: number | false;
25
29
  }
26
- /** Writes `/llms.txt` (the site's pages, with links to their Markdown), a Markdown copy of each, and `/llms-full.txt`, for language models, as llmstxt.org describes; and adds the template function `markdownCopy(page.url)`, the URL of a page's copy (or `""` when it has none), for a layout's `<link rel="alternate" type="text/markdown">`. */
30
+ /** A part of the site with an `llms.txt` of its own, in `paths`. */
31
+ export interface LlmsPath { path: string; title?: string; description?: string; details?: string }
32
+ /** Writes `/llms.txt` (the site's pages, with links to their Markdown), a Markdown copy of each, and `/llms-full.txt`, for language models, as llmstxt.org describes; and adds the template functions `markdownCopy(page.url)`, the URL of a page's copy (or `""` when it has none), for a layout's `<link rel="alternate" type="text/markdown">`, and `llmsTxt(page.url)`, the `llms.txt` that covers the page, for its `<link rel="describedby">`. */
27
33
  export function llmsPlugin(options?: LlmsPluginOptions): BuiltinPlugin;