@stackql/docusaurus-plugin-aeo 0.4.2 → 0.5.1

package/CHANGELOG.md

@@ -1,5 +1,60 @@
 # Changelog
 
+## 0.5.1
+
+Localized sites (Docusaurus i18n) and sites with a non-root `baseUrl`. Raised against stackql.io when it gained a `de` locale.
+
+### Fixed
+
+- **Companions and alternate links on a site whose `baseUrl` is not `/`.** Docusaurus permalinks carry `siteConfig.baseUrl` but the build output tree is rooted at it, and in a localized build the plugin runs once per locale with `baseUrl: '/<locale>/'` and `outDir: build/<locale>`. Feature 1 joined `outDir` with the full permalink, so a `de` site wrote its companions to `build/de/de/foo.md`, and feature 1b looked for the page's HTML at the same wrong path and injected nothing. Both now strip the base URL from the route before touching the file system (`outputRoute` in `src/companionPath.js`); the companion URLs, the `Source:` lines, `llms.txt` and the `_headers` routes keep the full route as before. A localized build therefore emits `/<locale>/foo.md`, `/<locale>/llms.txt` and `/<locale>/llms-full.txt` from the translated sources with no configuration.
+- **Route patterns in a localized build or under a non-root `baseUrl`.** `companions.exclude`, `llmsTxt.exclude` and `llmsTxt.include` were matched against the full permalink, which carries `siteConfig.baseUrl`: `exclude: ['/docs/internal/**']` skipped the page in the default locale and kept `/de/docs/internal/...` in the `de` build, and the default `llmsTxt.exclude` list had the same gap. A pattern now matches a route with or without the base URL in front of it (`src/routeGlob.js`), so one configuration covers every locale. A pattern that spells the base out (`/de/blog/**`, or `/docs/guide/**` on a site built at `/docs/`) matches as before and can single out one locale. The Ask AI button follows, since it shows on the routes that have a companion.
+- **`aiRoutes.validate` under a base URL.** The `/ai/*` convention was looked for at the start of the permalink, so pages at `/de/ai/...` or `/docs/ai/...` were never validated. It is now relative to `siteConfig.baseUrl`.
+- **`_headers` for a non-default locale** (`companions.linkHeader`). Each build wrote its rules to `<outDir>/_headers`, which for a non-default locale is `build/<locale>/_headers`. Netlify and Cloudflare Pages document the file at the root of the published directory, so rules in a sub-directory cannot be relied on. In a build of several locales, the rules of every locale now go in the one `_headers` at the build root (Docusaurus builds the default locale first, so the file is there). A locale built on its own (`docusaurus build --locale de`) or given its own `baseUrl` keeps the file in its `outDir`. A rule already in the file is not appended a second time.
+
+### Changed
+
+- The Ask AI button's visible strings ("Ask AI about this page", "Ask Claude", "Ask ChatGPT", "Ask Perplexity") go through `translate()` with the ids `aeo.askAi.button` and `aeo.askAi.provider.<key>`, so `docusaurus write-translations` extracts them into `i18n/<locale>/code.json` and a localized site can translate them.
+- `{pageUrl}` in the Ask AI prompt ends with a slash on a root: the site root (`https://site/`, previously `https://site`) and a base URL such as `https://site/de/`, as the `Source:` lines already did. Other pages are unchanged (no trailing slash), and so is the default prompt, which uses `{companionUrl}`.
+- The Ask AI button's hover background is a 4% tint of the site's `--ifm-color-primary` instead of a fixed colour.
+
+### Added
+
+- `aiRoutePatterns(baseUrl)` in `@stackql/docusaurus-plugin-aeo/helpers`, and an optional `baseUrl` argument on `isAiRoute` and `buildStructuredDataAiExcludes`. `AI_ROUTE_PATTERNS` and `sitemapExclude` are unchanged and describe a site at the root with one locale; a site with a non-root `baseUrl` or several locales builds its list with `['/', '/de/'].flatMap(aiRoutePatterns)`.
+- README: hosting notes. Which hosts `companions.linkHeader` applies to (those that read `_headers`: Netlify, Cloudflare Pages, with Cloudflare's 100-rule limit), where the file is written, `.nojekyll` on GitHub Pages, and absolute URLs on preview deployments.
+- Tests for all of the above and for the rest of this release: companion files and `llms.txt` under a base URL, the plugin run once per locale, and the Ask AI button rendered in Node (`test/companions.test.js`, `test/localizedBuild.test.js`, `test/askAiButton.test.js`, `test/routeGlob.test.js`, `test/aiHelpers.test.js`).
+
+## 0.5.0
+
+Plain-markdown companions, one rule for where a companion lives, and the companion advertised from its page. Raised against stackql.io, where the Ask AI prompt on the homepage read `https://stackql.io.md` and every `.md` URL returned raw MDX (imports, MUI components, tab definitions and inline SVG path data before the first sentence), and from an independent agent-readiness audit of the built site (alternate link, `llms-full.txt` reference and size, `Source:` URLs, the `how-tos` alias).
+
+### Changed (breaking)
+
+- **`companions.format` now defaults to `'plain'`.** The companion is plain markdown rendered from the MDX source rather than the source itself. The source is parsed with the remark toolchain Docusaurus compiles it with (`remark-mdx`, `remark-gfm`, `remark-directive`, `@slorber/remark-comment`) to locate every MDX construct, and the original text is edited by source position: imports, exports and `{expressions}` are removed; presentational elements (`svg`, `video`, `iframe`, icons) are dropped; `<Tabs>`/`<TabItem>` become a bold label per tab with the tab's content under it (the label is read from the `label` attribute or from the `<Tabs values={[...]}>` expression); `<a>`, `<img>`, `<b>`, `<i>`, `<code>`, `<h1>`-`<h6>`, `<br>`, `<details>`/`<summary>` become their markdown equivalent; a self-closing component that carries a link (`to`/`href` plus `text`/`label`/`title`) becomes a markdown link; every other element is unwrapped and the indentation its author added inside it is removed. HTML comments (including `<!-- truncate -->`) and `{#custom-id}` heading ids are removed; fenced code is never touched. Nothing outside MDX syntax is re-stringified, so `snake_case` stays unescaped and tables, admonitions and code fences are byte for byte the author's. A `# Title` / `> description` block from the page metadata is prepended (the title only when the body does not already open with an H1). A source that fails to parse falls back to the v0.4 regex stripper with a build warning. `format: md` files (front matter or `siteConfig.markdown.format`) are parsed as CommonMark and their raw HTML stripped. Set `companions: { format: 'raw' }` to keep emitting the source.
+- **Default `askAi.promptTemplate` is `'Read {companionUrl} and help me understand it. ...'`.** `{companionUrl}` is the page's `.md` file by the same rule feature 1 writes it. `{pageUrl}` is unchanged (the page, no trailing slash). A custom template still using the v0.4 idiom `{pageUrl}.md` keeps working: that sequence now resolves to `{companionUrl}` instead of being computed from the pathname, which produced `https://site.md` on the homepage and `/foo.md` on sites that build `/foo/index.html`.
+- **The Ask AI button is only rendered on pages that have a companion.** Generated category index pages, React pages and blog list pages have no source markdown and so no `.md` file; the button used to point readers at a 404 there. The route list is set as plugin global data from `allContentLoaded` (Docusaurus 3 merges it with the `contentLoaded` data); if it is absent the button shows everywhere as before. With `companions.enabled: false` the button shows everywhere and the prompt uses the page URL.
+- **`llms.txt` links are absolute** (`https://site/foo.md`), as the llms.txt spec's examples are. The file is read away from the site, where a relative link has no base.
+
+### Added
+
+- **`<link rel="alternate" type="text/markdown" href="...">` in every page that has a companion** (`companions.alternateLink`, default `true`), inserted into the built HTML in `postBuild`, with the absolute companion URL from the same rule the file is written by. `companions.linkHeader` (default `false`) also writes the relation as a `Link:` header per route in a Netlify-style `_headers` file, appended to one the site already ships.
+- **`## Optional` section in `llms.txt`** linking `llms-full.txt` (`llmsTxt.linkFullTxt`, default `true`), the llmstxt.org convention for secondary resources.
+- **`llmsTxt.fullTxt` object form** `{ include: ['<plugin>@<id>', ...], maxBytes }` to limit `llms-full.txt` to chosen content instances and/or a byte budget; the file stops before the page that would cross the cap. The boolean form is unchanged (`true` = everything).
+- Plain companions carry a `Source: <page URL>` line under the title and description, and `llms-full.txt` reuses it rather than adding a second one.
+- `companions.format: 'clean'` is accepted as a synonym for `'plain'`.
+- `src/features/aiRoutes.js`: `how-tos` and `how-to` directory names map to the `howTo` payload (only `howto` / `howtos` did; stackql.io uses `how-tos`).
+
+### Fixed
+
+- Feature 1, feature 2 and the Ask AI button each computed the companion location on their own; the button's version was wrong for the site root and for `trailingSlash: true`/undefined sites. All three now use `src/companionPath.js`.
+- `llms-full.txt` no longer doubles the page title (the plain companion already opens with it) and now puts a blank line after the `Source:` line, which used to run into the first paragraph.
+- Companions are written with LF line endings regardless of the source file's.
+
+### Added
+
+- `src/companionPath.js` (`companionPath`, `companionUrl`, `normalizeRoute`) and `src/askAiPrompt.js` (`fillPrompt`), plain CommonJS shared by the Node side and the theme component.
+- Runtime dependencies on `unified`, `remark-parse`, `remark-mdx`, `remark-gfm`, `remark-directive` and `@slorber/remark-comment` at the versions `@docusaurus/mdx-loader` 3.x already ships, so npm dedupes them.
+- A test suite (`npm test`, `node --test`): the companion path and HTML path rules, the prompt template, MDX fixtures for the converter (tabs, nested containers and dedent, link recovery, inline elements, details/summary, comments and heading ids, string-literal expressions, `md` format, the parse-failure fallback, CRLF sources), `llms.txt` rendering (sections, absolute links, the Optional block, the `fullTxt` forms), the alternate-link insertion (both `trailingSlash` layouts, idempotence, `_headers`), and the `/ai/*` directory classifier.
+
 ## 0.4.2
 
 ### Fixed

package/README.md

@@ -1,6 +1,9 @@
+[![NPM Version](https://img.shields.io/npm/v/%40stackql%2Fdocusaurus-plugin-aeo)](https://www.npmjs.com/package/@stackql/docusaurus-plugin-aeo) 
+[![NPM Downloads](https://img.shields.io/npm/d18m/%40stackql%2Fdocusaurus-plugin-aeo)](https://www.npmjs.com/package/@stackql/docusaurus-plugin-aeo)
+
 # @stackql/docusaurus-plugin-aeo
 
-AEO (Answer Engine Optimization) helpers for Docusaurus 3.x sites: emit raw `.md` companion files for every page, generate `llms.txt` and `llms-full.txt` at the site root, drop an "Ask AI" dropdown into doc and blog footers, and document a `/ai/*` routing convention for machine-readable companion content. This is a sibling to `@stackql/docusaurus-plugin-structured-data` (which emits JSON-LD); the two plugins compose without overlap.
+AEO (Answer Engine Optimization) helpers for Docusaurus 3.x sites: emit plain-markdown `.md` companions for docs and blog posts, generate `llms.txt` and `llms-full.txt` at the build root, add an "Ask AI" dropdown above doc and blog content, and document a `/ai/*` routing convention for machine-readable companion content. This is a sibling to `@stackql/docusaurus-plugin-structured-data` (which emits JSON-LD); the two plugins compose without overlap.
 
 ## Installation
 
@@ -25,7 +28,7 @@ module.exports = {
 };
 ```
 
-All four features are on by default. To configure, pass options:
+Companions, `llms.txt` / `llms-full.txt`, and the Ask AI button are enabled by default. The `/ai/*` docs instance requires site configuration, and its optional validator is off by default. To configure, pass options:
 
 ```js
 module.exports = {
@@ -45,12 +48,23 @@ module.exports = {
 
 ## Feature 1: `.md` companion files
 
-For every emitted HTML route from the docs and blog plugins, this plugin writes a sibling `.md` file. A page at `/docs/intro` gets a companion at `/docs/intro/index.md` (or `/docs/intro.md` if `siteConfig.trailingSlash` is `false`). The file contains the raw markdown source.
+For each doc or blog post with a readable source file, this plugin writes a sibling `.md` file unless its route matches `companions.exclude` (see [Route patterns](#route-patterns)). The companion mirrors the HTML layout: a page at `/docs/intro` gets `/docs/intro/index.md` (or `/docs/intro.md` if `siteConfig.trailingSlash` is `false`). A doc or blog post at the site root gets `/index.md`. The same URL rule (`src/companionPath.js`) is used by `llms.txt` and the Ask AI button.
 
 Two modes:
 
-- `companions.format: 'raw'` (default): the MDX source is emitted as-is. MDX `<Component />` tags pass through. LLMs handle them fine, and the file is a faithful representation of the page.
-- `companions.format: 'plain'`: a best-effort regex pass strips `import` / `export` lines and JSX tags, then prepends a `# Title\n\n> description` block from the page frontmatter. Not a full remark pipeline - use `'raw'` unless an MDX-heavy page is causing concrete problems.
+- `companions.format: 'plain'` (default since 0.5.0; `'clean'` is accepted as a synonym): plain markdown rendered from the MDX source, with a `# Title`, `> description` and `Source: <page URL>` block from the page metadata at the top, so an LLM can cite the page without parsing YAML. The converter uses remark-mdx, GFM, directives and comments to locate MDX constructs, then edits the original text by source position. It does not load the site's custom remark/rehype plugins:
+  - `import` / `export` statements and `{expressions}` are removed (a string literal expression keeps its value)
+  - presentational elements (`<svg>`, `<video>`, `<iframe>`, icons, ...) are dropped with their subtree
+  - `<Tabs>` / `<TabItem>` become a bold label per tab followed by the tab's content; the label comes from the `label` attribute or from the `<Tabs values={[...]}>` expression
+  - `<a>`, `<img>`, `<b>`, `<i>`, `<code>`, `<h2>`, `<br>`, `<details>` / `<summary>` become their markdown equivalent
+  - a self-closing component that plainly carries a link (`to` or `href` plus `text`, `label` or `title`) becomes a markdown link, so download buttons survive
+  - everything else (`<div>`, `<Box>`, `<span>`, your own layout components) is unwrapped: the tags go, the children stay, and indentation the author added inside the container is removed
+  - HTML comments (including `<!-- truncate -->`) and `{#custom-id}` heading ids are removed outside ordinary code fences; `mdx-code-block` fences are unwrapped and their contents are processed as markup
+
+  Markdown is not re-stringified, so `snake_case` is not escaped and Markdown syntax is retained. Output is not byte-for-byte identical: line endings become LF, surplus blank lines and outer whitespace are trimmed, leading/trailing horizontal rules are removed, and content inside unwrapped containers is dedented. An existing opening H1 is retained instead of adding another title. A Markdown/MDX parse or conversion failure uses a regex fallback and logs a warning; front matter parsing errors are not covered by that fallback.
+
+  Source format follows front matter `format`, then `siteConfig.markdown.format`; `detect` selects by extension (`.md` as Markdown, `.mdx` as MDX). Files resolved as `md` use a CommonMark parser with GFM and directives, with HTML tags stripped or converted.
+- `companions.format: 'raw'`: the MDX source is emitted as-is, front matter included. MDX `<Component />` tags, imports and inline SVG data all pass through.
 
 Fetch example:
 
@@ -58,15 +72,53 @@ Fetch example:
 curl https://your-site.example/docs/intro/index.md
 ```
 
-**MIME type note.** Setting `Content-Type: text/markdown` from a static-site plugin is not possible. The file simply gets a `.md` extension and the host's MIME table handles it. Vercel, Netlify, Cloudflare Pages, GitHub Pages, and S3+CloudFront all serve `.md` as `text/markdown` or `text/plain` out of the box. If you serve from Nginx, ensure `text/markdown md;` is in your `mime.types`.
+**MIME type note.** The plugin writes `.md` files; the host determines their `Content-Type`. Check a deployed companion with `curl -I` and configure the host if needed. The alternate link below advertises the companion as `text/markdown` but does not set its HTTP response type.
+
+### Advertising the companion from the page
+
+Every page that has a companion gets, in its `<head>`:
+
+```html
+<link rel="alternate" type="text/markdown" href="https://your-site.example/docs/intro/index.md">
+```
+
+inserted into the built HTML in `postBuild` (the same pattern `@stackql/docusaurus-plugin-structured-data` uses). An agent that lands on the HTML can find the markdown without guessing the URL, and agent-readiness checkers score it. The href is absolute and follows the same rule as the file on disk. Options:
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `companions.alternateLink` | `true` | Insert the `<link rel="alternate" type="text/markdown">` tag into each page that has a companion. |
+| `companions.linkHeader` | `false` | Also send the relation as an HTTP response header, on hosts that read a `_headers` file (Netlify, Cloudflare Pages): one `Link: <url>; rel="alternate"; type="text/markdown"` rule per page, written to `_headers` in the build root (appended to a `_headers` your `static/` already ships). It has no effect on other hosts; see [Hosting notes](#hosting-notes). Requires `companions.alternateLink: true`. |
+
+Generated category indexes, custom React pages and blog listing pages do not get companions. `verbose: true` logs unreadable source files and missing HTML during alternate-link insertion.
+
+### Localized sites and non-root `baseUrl`
+
+The plugin works from `siteConfig.url` and `siteConfig.baseUrl` as Docusaurus reports them for the build, so a site at the root, a site built at `/docs/` and a site with several locales are configured the same way.
 
-Non-content routes (custom React pages, redirects, the 404 page, search) are skipped silently. Enable `verbose: true` to log skips.
+On a site with Docusaurus i18n the plugin runs once per locale, like every plugin, and needs no extra configuration: each locale gets its own companions (`/de/docs/intro.md` from the translated source, or from the default-locale source when a page has no translation), its own `/de/llms.txt` and `/de/llms-full.txt`, and alternate links and `Source:` lines that carry the locale prefix. The localized home page `/de/` is a root, so its companion is `/de/index.md`. The same holds for a site built at a non-root `baseUrl`: files are written relative to the build directory, URLs carry the base.
+
+### Route patterns
+
+`companions.exclude`, `llmsTxt.exclude` and `llmsTxt.include` take glob patterns for routes. `**` matches across path segments; `*` and `?` match within a single segment.
+
+Write a pattern for the route as it is on the site, without the base URL: `/docs/internal/**`. Docusaurus permalinks carry the base URL (`/de/docs/internal/notes` in a `de` build, `/docs/guide/setup` on a site built at `/docs/`), so a pattern is matched against the route both with and without it. One list therefore applies to every locale, and a pattern that includes the base still works and can target a single locale:
+
+```js
+companions: {
+  exclude: [
+    '/docs/internal/**',  // every locale: /docs/internal/..., /de/docs/internal/...
+    '/de/blog/**',        // the de locale only
+  ],
+},
+```
+
+A page excluded from companions has no `.md` file, no alternate link, no `llms.txt` entry and no Ask AI button.
 
 ## Feature 2: `llms.txt` and `llms-full.txt`
 
-After feature 1 emits all companions, the plugin writes two files to the build root:
+With the defaults enabled, the plugin writes two files to the build root after emitting companions. Both use the emitted companion list, filtered by `llmsTxt.include` and `llmsTxt.exclude`. If companions are disabled, `llms.txt` contains only the site title, tagline and optional header; no page entries or `llms-full.txt` are emitted.
 
-- `llms.txt` - sectioned index of every doc/blog page, in the format described at <https://llmstxt.org>:
+- `llms.txt` - sectioned index of included doc/blog companions, in the format described at <https://llmstxt.org>. Links are absolute (`https://site/foo.md`) since 0.5.0, as the spec's examples are; the file is read away from the site, where a relative link has no base:
 
   ```text
   # StackQL
@@ -75,15 +127,21 @@ After feature 1 emits all companions, the plugin writes two files to the build r
 
   ## Documentation
 
-  - [Getting started](/docs/intro/index.md): Install StackQL and run your first query.
-  - [Providers](/docs/providers/index.md): Catalog of supported cloud APIs.
+  - [Getting started](https://stackql.io/docs/intro/index.md): Install StackQL and run your first query.
+  - [Providers](https://stackql.io/docs/providers/index.md): Catalog of supported cloud APIs.
 
   ## Blog
 
-  - [Querying Snowflake with StackQL](/blog/snowflake/index.md): ...
+  - [Querying Snowflake with StackQL](https://stackql.io/blog/snowflake/index.md): ...
+
+  ## Optional
+
+  - [Full text of every page](https://stackql.io/llms-full.txt): one file with the content of every page listed above
   ```
 
-- `llms-full.txt` - every companion concatenated with a `\n---\n\n` separator, each block prefixed with the page's title and URL so an LLM can cite individual sections.
+  The `## Optional` section is the llmstxt.org convention for secondary resources a reader can skip; it points at `llms-full.txt` whenever that file is emitted (`llmsTxt.linkFullTxt`, default `true`).
+
+- `llms-full.txt` - the included companions concatenated with a `\n---\n\n` separator, each block carrying a `Source: <page URL>` line and the page's title (a plain companion already opens with both; a raw one has them added) so an LLM can cite individual sections. Limit it to chosen content instances or a byte budget with the object form of `llmsTxt.fullTxt` (below).
 
 Options:
 
@@ -92,16 +150,17 @@ Options:
 | `llmsTxt.enabled` | `true` | Master switch. |
 | `llmsTxt.exclude` | `["/search", "/404", "/blog/tags/**", "/blog/page/**", "/blog/archive", "/blog/authors/**"]` | Route glob patterns to skip. |
 | `llmsTxt.include` | `null` | If set, only routes matching at least one pattern are included. Use for opt-in mode. |
-| `llmsTxt.header` | `null` | String prepended to `llms.txt` above the section list. Good for a project intro paragraph or links to key external resources. |
+| `llmsTxt.header` | `null` | String inserted after the site title/tagline and before the section list. Good for a project intro paragraph or links to key external resources. |
 | `llmsTxt.sections` | `{ docs: 'Documentation', blog: 'Blog', pages: 'Pages' }` | Section title overrides for the default per-type grouping. |
 | `llmsTxt.instanceSections` | `null` | Per-content-plugin-instance section map. When set, supersedes `sections` (see below). |
-| `llmsTxt.fullTxt` | `true` | Emit `llms-full.txt`. |
+| `llmsTxt.fullTxt` | `true` | Emit `llms-full.txt`. `true` is every section, uncapped. The object form `{ include?: string[], maxBytes?: number }` limits it: `include` names the content instances to concatenate (`'<pluginName>@<pluginId>'` keys, e.g. `'docusaurus-plugin-content-docs@ai'`; with per-type grouping the keys are `docs`, `blog`, `pages`), in `llms.txt` section order; `maxBytes` stops appending once the next page would cross the cap, so the file always ends on a whole page. The budget includes section headings and separators. If the first page exceeds it, the file is empty. An omitted or empty `include` selects all sections. |
+| `llmsTxt.linkFullTxt` | `true` | Add the `## Optional` section to `llms.txt` with a link to `llms-full.txt`. |
 
-Glob patterns are matched against route permalinks. `**` matches across path segments; `*` matches within a single segment.
+`exclude` and `include` follow the rules in [Route patterns](#route-patterns): a pattern matches a route with or without the base URL, so the defaults apply to every locale.
 
 ### Per-instance sectioning
 
-By default, `llms.txt` groups by content-plugin TYPE: every `@docusaurus/plugin-content-docs` instance lands under one `## Documentation` heading, every blog instance under `## Blog`, and so on. This is fine for sites with a single docs instance, but it breaks down for sites that run multiple docs instances with different audiences — for example, one for human-facing docs at `/docs/*` and one for AI-targeted reference content at `/ai/*`. In that case, `llms.txt` and `llms-full.txt` interleave both surfaces under one heading, hiding the corpus shape from the crawlers and agents the file exists to serve.
+By default, `llms.txt` groups by content-plugin TYPE: every `@docusaurus/plugin-content-docs` instance lands under one `## Documentation` heading, every blog instance under `## Blog`, and so on. This is fine for sites with a single docs instance, but it breaks down for sites that run multiple docs instances with different audiences - for example, one for human-facing docs at `/docs/*` and one for AI-targeted reference content at `/ai/*`. In that case, `llms.txt` and `llms-full.txt` interleave both surfaces under one heading, hiding the corpus shape from the crawlers and agents the file exists to serve.
 
 Set `llmsTxt.instanceSections` to switch to per-instance grouping. Keys are `"${pluginName}@${pluginId}"`; each value is `{ title, order? }`.
 
@@ -146,31 +205,35 @@ Resulting `llms.txt`:
 
 ## Documentation
 
-- [Getting started](/docs/intro/index.md): Install and run your first query.
-- [Providers](/docs/providers/index.md): Catalog of supported cloud APIs.
+- [Getting started](https://your-site.example/docs/intro/index.md): Install and run your first query.
+- [Providers](https://your-site.example/docs/providers/index.md): Catalog of supported cloud APIs.
 
 ## AI Reference
 
-- [What is StackQL](/ai/faqs/what-is-stackql/index.md): Canonical one-paragraph definition.
-- [Connecting to AWS](/ai/howto/connect-to-aws/index.md): Step-by-step authentication.
+- [What is StackQL](https://your-site.example/ai/faqs/what-is-stackql/index.md): Canonical one-paragraph definition.
+- [Connecting to AWS](https://your-site.example/ai/howto/connect-to-aws/index.md): Step-by-step authentication.
 
 ## Blog
 
-- [Querying Snowflake](/blog/snowflake/index.md): ...
+- [Querying Snowflake](https://your-site.example/blog/snowflake/index.md): ...
+
+## Optional
+
+- [Full text of every page](https://your-site.example/llms-full.txt): one file with the content of every page listed above
 ```
 
 `llms-full.txt` mirrors the same section structure: each section's title is emitted as an `## H2` heading above its concatenated companion blocks, in the same order.
 
 Behavior:
 
-- Sections appear in ascending `order`. Ties break alphabetically by title. Entries with no `order` go last.
+- Non-empty sections appear in ascending `order`. Ties break alphabetically by title. Sections with no `order` go last. Pages within each mapped section are sorted by permalink.
 - Instances not present in `instanceSections` are collected into a single appended `## Other` section, sorted alphabetically by page title. When `verbose: true`, each unmapped instance name is logged once so you notice and can map it.
 - `llmsTxt.sections` (the per-type titles) is ignored when `instanceSections` is set. When `verbose: true`, the plugin logs a one-line notice if both are configured.
-- When `instanceSections` is `null` (default), behavior is identical to v0.1.x: per-type grouping using `llmsTxt.sections` titles.
+- When `instanceSections` is `null` (default), grouping is by content type using `llmsTxt.sections` titles.
 
 ## Feature 3: "Ask AI" button
 
-An outlined pill button with a caret reading "Ask AI about this page" is injected at the top of every doc and blog-post content area, right-aligned in the breadcrumb row. Each item in the dropdown opens the corresponding AI surface in a new tab with a prefilled prompt that references the current page's `.md` companion. The button is hidden on viewports under 997px to keep the breadcrumb row uncluttered on mobile.
+An outlined pill button with a caret reading "Ask AI about this page" appears above eligible doc and blog-post content. Each item in the dropdown opens the corresponding AI surface in a new tab with a prefilled prompt that references the current page's `.md` companion. The button is hidden on viewports under 997px to keep the breadcrumb row uncluttered on mobile.
 
 Placement specifics:
 
@@ -188,11 +251,21 @@ Providers and URL patterns:
 Default prompt:
 
 ```text
-Read https://your-site.example/path/to/page.md and help me understand it. Summarize the key points, then ask me one clarifying question to dig deeper.
+Read https://your-site.example/path/to/page/index.md and help me understand it. Summarize the key points, then ask me one clarifying question to dig deeper.
 ```
 
 The default is self-contained - submitting it as-is yields a useful summary plus a follow-up question. The prompt is prefilled in each provider's input box, so users can still edit or replace it before sending.
 
+Template placeholders:
+
+| Placeholder | Value |
+| --- | --- |
+| `{companionUrl}` | The page's `.md` companion, by the same rule feature 1 writes it: `https://site/index.md` for the homepage, `https://site/foo.md` or `https://site/foo/index.md` depending on `trailingSlash`. With feature 1 disabled, the page URL. |
+| `{pageUrl}` | The page itself: canonical URL, no trailing slash, except on a root (`https://site/`, or a base URL such as `https://site/de/`). |
+| `{pageUrl}.md` | Still honoured for templates written against v0.4, and resolves to `{companionUrl}`. (On its own it produced `https://site.md` for the homepage and the wrong path on sites that build `/foo/index.html`.) |
+
+The button only renders on pages that have a companion. Generated category index pages, React pages and blog list pages have no `.md` file, so on those the button is not shown rather than sending the reader to a 404. (The list of routes comes from `allContentLoaded`; with feature 1 disabled, the companion-route filter is disabled and the prompt uses the page URL wherever the button is rendered.)
+
 Provider icons are hand-rolled inline SVGs in each provider's brand color (Claude `#D97757`, ChatGPT `#000000`, Perplexity `#21808D`). The Claude and Perplexity SVG paths come from [simple-icons](https://simpleicons.org) (CC0-1.0); the OpenAI mark is sourced from the @lobehub/icons project. Bundle cost is ~3KB across the three components combined.
 
 > **Gemini was previously supported but was removed in 0.4.2** because `gemini.google.com` silently ignores URL-encoded prompts, leaving users on an empty prompt box. There is no documented Gemini URL prefill API and Google has not indicated one is coming.
@@ -204,15 +277,28 @@ Options:
 | Option | Default | Description |
 | --- | --- | --- |
 | `askAi.enabled` | `true` | When `false`, the theme components are not registered. |
-| `askAi.providerOrder` | `['claude', 'chatgpt', 'perplexity']` | Order of items in the dropdown. Drop entries to hide them. Valid values: `'claude'`, `'chatgpt'`, `'perplexity'`. |
-| `askAi.promptTemplate` | `'Read {pageUrl}.md and help me understand it. Summarize the key points, then ask me one clarifying question to dig deeper.'` | Prompt sent to each provider. `{pageUrl}` is the page's canonical URL (no trailing slash). If feature 1 is disabled, the consumer should remove the `.md` from the template. |
-| `askAi.placement` | `'breadcrumb-row'` | `'breadcrumb-row'` puts the button at the top of every doc/blog page (docs breadcrumb row, or above the blog title). `'none'` does not register any theme components - swizzle the button into your preferred location manually. |
+| `askAi.providerOrder` | `['claude', 'chatgpt', 'perplexity']` | Order of items in the dropdown. Omit providers to hide them; an empty array falls back to the default order. Valid values: `'claude'`, `'chatgpt'`, `'perplexity'`. |
+| `askAi.promptTemplate` | `'Read {companionUrl} and help me understand it. Summarize the key points, then ask me one clarifying question to dig deeper.'` | Prompt sent to each provider. `{companionUrl}` is the page's `.md` companion (the page URL when feature 1 is disabled); `{pageUrl}` is the page's canonical URL (no trailing slash except on a root). See the placeholder table above. |
+| `askAi.placement` | `'breadcrumb-row'` | `'breadcrumb-row'` puts the button at the top of every doc/blog page (docs breadcrumb row, or above the blog title). `'none'` disables all bundled theme components, including the `@theme/AskAiButton` alias. See custom placement below. |
+
+The button is built from [MUI](https://mui.com) primitives - outlined `Button` with a `KeyboardArrowDownIcon` caret as the trigger, and a `Menu` of `MenuItem` rows for the providers. Theming reads `--ifm-color-primary` and `--ifm-font-family-base` via MUI's `sx` prop, so the button takes the site's own colours and dark/light mode work automatically. The MUI `Menu` handles click-outside-to-close and Esc-to-close natively.
+
+### Translating the button
+
+The visible strings go through Docusaurus `translate()`, so `docusaurus write-translations --locale <locale>` extracts them into `i18n/<locale>/code.json` with the rest of the theme's strings:
+
+| Id | Default |
+| --- | --- |
+| `aeo.askAi.button` | Ask AI about this page |
+| `aeo.askAi.provider.claude` | Ask Claude |
+| `aeo.askAi.provider.chatgpt` | Ask ChatGPT |
+| `aeo.askAi.provider.perplexity` | Ask Perplexity |
 
-The button is built from [MUI](https://mui.com) primitives - outlined `Button` with a `KeyboardArrowDownIcon` caret as the trigger, and a `Menu` of `MenuItem` rows for the providers. Theming reads `--ifm-color-primary` and `--ifm-font-family-base` via MUI's `sx` prop, so dark/light mode work automatically. The MUI `Menu` handles click-outside-to-close and Esc-to-close natively.
+A locale with no translation for an id shows the default. The prompt itself is `askAi.promptTemplate`; in a localized build its URLs point at that locale's page and companion.
 
 ### Peer dependencies
 
-When `askAi.enabled` is `true` (the default), the following peer dependencies must be installed by the consumer site:
+The package declares these MUI/Emotion peers as required, in addition to Docusaurus `^3.0.0` and React/React DOM `^18.0.0 || ^19.0.0`:
 
 | Package | Range |
 | --- | --- |
@@ -221,19 +307,21 @@ When `askAi.enabled` is `true` (the default), the following peer dependencies mu
 | `@emotion/react` | `^11.0.0` |
 | `@emotion/styled` | `^11.0.0` |
 
-These are declared as peer dependencies (not direct dependencies) so consumers that already use MUI - common in Docusaurus sites - get a single deduped copy at install time. Consumers without MUI will get a clean npm peer-dependency error pointing at exactly what to install.
+These are peer dependencies so the plugin can share the site's MUI installation. npm can install missing peers automatically; other package managers may require you to add them explicitly.
 
-Consumers who disable the Ask AI button (`askAi.enabled: false`) can ignore these peers; the theme components are not registered in that case and MUI is never imported.
+Disabling Ask AI (`askAi.enabled: false`) prevents the plugin from registering its theme components, so they do not import MUI at runtime. It does not change the package's required peer declarations or the package manager's installation requirements.
 
 ### Customizing placement
 
-To put the button somewhere other than the breadcrumb row - sidebar, header, a specific page region - set `askAi.placement: 'none'` to disable the bundled swizzles, then import and place the component manually wherever you want:
+`askAi.placement: 'none'` disables the entire theme path, so `@theme/AskAiButton` is unavailable with that setting.
+
+To reuse the bundled button in a custom location, keep Ask AI enabled with `placement: 'breadcrumb-row'`. Add site-level theme overrides for `DocBreadcrumbs` and `BlogPostItem/Header/Title` that omit the default button, then import it in your chosen site component:
 
 ```jsx
 import AskAiButton from '@theme/AskAiButton';
 ```
 
-Wrap an existing theme component (e.g. `@theme/Layout`) the same way Docusaurus documents for any swizzle.
+The button retains its companion-route filter and desktop-only styling in a custom location.
 
 ## Feature 4: `/ai/*` routing convention
 
@@ -241,7 +329,7 @@ This plugin documents but does not auto-configure a second docs instance for mac
 
 ```js
 // docusaurus.config.js
-const { sitemapExclude, AI_ROUTE_PATTERNS } = require('@stackql/docusaurus-plugin-aeo/helpers');
+const { sitemapExclude } = require('@stackql/docusaurus-plugin-aeo/helpers');
 
 module.exports = {
   plugins: [
@@ -292,23 +380,38 @@ ai-content/
     └── stackql-cli.md         # frontmatter: softwareApplication: { ... }
 ```
 
-The companion files emitted by feature 1 will live at `/ai/faqs/<slug>/index.md` (etc.), and they will appear in `llms.txt` and `llms-full.txt` just like any other doc, which is the point.
+With default filters, these docs appear in `llms.txt` and `llms-full.txt`. Their companions follow the same URL rule: `/ai/faqs/<slug>/index.md`, or `/ai/faqs/<slug>.md` when `trailingSlash: false`.
+
+The convention is relative to the site's base URL: on a site built at `/docs/` the routes are `/docs/ai/*`, and in a `de` build they are `/de/ai/*`.
 
 ### Optional validation
 
-Set `aiRoutes.validate: true` to fail the build (with warnings, not errors) when files under `ai/faqs/`, `ai/howto/`, or `ai/apps/` are missing the corresponding frontmatter payload (`faq`, `howTo`, `softwareApplication`). Off by default.
+Set `aiRoutes.validate: true` to warn when docs at `/ai/faqs/*`, `/ai/howto/*`, or `/ai/apps/*` lack the corresponding frontmatter payload (`faq`, `howTo`, `softwareApplication`). Validation uses route permalinks (relative to the base URL, so every locale is checked), not source directories, and does not fail the build. Set `verbose: true` for per-page details. Off by default.
 
 ### Helpers
 
 ```js
 const {
   AI_ROUTE_PATTERNS,             // ['/ai', '/ai/**'] - glob patterns
+  aiRoutePatterns,               // (baseUrl) => the same patterns under a base URL
   sitemapExclude,                // { ignorePatterns: AI_ROUTE_PATTERNS }
-  isAiRoute,                     // (permalink) => boolean
-  buildStructuredDataAiExcludes, // (routePaths[]) => string[]
+  isAiRoute,                     // (route, baseUrl?) true for /ai/*; false for /ai itself
+  buildStructuredDataAiExcludes, // (routePaths[], baseUrl?) => string[]
 } = require('@stackql/docusaurus-plugin-aeo/helpers');
 ```
 
+`AI_ROUTE_PATTERNS` and `sitemapExclude` describe a site at the root with one locale. These helpers run in your config, where other plugins match the full route path, base URL included (the Docusaurus docs say as much of the sitemap plugin's `ignorePatterns`). With a non-root `baseUrl` or several locales, build the list from `aiRoutePatterns`, one call per base URL:
+
+```js
+const { aiRoutePatterns } = require('@stackql/docusaurus-plugin-aeo/helpers');
+
+// a site with en at / and de at /de/
+const ignorePatterns = ['/', '/de/'].flatMap(aiRoutePatterns);
+// ['/ai', '/ai/**', '/de/ai', '/de/ai/**']
+```
+
+`isAiRoute` and `buildStructuredDataAiExcludes` take the base URL as an optional second argument.
+
 **Heads-up on the structured-data plugin.** `@stackql/docusaurus-plugin-structured-data`'s `themeConfig.structuredData.excludedRoutes` compares routes with strict equality, so glob patterns like `/ai/**` do not work there. Use one of these approaches:
 
 - List the exact `/ai/*` permalinks you want skipped by hand (works if your `/ai/*` tree is small and stable).
@@ -328,6 +431,16 @@ const {
 
 The `sitemapExclude` helper, by contrast, does work with globs because `@docusaurus/plugin-sitemap` uses `ignorePatterns` (micromatch under the hood).
 
+## Hosting notes
+
+Everything the plugin produces is a static file or an edit to the built HTML, so companions, `llms.txt`, `llms-full.txt`, the alternate link tag and the Ask AI button work on any static host with no host configuration. A few things do depend on where the site is deployed:
+
+- **`Content-Type` of `.md` files.** Set by the host; see the MIME type note under Feature 1.
+- **GitHub Pages.** GitHub Pages runs published files through Jekyll by default. Jekyll builds sites from Markdown, so `.md` companions may be turned into pages instead of being published as they are. Add an empty `.nojekyll` file to `static/`, as the [Docusaurus deployment guide](https://docusaurus.io/docs/deployment) recommends. A project site is served under `/<repo>/`: set `baseUrl` as usual and the plugin follows it.
+- **`companions.linkHeader` is for hosts that read `_headers`.** That is Netlify and Cloudflare Pages. Vercel documents response headers through `vercel.json`, and GitHub Pages has no server-side configuration for them, so the option does nothing there. The alternate link tag, on by default, needs nothing from the host and is the portable way to advertise a companion. The file gets one rule per page that has a companion; Cloudflare Pages documents a limit of 100 header rules, so leave the option off there on a larger site.
+- **Where `_headers` is written.** At the root of the build directory, which is where both hosts document the file. A build of several locales (`docusaurus build`, one deployment serving `/` and `/de/`) puts every locale's rules in that one file. A locale built on its own (`docusaurus build --locale de`) or given its own `baseUrl` is a separate deployment and gets the file in its own output directory.
+- **Preview deployments.** Companion URLs, `Source:` lines, `llms.txt` links and the Ask AI prompt are absolute and come from `siteConfig.url`, so on a deploy preview they point at the production site.
+
 ## Full options reference
 
 ```js
@@ -335,8 +448,10 @@ The `sitemapExclude` helper, by contrast, does work with globs because `@docusau
   // Feature 1
   companions: {
     enabled: true,            // emit .md siblings
-    format: 'raw',            // 'raw' | 'plain'
+    format: 'plain',          // 'plain' ('clean' is a synonym) | 'raw'
     exclude: [],              // route glob patterns to skip
+    alternateLink: true,      // <link rel="alternate" type="text/markdown"> in each page
+    linkHeader: false,        // also a Link: header via a _headers file (Netlify, Cloudflare Pages)
   },
 
   // Feature 2
@@ -351,7 +466,7 @@ The `sitemapExclude` helper, by contrast, does work with globs because `@docusau
       '/blog/authors/**',
     ],
     include: null,            // null = all not-excluded
-    header: null,             // optional string prepended to llms.txt
+    header: null,             // optional text after the site title/tagline
     sections: {               // per-type section titles (used when
       docs: 'Documentation',  // instanceSections is null)
       blog: 'Blog',
@@ -361,14 +476,16 @@ The `sitemapExclude` helper, by contrast, does work with globs because `@docusau
                               // sections when set. Shape:
                               //   { '${pluginName}@${pluginId}':
                               //       { title: string, order?: number } }
-    fullTxt: true,            // emit llms-full.txt
+    fullTxt: true,            // emit llms-full.txt: true | false |
+                              //   { include: ['<plugin>@<id>'], maxBytes: 65536 }
+    linkFullTxt: true,        // "## Optional" section in llms.txt -> llms-full.txt
   },
 
   // Feature 3
   askAi: {
     enabled: true,
     providerOrder: ['claude', 'chatgpt', 'perplexity'],
-    promptTemplate: 'Read {pageUrl}.md and help me understand it. Summarize the key points, then ask me one clarifying question to dig deeper.',
+    promptTemplate: 'Read {companionUrl} and help me understand it. Summarize the key points, then ask me one clarifying question to dig deeper.',
     placement: 'breadcrumb-row',  // 'breadcrumb-row' | 'none'
   },
 
@@ -384,12 +501,29 @@ The `sitemapExclude` helper, by contrast, does work with globs because `@docusau
 
 Options are validated by a small handwritten validator at plugin construction. Invalid options throw a clear error before the build starts.
 
+## Development
+
+Use Node.js 24.x to match CI. From the repository root:
+
+```bash
+npm ci
+npm test
+```
+
+`npm test` runs every `test/*.test.js` file with Node's test runner: companion paths and files, route patterns, Ask AI prompts and the rendered button, plain Markdown, alternate links/headers, `llms.txt`, AI route classification and helpers, and the plugin run once per locale as Docusaurus runs it (`test/localizedBuild.test.js`). Shared fixtures and the loader that renders theme components in Node are in `test/helpers/`. There are no dev dependencies: the tests use the peer dependencies and the Babel setup that comes with `@docusaurus/core`.
+
+The [Tests workflow](.github/workflows/tests.yml) runs on every pull request targeting `main` and on pushes to `main`, using the committed lockfile and Node 24.x. To make it a merge gate, require the **Tests (Node 24)** status check in the branch protection rule or ruleset for `main`. Until that rule is enabled, a failed check does not block merging.
+
+When `npm audit` reports advisories, they are almost always in the Docusaurus toolchain, which is installed here only because it is a peer dependency. Do not run `npm audit fix`: it clears the report by downgrading the `@docusaurus/core` peer to an older release with a different set of advisories. Update the named packages instead (`npm update <package> ...`), which moves them within the ranges Docusaurus already allows, then re-run `npm audit` and `npm test`.
+
+The plain-markdown converter lives in `src/features/plainMarkdown.js` and is exercised by `test/plainMarkdown.test.js` with small MDX fixtures; add a fixture there when a new construct needs handling.
+
 ## Composing with `@stackql/docusaurus-plugin-structured-data`
 
 The two plugins are complementary and intended to be used together:
 
 - `@stackql/docusaurus-plugin-structured-data` emits JSON-LD into page `<head>`. <https://github.com/stackql/docusaurus-plugin-structured-data>
-- `@stackql/docusaurus-plugin-aeo` does everything else AEO-adjacent: raw `.md` companions, `llms.txt`, the Ask AI button, the `/ai/*` convention.
+- `@stackql/docusaurus-plugin-aeo` provides plain-markdown `.md` companions by default (raw mode is optional), `llms.txt` / `llms-full.txt`, the Ask AI button, and the `/ai/*` convention.
 
 There is no overlap and no shared state. Add both:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@stackql/docusaurus-plugin-aeo",
-  "version": "0.4.2",
+  "version": "0.5.1",
   "description": "AEO (Answer Engine Optimization) helpers for Docusaurus: .md companion files, llms.txt / llms-full.txt, an Ask AI dropdown, and /ai/* route conventions.",
   "main": "src/index.js",
   "exports": {
@@ -17,7 +17,7 @@
     "access": "public"
   },
   "scripts": {
-    "test": "echo \"Error: no test specified\" && exit 1"
+    "test": "node --test test/*.test.js"
   },
   "peerDependencies": {
     "@docusaurus/core": "^3.0.0",
@@ -43,11 +43,21 @@
     }
   },
   "dependencies": {
-    "gray-matter": "^4.0.3"
+    "@slorber/remark-comment": "^1.0.0",
+    "gray-matter": "npm:@11ty/gray-matter@1.0.0",
+    "remark-directive": "^3.0.0",
+    "remark-gfm": "^4.0.0",
+    "remark-mdx": "^3.0.0",
+    "remark-parse": "^11.0.0",
+    "unified": "^11.0.3"
   },
   "overrides": {
     "serialize-javascript": "^7.0.5",
-    "uuid": "^11.1.1"
+    "uuid": "^11.1.1",
+    "@docusaurus/core": {
+      "tinypool": "2.1.2"
+    },
+    "postcss-selector-parser": "7.1.6"
   },
   "repository": {
     "type": "git",
@@ -67,5 +77,11 @@
   "bugs": {
     "url": "https://github.com/stackql-labs/docusaurus-plugin-aeo/issues"
   },
-  "homepage": "https://github.com/stackql-labs/docusaurus-plugin-aeo#readme"
+  "homepage": "https://github.com/stackql-labs/docusaurus-plugin-aeo#readme",
+  "engines": {
+    "node": ">=18.0"
+  },
+  "allowScripts": {
+    "core-js": false
+  }
 }

package/src/askAiPrompt.js

@@ -0,0 +1,22 @@
+// Fills the Ask AI prompt template. Plain CommonJS so the theme component
+// can bundle it and the tests can require it.
+//
+//   {companionUrl}  the page's .md companion (absolute URL)
+//   {pageUrl}       the page itself (canonical: no trailing slash, except
+//                   on a root - the site root or a base URL such as '/de/')
+//
+// The v0.4 idiom `{pageUrl}.md` is still honoured so a custom template
+// keeps working. It was wrong for the homepage (`https://site.md`) and for
+// sites that build /foo/index.html; it now resolves to the real companion
+// URL.
+function fillPrompt(template, { pageUrl, companionUrl }) {
+  return String(template || '')
+    .split('{pageUrl}.md')
+    .join(companionUrl)
+    .split('{companionUrl}')
+    .join(companionUrl)
+    .split('{pageUrl}')
+    .join(pageUrl);
+}
+
+module.exports = { fillPrompt };