@stackql/docusaurus-plugin-aeo 0.1.2 → 0.2.0

package/CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## 0.2.0
+
+- Added: `llmsTxt.instanceSections` option for per-content-plugin-instance llms.txt section grouping. Use case: consumers with multiple content-docs instances (e.g. human docs + AI reference) can split them into named sections. Map keys are `"${pluginName}@${pluginId}"`; values are `{ title, order? }`. Unmapped instances are collected into a single appended "Other" section. `llms-full.txt` mirrors the same section structure as `llms.txt`.
+- No breaking changes: when `llmsTxt.instanceSections` is unset (the default), grouping is identical to v0.1.x — per content-plugin type, using `llmsTxt.sections` titles.
+
 ## 0.1.2
 
 Fix: cross-plugin loaded content was not captured because contentLoaded does not receive allContent in Docusaurus 3.x. Use allContentLoaded hook. Without this fix, feature 1 (.md companions) emitted zero files and feature 2 (llms.txt / llms-full.txt) was empty.

package/README.md

@@ -93,11 +93,81 @@ Options:
 | `llmsTxt.exclude` | `["/search", "/404", "/blog/tags/**", "/blog/page/**", "/blog/archive", "/blog/authors/**"]` | Route glob patterns to skip. |
 | `llmsTxt.include` | `null` | If set, only routes matching at least one pattern are included. Use for opt-in mode. |
 | `llmsTxt.header` | `null` | String prepended to `llms.txt` above the section list. Good for a project intro paragraph or links to key external resources. |
-| `llmsTxt.sections` | `{ docs: 'Documentation', blog: 'Blog', pages: 'Pages' }` | Section title overrides. |
+| `llmsTxt.sections` | `{ docs: 'Documentation', blog: 'Blog', pages: 'Pages' }` | Section title overrides for the default per-type grouping. |
+| `llmsTxt.instanceSections` | `null` | Per-content-plugin-instance section map. When set, supersedes `sections` (see below). |
 | `llmsTxt.fullTxt` | `true` | Emit `llms-full.txt`. |
 
 Glob patterns are matched against route permalinks. `**` matches across path segments; `*` matches within a single segment.
 
+### Per-instance sectioning
+
+By default, `llms.txt` groups by content-plugin TYPE: every `@docusaurus/plugin-content-docs` instance lands under one `## Documentation` heading, every blog instance under `## Blog`, and so on. This is fine for sites with a single docs instance, but it breaks down for sites that run multiple docs instances with different audiences — for example, one for human-facing docs at `/docs/*` and one for AI-targeted reference content at `/ai/*`. In that case, `llms.txt` and `llms-full.txt` interleave both surfaces under one heading, hiding the corpus shape from the crawlers and agents the file exists to serve.
+
+Set `llmsTxt.instanceSections` to switch to per-instance grouping. Keys are `"${pluginName}@${pluginId}"`; each value is `{ title, order? }`.
+
+Two content-docs instances + blog in `docusaurus.config.js`:
+
+```js
+module.exports = {
+  plugins: [
+    '@docusaurus/plugin-content-blog',           // id: 'default'
+    '@docusaurus/plugin-content-docs',           // id: 'default'
+    [
+      '@docusaurus/plugin-content-docs',
+      {
+        id: 'ai',
+        routeBasePath: '/ai',
+        path: 'ai-content',
+        sidebarPath: false,
+      },
+    ],
+    [
+      '@stackql/docusaurus-plugin-aeo',
+      {
+        llmsTxt: {
+          instanceSections: {
+            'docusaurus-plugin-content-docs@default': { title: 'Documentation', order: 1 },
+            'docusaurus-plugin-content-docs@ai':      { title: 'AI Reference',  order: 2 },
+            'docusaurus-plugin-content-blog@default': { title: 'Blog',          order: 3 },
+          },
+        },
+      },
+    ],
+  ],
+};
+```
+
+Resulting `llms.txt`:
+
+```text
+# Your site
+
+> Your tagline.
+
+## Documentation
+
+- [Getting started](/docs/intro/index.md): Install and run your first query.
+- [Providers](/docs/providers/index.md): Catalog of supported cloud APIs.
+
+## AI Reference
+
+- [What is StackQL](/ai/faqs/what-is-stackql/index.md): Canonical one-paragraph definition.
+- [Connecting to AWS](/ai/howto/connect-to-aws/index.md): Step-by-step authentication.
+
+## Blog
+
+- [Querying Snowflake](/blog/snowflake/index.md): ...
+```
+
+`llms-full.txt` mirrors the same section structure: each section's title is emitted as an `## H2` heading above its concatenated companion blocks, in the same order.
+
+Behavior:
+
+- Sections appear in ascending `order`. Ties break alphabetically by title. Entries with no `order` go last.
+- Instances not present in `instanceSections` are collected into a single appended `## Other` section, sorted alphabetically by page title. When `verbose: true`, each unmapped instance name is logged once so you notice and can map it.
+- `llmsTxt.sections` (the per-type titles) is ignored when `instanceSections` is set. When `verbose: true`, the plugin logs a one-line notice if both are configured.
+- When `instanceSections` is `null` (default), behavior is identical to v0.1.x: per-type grouping using `llmsTxt.sections` titles.
+
 ## Feature 3: "Ask AI" button
 
 A swizzle-friendly dropdown is injected above the existing `DocItem/Footer` and `BlogPostItem/Footer`. Each item opens the corresponding AI surface in a new tab with a prefilled prompt that references the current page's `.md` companion.
@@ -255,11 +325,15 @@ The `sitemapExclude` helper, by contrast, does work with globs because `@docusau
     ],
     include: null,            // null = all not-excluded
     header: null,             // optional string prepended to llms.txt
-    sections: {
-      docs: 'Documentation',
+    sections: {               // per-type section titles (used when
+      docs: 'Documentation',  // instanceSections is null)
       blog: 'Blog',
       pages: 'Pages',
     },
+    instanceSections: null,   // per-instance section map, supersedes
+                              // sections when set. Shape:
+                              //   { '${pluginName}@${pluginId}':
+                              //       { title: string, order?: number } }
     fullTxt: true,            // emit llms-full.txt
   },
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@stackql/docusaurus-plugin-aeo",
-  "version": "0.1.2",
+  "version": "0.2.0",
   "description": "AEO (Answer Engine Optimization) helpers for Docusaurus: .md companion files, llms.txt / llms-full.txt, an Ask AI dropdown, and /ai/* route conventions.",
   "main": "src/index.js",
   "exports": {

package/src/features/llmsTxt.js

@@ -52,6 +52,103 @@ function groupKind(kind, pluginId) {
   return 'pages';
 }
 
+function instanceKey(item) {
+  return `${item.pluginName}@${item.pluginId}`;
+}
+
+// Returns an ordered array of { key, title, items[] }. The same shape is
+// consumed by both llms.txt and llms-full.txt emitters so their section
+// structure stays in lock-step.
+function buildSections(filtered, options, verbose) {
+  if (options.instanceSections) {
+    if (verbose && options.sections && Object.keys(options.sections).length > 0) {
+      // Both options set is allowed; instanceSections wins. Log so the
+      // consumer isn't confused that their `sections` titles aren't
+      // appearing.
+      console.log(
+        '[plugin-aeo] llmsTxt: both `sections` and `instanceSections` are set; `instanceSections` takes precedence',
+      );
+    }
+
+    const mapped = new Map(); // key -> { title, order, items[] }
+    const unmapped = []; // items that don't match any configured instance
+    const seenUnmappedKeys = new Set();
+
+    for (const [key, cfg] of Object.entries(options.instanceSections)) {
+      mapped.set(key, { title: cfg.title, order: cfg.order, items: [] });
+    }
+
+    for (const item of filtered) {
+      const key = instanceKey(item);
+      if (mapped.has(key)) {
+        mapped.get(key).items.push(item);
+      } else {
+        unmapped.push(item);
+        if (verbose && !seenUnmappedKeys.has(key)) {
+          seenUnmappedKeys.add(key);
+          console.log(
+            `[plugin-aeo] llmsTxt: instance "${key}" is not in instanceSections; routing to "Other"`,
+          );
+        }
+      }
+    }
+
+    const sections = [];
+    const ordered = [...mapped.entries()]
+      .map(([key, v]) => ({
+        key,
+        title: v.title,
+        order: typeof v.order === 'number' ? v.order : Number.POSITIVE_INFINITY,
+        items: v.items,
+      }))
+      .sort((a, b) => {
+        if (a.order !== b.order) return a.order - b.order;
+        return a.title.localeCompare(b.title);
+      });
+
+    for (const s of ordered) {
+      if (s.items.length === 0) continue;
+      s.items.sort((a, b) => a.permalink.localeCompare(b.permalink));
+      sections.push({ key: s.key, title: s.title, items: s.items });
+    }
+
+    if (unmapped.length > 0) {
+      // Spec: unmapped items go in a single "Other" section, sorted by
+      // page title (alphabetical). Stable tiebreak on permalink.
+      unmapped.sort((a, b) => {
+        const at = a.title || a.permalink;
+        const bt = b.title || b.permalink;
+        const cmp = at.localeCompare(bt);
+        return cmp !== 0 ? cmp : a.permalink.localeCompare(b.permalink);
+      });
+      sections.push({ key: '__other__', title: 'Other', items: unmapped });
+    }
+
+    return sections;
+  }
+
+  // Default path - per-type grouping, identical to v0.1.x behavior.
+  const groups = { docs: [], blog: [], pages: [] };
+  for (const item of filtered) {
+    const g = groupKind(item.kind, item.pluginId);
+    groups[g].push(item);
+  }
+
+  const sections = [];
+  const order = [
+    ['docs', options.sections.docs],
+    ['blog', options.sections.blog],
+    ['pages', options.sections.pages],
+  ];
+  for (const [key, title] of order) {
+    const items = groups[key];
+    if (!items || items.length === 0) continue;
+    items.sort((a, b) => a.permalink.localeCompare(b.permalink));
+    sections.push({ key, title, items });
+  }
+  return sections;
+}
+
 module.exports = async function emitLlmsTxt({
   props,
   options,
@@ -73,12 +170,7 @@ module.exports = async function emitLlmsTxt({
     return true;
   });
 
-  // Group by kind.
-  const groups = { docs: [], blog: [], pages: [] };
-  for (const item of filtered) {
-    const g = groupKind(item.kind, item.pluginId);
-    groups[g].push(item);
-  }
+  const sections = buildSections(filtered, options, verbose);
 
   // Build llms.txt.
   const lines = [];
@@ -93,20 +185,10 @@ module.exports = async function emitLlmsTxt({
     lines.push('');
   }
 
-  const sectionOrder = [
-    ['docs', options.sections.docs],
-    ['blog', options.sections.blog],
-    ['pages', options.sections.pages],
-  ];
-
-  for (const [key, title] of sectionOrder) {
-    const items = groups[key];
-    if (!items || items.length === 0) continue;
-    lines.push(`## ${title}`);
+  for (const section of sections) {
+    lines.push(`## ${section.title}`);
     lines.push('');
-    // Stable order: by permalink.
-    items.sort((a, b) => a.permalink.localeCompare(b.permalink));
-    for (const item of items) {
+    for (const item of section.items) {
       const url = companionsEnabled
         ? companionUrl(item.permalink, trailingSlash)
         : item.permalink;
@@ -128,7 +210,8 @@ module.exports = async function emitLlmsTxt({
   }
 
   // Build llms-full.txt by concatenating the companion files that we just
-  // emitted in feature 1. Requires feature 1 to have been enabled.
+  // emitted in feature 1. Requires feature 1 to have been enabled. Section
+  // headings mirror llms.txt so an LLM can navigate the corpus by H2.
   if (options.fullTxt) {
     if (!companionsEnabled) {
       if (verbose) {
@@ -139,11 +222,10 @@ module.exports = async function emitLlmsTxt({
       return;
     }
 
-    const blocks = [];
-    for (const [key] of sectionOrder) {
-      const items = groups[key];
-      if (!items) continue;
-      for (const item of items) {
+    const sectionChunks = [];
+    for (const section of sections) {
+      const blocks = [];
+      for (const item of section.items) {
         let body;
         try {
           body = await fs.readFile(item.companionPath, 'utf8');
@@ -164,10 +246,12 @@ module.exports = async function emitLlmsTxt({
         ].join('\n');
         blocks.push(`${header}${body.trim()}\n`);
       }
+      if (blocks.length === 0) continue;
+      sectionChunks.push(`## ${section.title}\n\n${blocks.join('\n---\n\n')}`);
     }
 
     const fullPath = path.join(outDir, 'llms-full.txt');
-    await fs.writeFile(fullPath, blocks.join('\n---\n\n'), 'utf8');
+    await fs.writeFile(fullPath, sectionChunks.join('\n---\n\n'), 'utf8');
     if (verbose) {
       console.log(`[plugin-aeo] llmsTxt: wrote ${fullPath}`);
     }

package/src/index.js

@@ -41,6 +41,10 @@ function normalizeOptions(raw) {
         blog: llmsSections.blog || 'Blog',
         pages: llmsSections.pages || 'Pages',
       },
+      // Per-instance section map. Keys are "${pluginName}@${pluginId}".
+      // When set, supersedes `sections` and switches feature 2 from
+      // per-type grouping to per-instance grouping.
+      instanceSections: llmsTxt.instanceSections || null,
       fullTxt: llmsTxt.fullTxt !== false,
     },
     askAi: {
@@ -77,6 +81,36 @@ function validateOptions(opts) {
   if (opts.llmsTxt.header !== null && typeof opts.llmsTxt.header !== 'string') {
     errs.push('llmsTxt.header must be a string or null');
   }
+  if (opts.llmsTxt.instanceSections !== null) {
+    const is = opts.llmsTxt.instanceSections;
+    if (typeof is !== 'object' || Array.isArray(is)) {
+      errs.push(
+        'llmsTxt.instanceSections must be an object keyed by "${pluginName}@${pluginId}" or undefined',
+      );
+    } else {
+      for (const [key, value] of Object.entries(is)) {
+        if (!value || typeof value !== 'object' || Array.isArray(value)) {
+          errs.push(
+            `llmsTxt.instanceSections["${key}"] must be an object with { title, order? }`,
+          );
+          continue;
+        }
+        if (typeof value.title !== 'string' || value.title.length === 0) {
+          errs.push(
+            `llmsTxt.instanceSections["${key}"].title must be a non-empty string`,
+          );
+        }
+        if (
+          value.order !== undefined &&
+          (typeof value.order !== 'number' || !Number.isFinite(value.order))
+        ) {
+          errs.push(
+            `llmsTxt.instanceSections["${key}"].order must be a finite number when set`,
+          );
+        }
+      }
+    }
+  }
 
   if (!Array.isArray(opts.askAi.providerOrder)) {
     errs.push('askAi.providerOrder must be an array');