npm - poops - Versions diffs - 1.5.2 → 1.6.0 - Mend

poops 1.5.2 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (10) hide show

package/README.md +120 -7
package/lib/exec.js +29 -0
package/lib/markup/collections.js +2 -1
package/lib/markup/engines/liquid.js +5 -1
package/lib/markup/engines/nunjucks.js +5 -1
package/lib/markup/helpers.js +281 -0
package/lib/markup/indexer.js +370 -0
package/lib/markups.js +49 -5
package/package.json +1 -1
package/poops.js +23 -9

package/README.md CHANGED Viewed

@@ -45,7 +45,7 @@ It uses a simple config file where you define your input and output paths and it
       - [googleFonts](#googlefonts)
       - [highlight](#highlight)
     - [Custom Filters](#custom-filters)
-    - [Search Index, Sitemap & Navigation](#search-index-sitemap--navigation)
+    - [Search Index, Sitemap, llms.txt, robots.txt & Navigation](#search-index-sitemap-llmstxt-robotstxt--navigation)
   - [Images (optional)](#images-optional)
   - [Copy](#copy)
   - [Banner (optional)](#banner-optional)
@@ -703,7 +703,7 @@ sort: date
 - `url` - the item's output path relative to the site root (e.g. `changelog/my-post.html`)
 - `title` - falls back to the file name if not set in front matter
 - `date` - falls back to the file's modification time if not set, with a build warning. Set a real `date` in front matter — mtime is meaningless on CI checkouts (git clone resets it), so undated posts will reshuffle between deploys.
-- `wordcount`, `fileName`, `filePath`, `collection`
+- `wordcount`, `excerpt` (first paragraph, plain text — a meta-description fallback), `fileName`, `filePath`, `collection`
 An item with `published: false` in its front matter is excluded from the collection and its page is not built.
@@ -930,6 +930,54 @@ All filters are available in both engines. The only syntax difference is how arg
   - Nunjucks: `{{ someCodeVariable | highlight('javascript') }}`
   - Liquid: `{{ someCodeVariable | highlight: 'javascript' }}`
+- `og` — generates Open Graph (and a Twitter card) `<meta>` tags from a page's front matter and your `site` data, for link previews on social/chat platforms. Put it in your layout `<head>`. `og:type` auto-detects: `article` when the page has a `date`, otherwise `website`.
+  - Nunjucks: `{{ page | og(site) }}`
+  - Liquid: `{{ page | og: site }}`
+  Emits `og:title`, `og:description` (a missing `description` falls back to the page's auto-`excerpt`, then `site.description`), `og:type`, `og:url` (made absolute with `site.url`), `og:site_name` (from `site.title`), `og:locale` (`page.lang`/`site.lang`), `og:image` (`page.image`/`site.image`, made absolute), and `twitter:card` (`summary_large_image` when there's an image, else `summary`). For articles it adds `article:published_time`, `article:modified_time` and `article:author`. Attribute values are escaped. Set an `og` object in front matter to add or override any tag (e.g. `og:image:alt`, a fixed `twitter:card`):
+  ```yaml
+  ---
+  title: My post
+  date: 2026-01-01
+  image: static/cover.jpg
+  og:
+    "og:image:alt": Cover illustration
+  ---
+  ```
+- `canonical` — generates a `<link rel="canonical">` tag pointing at a page's authoritative absolute URL (`site.url` + the page's `url`), the dedup signal that stops query-string and duplicate URLs splitting your ranking. Put it in your layout `<head>`. Front matter `canonical` overrides the target — an absolute URL as-is, or a path resolved against `site.url` (for cross-domain or hand-picked canonicals). The homepage canonicals to the site root. Returns nothing without `site.url`.
+  - Nunjucks: `{{ page | canonical(site) }}`
+  - Liquid: `{{ page | canonical: site }}`
+- `jsonld` — generates a schema.org JSON-LD `<script type="application/ld+json">` block from a page's front matter and your `site` data, for GEO (Generative Engine Optimization) and structured data. Put it in your layout `<head>`. The `@type` auto-detects: `BlogPosting` when the page has a `date`, otherwise `WebPage`.
+  - Nunjucks: `{{ page | jsonld(site) }}`
+  - Liquid: `{{ page | jsonld: site }}`
+  It reads these front-matter fields when present: `title`, `description` (falls back to the page's auto-`excerpt`, then `site.description`), `url` (made absolute with `site.url`), `date` → `datePublished`, `updated` → `dateModified`, `author` (string or `{ name }`, falls back to `site.author`), `image`, `lang` → `inLanguage`, and `wordcount`. `publisher` comes from `site.title`; set `site.logo` to add a `publisher.logo` ImageObject (made absolute) — Google Article rich results require it. Front-matter values are escaped so they can't break out of the `<script>` tag.
+  On the homepage (a page with no `url`) it also emits a site-level `WebSite` block with `name` + `url`, which declares the site name for search results. On nested pages (a `url` with at least one folder) it auto-appends a `BreadcrumbList` block derived from URL depth — a Google breadcrumb rich result, no extra markup (needs `site.url` for the absolute item URLs). See the `breadcrumb` filter below for a visible trail from the same data.
+  For full control, set a `jsonld` object in front matter — its keys are merged over (and override) the generated defaults, including `@type`:
+  ```yaml
+  ---
+  title: How to brew coffee
+  date: 2026-01-01
+  jsonld:
+    "@type": HowTo
+    totalTime: PT5M
+  ---
+  ```
+  poops auto-picks `BlogPosting` (page has a `date`) or `WebPage`. Override `@type` with the `jsonld` object for any schema.org type — common ones search/generative engines act on: `Article`, `NewsArticle`, `HowTo`, `FAQPage`, `QAPage`, `Product`, `Recipe`, `Event`, `Course`, `VideoObject`, `SoftwareApplication`, `Organization`, `Person`, `BreadcrumbList`, `WebSite`. Full list at [schema.org/docs/full](https://schema.org/docs/full.html); validate with the [Rich Results Test](https://search.google.com/test/rich-results). A per-`@type` table with the notable fields is in the [Templating docs](example/src/markup/docs/quick-start/templating-html.md).
+- `breadcrumb` — generates a visible breadcrumb `<nav class="breadcrumb"><ol>…</ol></nav>` trail for the page body (blog posts, nested pages), from the same URL-depth data the `jsonld` `BreadcrumbList` uses: the site root, each ancestor folder (humanized, e.g. `docs/static-site` → *Static Site*), then the current page as `aria-current` text. Pass `relativePathPrefix` so links resolve against the current output location (localhost in dev, your deployed subpath in prod) — not the absolute domain.
+  - Nunjucks: `{{ page | breadcrumb(site, relativePathPrefix) }}`
+  - Liquid: `{{ page | breadcrumb: site, relativePathPrefix }}`
+  The home crumb is optional: set `breadcrumb: { home: false }` (or `{ homeLabel: "Start" }` to rename it) under `site` or in a page's front matter — front matter wins. With the home crumb off, top-level pages fall to a single crumb and render nothing, while nested pages still show their folder trail. `breadcrumb: false` on a page or on `site` disables both the visible trail and the auto `BreadcrumbList` JSON-LD. Returns nothing on the homepage or any single-crumb page.
 - `groupby` — groups an array of objects by a field value. Returns an array of `{ key, items }` objects. Supports an optional second argument for date part extraction (`year`, `month`, `day`). Groups preserve insertion order, so if items are sorted by date descending, groups will be too.
   - Nunjucks: `{{ changelog.items | groupby("author") }}` or `{{ changelog.items | groupby("date", "year") }}`
   - Liquid: `{{ changelog.items | groupby: "author" }}` or `{{ changelog.items | groupby: "date", "year" }}`
@@ -1026,11 +1074,11 @@ Returns: `static/photo-320w.webp 320w, static/photo-640w.webp 640w, static/photo
   {% endfor %}
   ```
-#### Search Index, Sitemap & Navigation
+#### Search Index, Sitemap, llms.txt, robots.txt & Navigation
-Poops can automatically generate a JSON search index, an XML sitemap and a navigation tree from your compiled pages. All are generated in a single pass during the markup compilation phase.
+Poops can automatically generate a JSON search index, an XML sitemap, an `llms.txt`, a `robots.txt` and a navigation tree from your compiled pages. All are generated in a single pass during the markup compilation phase.
-To enable, add `searchIndex`, `sitemap` and/or `nav` to your markup config:
+To enable, add `searchIndex`, `sitemap`, `llms`, `robots` and/or `nav` to your markup config:
 ```json
 {
@@ -1039,7 +1087,9 @@ To enable, add `searchIndex`, `sitemap` and/or `nav` to your markup config:
     "out": "dist",
     "options": {
       "searchIndex": "search-index.json",
-      "sitemap": "sitemap.xml"
+      "sitemap": "sitemap.xml",
+      "llms": "llms.txt",
+      "robots": "robots.txt"
     }
   }
 }
@@ -1058,6 +1108,13 @@ The string shorthand sets the output filename with default options. For more con
   },
   "sitemap": {
     "output": "sitemap.xml"
+  },
+  "llms": {
+    "output": "llms.txt",
+    "title": "My Site",
+    "description": "One-line summary of the site.",
+    "intro": "src/llms-intro.md",
+    "full": true
   }
 }
 ```
@@ -1094,7 +1151,29 @@ All front matter fields are passed through to the index automatically. Internal
 **Sitemap** generates a standard `sitemap.xml` with `<loc>` and `<lastmod>` (from front matter `date`). If `site.url` is set in your markup config, it is prepended to all URLs. Collection index/pagination pages are included in the sitemap but excluded from the search index.
-Pages with `published: false` in their front matter are excluded from both outputs.
+**llms.txt** generates an [`llms.txt`](https://llmstxt.org) — a Markdown index of your pages that LLMs and generative engines (GEO) read to understand your site. It has an `# H1` title, a `> ` blockquote summary, then `- [title](url): description` links grouped by URL path: the first folder is a `## section`, a second folder nests as a `### subsection` under it, and root-level pages fall under a lead "Pages" section. So `docs/config-reference.html` lands directly under `## Docs` while `docs/quick-start/x.html` lands under `### Quick Start` inside it. Collection items (which live under `collection/…`) group the same way and are ordered newest-first by their `date`; other sections keep file order. Set `intro` to a Markdown file path (relative to the project root) to insert free-form context between the blockquote and the link sections — a file authored for LLMs, e.g. `llms-intro.md`. Avoid `##` headings in it; they read as sections. (A raw README is a poor fit — badges, install noise and its own headings collide.) `title` and `description` default to your `site.title`/`site.description`; override them (and the lead section name via `sectionTitle`) with the object form. `site.url` makes the links absolute. Collection index/pagination pages are skipped, like the search index.
+Set `full` to also write a companion full-content file — every page's Markdown body concatenated into one file an LLM can ingest whole (the index is the link map; this is the corpus). `true` names it after `output` with a `-full` suffix (`llms.txt` → `llms-full.txt`, `ai.txt` → `ai-full.txt`); pass a string to set the path yourself. The file opens with a `# Full Documentation Archive for {title}` header, a one-line intro naming the site and a `> ` blockquote of the `description` so a whole-file ingest starts with context, then each page becomes an `# title` (its own leading H1 if it has one) + `URL:` line + body, joined by `---`. Set `fullIntro` to a Markdown file path (from the project root) to insert your own preamble after that header — the `full` counterpart to `intro` (inserted verbatim; a missing file warns and is skipped). Only `.md`/`.markdown` sources qualify (a `.njk`/`.liquid` source is template code, not prose); `noindex` and collection-index pages are dropped. Content is the Markdown **source**, so unrendered `{% raw %}{% … %}{% endraw %}` tags or shortcodes in a body pass through verbatim.
+**robots.txt** generates a `robots.txt`. The string shorthand writes an allow-all file (`User-agent: *`, empty `Disallow:`) with a `Sitemap:` line pointing at your generated sitemap — absolute when `site.url` is set. The object form takes `output`, `userAgent`, `allow`/`disallow` (a path or array of paths), and `sitemap` (an explicit URL, or `false` to omit the line):
+```json
+{
+  "robots": {
+    "output": "robots.txt",
+    "disallow": ["/admin", "/drafts"],
+    "sitemap": false
+  }
+}
+```
+Pages with `published: false` in their front matter are excluded from all outputs.
+A page's front matter `robots: noindex` (or `none`) drops it from the **sitemap and llms.txt** — for drafts, thin or utility pages (a 404, say) you don't want crawled or fed to LLMs. It stays in the search index (that's your own on-site search). Emit the matching crawler directive in your layout `<head>` so the page itself carries it:
+```html
+{% if page.robots %}<meta name="robots" content="{{ page.robots }}">{% endif %}
+```
 **Navigation tree** builds your page hierarchy as sidebar-ready data, exposed two ways: as the `nav` template global (loaded automatically, always reflecting the current build) and as a nested JSON file for client-side rendering. Subpages nest automatically from URL structure: `guide/index.md` becomes a parent node and `guide/getting-started.md`, `guide/advanced/config.md` become its (and its subsections') children. Add `nav` to your markup config:
@@ -1206,6 +1285,40 @@ Liquid — a partial that recurses via `render` (save as `_partials/navtree.liqu
 {% render 'navtree', items: nav, relativePathPrefix: relativePathPrefix %}
 ```
+#### RSS / Atom feeds
+Generate a subscription feed for a [collection](#collections) — no hand-authored feed template. Each feed lists the collection's posts newest-first (by `date`), with the channel metadata taken from your `site` data.
+```json
+{
+  "markup": {
+    "in": "src/markup",
+    "out": "dist",
+    "options": {
+      "feed": { "collection": "blog", "output": "blog/feed.rss" }
+    }
+  }
+}
+```
+**Feed options:**
+- `collection` — the collection to build the feed from. Omit it to emit a feed for **every** collection.
+- `output` — the file to write. A bare filename (`feed.xml`, the default) is placed inside the collection's own folder (`blog/feed.xml`); a value with a slash is used as-is under the output directory.
+- `type` — `"rss"` (default) or `"atom"`.
+- `limit` — max items, newest first (default `20`).
+- `title` — channel title (default `"<Collection> | <site.title>"`).
+- `description` — channel description (default `site.description`).
+- `author`, `lang` — default to `site.author` / `site.lang`.
+Shorthand forms: `"feed": true` (or a filename string) emits an RSS feed for every collection; an array of the objects above generates several feeds at once (e.g. an RSS and an Atom for the same collection). Item `<description>` uses each post's `description`, falling back to its auto-`excerpt`; links, `guid`s and `<atom:link rel="self">` are made absolute with `site.url`. `robots: noindex` posts are excluded, matching the sitemap.
+Point browsers and readers at it from your layout `<head>`:
+```html
+<link rel="alternate" type="application/rss+xml" href="{{ site.url }}/blog/feed.rss">
+```
 ### Images (optional)
 Process and optimize images — compression, responsive size variants, format conversion (WebP/AVIF), crops and EXIF extraction — by running [poops-images](https://github.com/stamat/poops-images) as part of the build. This is what feeds the `{% image %}` tag, the `exif`/`images` filters and the `.poops-images-cache.json` compile cache described in [Custom Tags](#custom-tags) and [Custom Filters](#custom-filters).

package/lib/exec.js ADDED Viewed

@@ -0,0 +1,29 @@
+import { execSync } from 'node:child_process'
+import log, { styledLog } from './utils/log.js'
+// Post-stage shell hooks. `config.exec` maps a pipeline stage to a command (or
+// array of commands) run after that stage compiles — in both build and watch,
+// so a post-processor (e.g. stripping CSS comments, regenerating a reference)
+// stays in sync during dev instead of only running behind `poops -b && cmd`.
+// Sync + stdio:inherit so ordering is deterministic and output streams live;
+// cwd is the project root, matching how the compilers resolve their paths.
+// Returns true if any command failed, so build can fail the exit code; in watch
+// the failure is logged and swallowed so the watcher survives.
+// Stages: reactor, scripts, images, markup, styles (fires after PostCSS, so the
+// CSS is final), copy, and build (once, after the full initial pipeline).
+// `run` is injectable so tests exercise the branching without forking a shell.
+export default function runExec(config, cwd, stage, run = execSync) {
+  const hooks = config.exec && config.exec[stage]
+  if (!hooks) return false
+  let failed = false
+  for (const cmd of [hooks].flat()) {
+    try {
+      styledLog(`⚡ {dim}exec ${stage}:{/} ${cmd}`)
+      run(cmd, { cwd, stdio: 'inherit' })
+    } catch (err) {
+      failed = true
+      log({ tag: 'error', text: `exec ${stage} failed: ${cmd}` })
+    }
+  }
+  return failed
+}

package/lib/markup/collections.js CHANGED Viewed

@@ -3,7 +3,7 @@ import { globSync } from 'glob'
 import path from 'node:path'
 import log from '../utils/log.js'
 import { mkDir, toPosix } from '../utils/helpers.js'
-import { replaceOutExtensions, getRelativePathPrefix, getPageUrlRelativeToOutput, parseFrontMatter, wordcount } from './helpers.js'
+import { replaceOutExtensions, getRelativePathPrefix, getPageUrlRelativeToOutput, parseFrontMatter, wordcount, excerpt } from './helpers.js'
 export function getSingleCollectionData(markupInDir, collectionName) {
   const collectionData = []
@@ -30,6 +30,7 @@ export function getSingleCollectionData(markupInDir, collectionName) {
       log({ tag: 'markup', warn: true, text: 'No date in front matter, falling back to file mtime:', link: file })
     }
     frontMatter.wordcount = wordcount(content)
+    frontMatter.excerpt = excerpt(content)
     frontMatter.fileName = path.basename(file)
     frontMatter.filePath = path.relative(process.cwd(), file)
     frontMatter.collection = collectionName

package/lib/markup/engines/liquid.js CHANGED Viewed

@@ -3,7 +3,7 @@ import path from 'node:path'
 import { Liquid } from 'liquidjs'
 import { highlightCode } from '../highlight.js'
 import { marked, renderMarkdownCached } from '../renderer.js'
-import { discoverImageVariants, parseFrontMatter, groupby, buildImageTag, buildPaginationTag, renderToc } from '../helpers.js'
+import { discoverImageVariants, parseFrontMatter, groupby, buildImageTag, buildPaginationTag, renderToc, buildJsonLd, buildOpenGraph, buildCanonical, buildBreadcrumb } from '../helpers.js'
 import { getImageExif, listImages } from '../image-cache.js'
 import { slugify } from 'book-of-spells'
 import dayjs from 'dayjs'
@@ -115,6 +115,10 @@ export default class LiquidEngine {
       const outputDir = path.resolve(process.cwd(), markupOut)
       return listImages(dirPath, outputDir)
     })
+    engine.registerFilter('jsonld', (page, site) => buildJsonLd(page, site))
+    engine.registerFilter('og', (page, site) => buildOpenGraph(page, site))
+    engine.registerFilter('canonical', (page, site) => buildCanonical(page, site))
+    engine.registerFilter('breadcrumb', (page, site, prefix) => buildBreadcrumb(page, site, prefix))
     engine.registerFilter('groupby', (arr, key, datePart) => groupby(arr, key, datePart))
     engine.registerFilter('highlight', (code, lang) => {
       const highlighted = highlightCode(code, lang)

package/lib/markup/engines/nunjucks.js CHANGED Viewed

@@ -2,7 +2,7 @@ import fs from 'node:fs'
 import { globSync } from 'glob'
 import nunjucks from 'nunjucks'
 import path from 'node:path'
-import { discoverImageVariants, parseFrontMatter, groupby, buildImageTag, buildPaginationTag, renderToc } from '../helpers.js'
+import { discoverImageVariants, parseFrontMatter, groupby, buildImageTag, buildPaginationTag, renderToc, buildJsonLd, buildOpenGraph, buildCanonical, buildBreadcrumb } from '../helpers.js'
 import { getImageExif, listImages } from '../image-cache.js'
 import { toPosix } from '../../utils/helpers.js'
 import { highlightCode } from '../highlight.js'
@@ -177,6 +177,10 @@ export default class NunjucksEngine {
       const outputDir = path.resolve(process.cwd(), markupOut)
       return listImages(dirPath, outputDir)
     })
+    env.addFilter('jsonld', (page, site) => new nunjucks.runtime.SafeString(buildJsonLd(page, site)))
+    env.addFilter('og', (page, site) => new nunjucks.runtime.SafeString(buildOpenGraph(page, site)))
+    env.addFilter('canonical', (page, site) => new nunjucks.runtime.SafeString(buildCanonical(page, site)))
+    env.addFilter('breadcrumb', (page, site, prefix) => new nunjucks.runtime.SafeString(buildBreadcrumb(page, site, prefix)))
     env.addFilter('groupby', (arr, key, datePart) => groupby(arr, key, datePart))
     env.addFilter('highlight', (code, lang) => {
       const highlighted = highlightCode(code, lang)

package/lib/markup/helpers.js CHANGED Viewed

@@ -68,6 +68,26 @@ export function wordcount(text) {
   return words ? words.length : 0
 }
+// First prose paragraph of content as plain text, for a meta-description
+// fallback when front matter has none. Splits on blank lines, skips leading
+// headings and HTML comments, strips markdown/HTML like wordcount, then caps at
+// ~160 chars (search-snippet size) on a word boundary.
+// ponytail: content-level split; a leading code fence or raw template block
+// could still slip in — refine the skip list if a snippet looks wrong.
+export function excerpt(text, max = 160) {
+  if (!text) return ''
+  for (const block of text.split(/\n\s*\n/)) {
+    const trimmed = block.trim()
+    if (!trimmed || /^#{1,6}\s/.test(trimmed) || trimmed.startsWith('<!--')) continue
+    // eslint-disable-next-line no-useless-escape
+    const stripped = trimmed.replace(/<[^>]*>/g, ' ').replace(/[#*_`~\[\]()>|{}\\]/g, ' ').replace(/\s+/g, ' ').trim()
+    if (!stripped) continue
+    if (stripped.length <= max) return stripped
+    return stripped.slice(0, max).replace(/\s+\S*$/, '').trim() + '…'
+  }
+  return ''
+}
 // marked HTML-encodes quotes/brackets in template tags it treats as prose,
 // breaking string args like groupby("date"). Decode entities inside {{ }} / {% %}
 // after markdown, before the template engine parses. Decode &amp; last so
@@ -371,3 +391,264 @@ export function buildImageTag(imagePath, prefix, kwargs, getOutputDir) {
   }
   return `<img ${attrs.join(' ')}>`
 }
+// Serializes a JSON-LD data object into a <script> block. Drops empty top-level
+// keys (GEO parsers ignore them) and neutralizes `<`, `>`, `&` so front-matter
+// values can't break out of the <script> or inject entities.
+function jsonLdScript(data) {
+  for (const k of Object.keys(data)) {
+    if (data[k] === undefined || data[k] === null || data[k] === '') delete data[k]
+  }
+  const json = JSON.stringify(data).replace(/</g, '\\u003c').replace(/>/g, '\\u003e').replace(/&/g, '\\u0026')
+  return `<script type="application/ld+json">${json}</script>`
+}
+// Turns a URL segment (or a filename) into a human title for a breadcrumb crumb
+// that has no page object of its own — same rule the nav tree uses for virtual
+// parents. Strips a trailing extension, then dash/underscore → spaced Title Case.
+function humanizeSegment(seg) {
+  return String(seg)
+    .replace(/\.[a-z0-9]+$/i, '')
+    .replace(/[-_]+/g, ' ')
+    .replace(/\s+/g, ' ')
+    .trim()
+    .replace(/\b\w/g, c => c.toUpperCase())
+}
+// Builds a breadcrumb trail — a { name, path } array from the site root down to
+// the page — derived purely from URL depth (no nav tree needed). Intermediate
+// path segments are humanized (they have no page object); the last crumb is the
+// page's own title. The home crumb (site root) leads unless disabled.
+// `path` is the site-root-relative URL (page.url form: '' for home, no domain);
+// each consumer prefixes it — the JSON-LD with `site.url` (absolute, required by
+// Google), the visible filter with `relativePathPrefix` (so local-dev links stay
+// on localhost instead of jumping to the production domain).
+// Config: `site.breadcrumb` overlaid by front-matter `page.breadcrumb` (object
+// with `home` boolean / `homeLabel` string); `breadcrumb: false` on either
+// disables it. Returns [] for the homepage, a single-crumb trail, or when
+// disabled — every caller renders nothing on [].
+export function breadcrumbCrumbs(page, site = {}) {
+  page = page || {}
+  site = site || {}
+  if (page.breadcrumb === false || site.breadcrumb === false) return []
+  const cfg = { home: true, homeLabel: 'Home' }
+  if (site.breadcrumb && typeof site.breadcrumb === 'object') Object.assign(cfg, site.breadcrumb)
+  if (page.breadcrumb && typeof page.breadcrumb === 'object') Object.assign(cfg, page.breadcrumb)
+  const rawUrl = page.url != null ? String(page.url) : ''
+  if (rawUrl === '') return [] // homepage — you're already home
+  const parts = rawUrl.split('/').filter(Boolean)
+  const crumbs = []
+  if (cfg.home) crumbs.push({ name: cfg.homeLabel, path: '' })
+  // Every part but the last is an ancestor directory; accumulate its path.
+  let acc = ''
+  for (let i = 0; i < parts.length - 1; i++) {
+    acc = acc ? `${acc}/${parts[i]}` : parts[i]
+    crumbs.push({ name: humanizeSegment(parts[i]), path: acc })
+  }
+  crumbs.push({ name: page.title || humanizeSegment(parts[parts.length - 1]), path: rawUrl })
+  // A lone crumb (e.g. home off, top-level page) is not a trail.
+  return crumbs.length >= 2 ? crumbs : []
+}
+// BreadcrumbList JSON-LD <script> for a page's position in the site hierarchy —
+// a Google rich result. Absolute item URLs are mandatory, so it's skipped
+// without `site.url` (like canonical). Names come from front matter / segments,
+// so jsonLdScript escapes them for the <script> context.
+function breadcrumbJsonLd(page, site) {
+  if (!(site && site.url)) return ''
+  const crumbs = breadcrumbCrumbs(page, site)
+  if (crumbs.length < 2) return ''
+  const baseUrl = String(site.url).replace(/\/+$/, '')
+  const abs = (p) => {
+    const rel = String(p).replace(/^\/+/, '')
+    return rel ? `${baseUrl}/${rel}` : baseUrl
+  }
+  return jsonLdScript({
+    '@context': 'https://schema.org',
+    '@type': 'BreadcrumbList',
+    itemListElement: crumbs.map((c, i) => ({
+      '@type': 'ListItem',
+      position: i + 1,
+      name: c.name,
+      item: abs(c.path)
+    }))
+  })
+}
+// Builds a visible <nav> breadcrumb trail for display in the page body (blog
+// posts, nested pages). Same crumbs as the JSON-LD, but hrefs are relative:
+// `prefix` is the page's `relativePathPrefix`, so links resolve against the
+// current output location (localhost in dev, the deployed path in prod) exactly
+// like the nav/header links — never the absolute production domain. The last
+// crumb renders as aria-current text (the current page is not a link). Returns
+// '' when there's no trail. Names/paths land in attributes, so they're escaped.
+export function buildBreadcrumb(page, site = {}, prefix = '') {
+  const crumbs = breadcrumbCrumbs(page, site)
+  if (crumbs.length < 2) return ''
+  const items = crumbs.map((c, i) => {
+    const name = escapeAttr(c.name)
+    if (i === crumbs.length - 1) return `<li><span aria-current="page">${name}</span></li>`
+    return `<li><a href="${escapeAttr(prefix + c.path)}">${name}</a></li>`
+  })
+  return `<nav class="breadcrumb" aria-label="Breadcrumb"><ol>${items.join('')}</ol></nav>`
+}
+// Builds a schema.org JSON-LD <script> block from a page's front matter + site
+// data, for GEO / structured data (generative engines lean on schema.org to
+// understand a page). @type auto-detects — BlogPosting when the page has a
+// `date`, else WebPage. Article-only fields (headline, dates, author,
+// wordCount) are added only for the article case. The publisher Organization
+// gains a `logo` ImageObject when `site.logo` is set — Google Article rich
+// results require it. On the homepage (page has no `url`) a second `WebSite`
+// block is emitted, declaring the site name for search results.
+// Anything in `page.jsonld` (an object) is merged over the generated defaults,
+// so a page can add or replace any field — including @type — as a full escape
+// hatch. Values come from front matter, so the JSON is escaped for a <script>
+// context to prevent a `</script>` break-out (XSS).
+export function buildJsonLd(page, site = {}) {
+  page = page || {}
+  site = site || {}
+  const baseUrl = site.url ? String(site.url).replace(/\/+$/, '') : ''
+  const absUrl = (u) => {
+    if (!u) return undefined
+    if (/^https?:\/\//i.test(u)) return u
+    return baseUrl ? `${baseUrl}/${String(u).replace(/^\/+/, '')}` : u
+  }
+  const author = (page.author && typeof page.author === 'object' ? page.author.name : page.author) || site.author
+  const publisherName = site.title || site.name
+  let publisher
+  if (publisherName) {
+    publisher = { '@type': 'Organization', name: publisherName }
+    const logo = absUrl(site.logo)
+    if (logo) publisher.logo = { '@type': 'ImageObject', url: logo }
+  }
+  const isArticle = !!page.date
+  const data = {
+    '@context': 'https://schema.org',
+    '@type': isArticle ? 'BlogPosting' : 'WebPage',
+    name: page.title,
+    description: page.description || page.excerpt || site.description,
+    url: absUrl(page.url),
+    inLanguage: page.lang || site.lang,
+    image: absUrl(page.image),
+    publisher
+  }
+  if (isArticle) {
+    data.headline = page.title
+    data.datePublished = page.date
+    data.dateModified = page.updated || page.date
+    if (author) data.author = { '@type': 'Person', name: author }
+    if (page.wordcount) data.wordCount = page.wordcount
+  }
+  // page.jsonld extends/overrides. Shallow merge — nested schema is rare and
+  // the escape hatch is meant for whole-key replacement.
+  if (page.jsonld && typeof page.jsonld === 'object') Object.assign(data, page.jsonld)
+  const blocks = [jsonLdScript(data)]
+  // Homepage (no page.url) gets a site-level WebSite block — defines the site
+  // name for search results.
+  if (!page.url && baseUrl) {
+    blocks.push(jsonLdScript({
+      '@context': 'https://schema.org',
+      '@type': 'WebSite',
+      name: publisherName,
+      url: baseUrl,
+      inLanguage: page.lang || site.lang
+    }))
+  }
+  // Nested pages get a BreadcrumbList block (zero-config rich result).
+  const breadcrumb = breadcrumbJsonLd(page, site)
+  if (breadcrumb) blocks.push(breadcrumb)
+  return blocks.join('\n')
+}
+// Builds Open Graph (+ a Twitter card) <meta> tags from a page's front matter
+// and site data, for link previews on social/chat platforms. og:type is
+// `article` when the page has a `date`, else `website`. Twitter tags fall back
+// to the og:* values on most crawlers, so only `twitter:card` is emitted. A
+// `page.og` object merges over (and overrides) the defaults — set any extra
+// property there, e.g. `og:image:alt` or a fixed `twitter:card`. Values come
+// from front matter and land in attributes, so they're escaped (escapeAttr).
+export function buildOpenGraph(page, site = {}) {
+  page = page || {}
+  site = site || {}
+  const baseUrl = site.url ? String(site.url).replace(/\/+$/, '') : ''
+  const absUrl = (u) => {
+    if (!u) return undefined
+    if (/^https?:\/\//i.test(u)) return u
+    return baseUrl ? `${baseUrl}/${String(u).replace(/^\/+/, '')}` : u
+  }
+  const author = (page.author && typeof page.author === 'object' ? page.author.name : page.author) || site.author
+  const isArticle = !!page.date
+  const image = absUrl(page.image || site.image)
+  const tags = {
+    'og:title': page.title || site.title,
+    'og:description': page.description || page.excerpt || site.description,
+    'og:type': isArticle ? 'article' : 'website',
+    'og:url': absUrl(page.url),
+    'og:site_name': site.title,
+    'og:locale': page.lang || site.lang,
+    'og:image': image,
+    'twitter:card': image ? 'summary_large_image' : 'summary'
+  }
+  if (isArticle) {
+    tags['article:published_time'] = page.date
+    tags['article:modified_time'] = page.updated || page.date
+    if (author) tags['article:author'] = author
+  }
+  // page.og extends/overrides. Shallow — one flat property:content map.
+  if (page.og && typeof page.og === 'object') Object.assign(tags, page.og)
+  const lines = []
+  for (const [prop, val] of Object.entries(tags)) {
+    if (val === undefined || val === null || val === '') continue
+    // Twitter's spec uses name=, Open Graph uses property=.
+    const attr = prop.startsWith('twitter:') ? 'name' : 'property'
+    lines.push(`<meta ${attr}="${escapeAttr(prop)}" content="${escapeAttr(val)}">`)
+  }
+  return lines.join('\n')
+}
+// Builds a `<link rel="canonical">` tag — the dedup signal that names a page's
+// authoritative URL. The href is absolute (needs `site.url`): front matter
+// `canonical` wins (an absolute URL as-is, or a path resolved against
+// `site.url`), else the page's own `url`. Homepage (`url` === '') canonicals to
+// the site root. Returns '' when no absolute URL can be formed (no site.url).
+export function buildCanonical(page, site = {}) {
+  page = page || {}
+  site = site || {}
+  const baseUrl = site.url ? String(site.url).replace(/\/+$/, '') : ''
+  const abs = (u) => {
+    if (u == null) return ''
+    if (/^https?:\/\//i.test(u)) return u
+    if (!baseUrl) return ''
+    const rel = String(u).replace(/^\/+/, '')
+    return rel ? `${baseUrl}/${rel}` : baseUrl
+  }
+  // The root index page has no `url` (poops leaves it unset), so a nullish url
+  // means the homepage → canonical to the site root.
+  const target = page.canonical != null ? page.canonical : (page.url != null ? page.url : '')
+  const href = abs(target)
+  return href ? `<link rel="canonical" href="${escapeAttr(href)}">` : ''
+}

package/lib/markup/indexer.js CHANGED Viewed

@@ -2,6 +2,8 @@ import fs from 'node:fs'
 import path from 'node:path'
 import { fileURLToPath } from 'node:url'
 import { readJsonFile, fileSize } from '../utils/helpers.js'
+import { parseFrontMatter } from './helpers.js'
+import { renderMarkdownCached } from './renderer.js'
 import log from '../utils/log.js'
 const __dirname = path.dirname(fileURLToPath(import.meta.url))
@@ -37,6 +39,12 @@ function normalizeConfig(config) {
   return { ...DEFAULTS, ...config }
 }
+// A page front-matter `robots: noindex` (or `none`) opts it out of the
+// crawler/discovery artifacts — the sitemap and llms.txt.
+function isNoindex(entry) {
+  return entry.robots ? /\b(noindex|none)\b/i.test(String(entry.robots)) : false
+}
 export function extractKeywords(htmlContent, options = {}) {
   const { minWordLength = DEFAULTS.minWordLength, stopWords = new Set() } = options
@@ -163,6 +171,7 @@ export function generateSitemap(pageEntries, outputDir, siteUrl, config) {
   xml += '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n'
   for (const entry of pageEntries) {
+    if (isNoindex(entry)) continue
     const loc = baseUrl ? `${baseUrl}/${entry.url}` : entry.url
     xml += '  <url>\n'
     xml += `    <loc>${escapeXml(loc)}</loc>\n`
@@ -182,6 +191,187 @@ export function generateSitemap(pageEntries, outputDir, siteUrl, config) {
   log({ tag: 'indexer', text: 'Generated sitemap:', link: path.relative(process.cwd(), outputPath), size: fileSize(outputPath) })
 }
+// Generates an llms.txt (https://llmstxt.org): a markdown index of the site's
+// pages that LLMs / generative engines ingest to understand the site (GEO).
+// Structure: `# title`, a `> description` blockquote, then one `## section` per
+// collection (uncollected pages go under a lead "Pages" section) with
+// `- [title](absolute url): description` links. `title`/`description` are
+// resolved from site data by the caller. Mirrors generateSitemap's page set —
+// isIndex (collection landing/pagination) pages are skipped.
+export function generateLlmsTxt(pageEntries, outputDir, siteUrl, config) {
+  config = normalizeConfig(config)
+  if (!config) return
+  const baseUrl = siteUrl ? siteUrl.replace(/\/+$/, '') : ''
+  const absUrl = (url) => baseUrl ? `${baseUrl}/${url}` : url
+  let out = `# ${config.title || 'Site'}\n`
+  if (config.description) out += `\n> ${config.description}\n`
+  // Optional free-form body (llmstxt.org allows markdown between the blockquote
+  // and the link sections). `intro` is a path to a markdown file authored for
+  // LLM context — inserted verbatim. Avoid H2s in it; they'd read as sections.
+  if (config.intro) {
+    try {
+      const body = fs.readFileSync(path.resolve(process.cwd(), config.intro), 'utf-8').trim()
+      if (body) out += `\n${body}\n`
+    } catch {
+      log({ tag: 'indexer', warn: true, text: 'llms.txt intro file not found:', link: config.intro })
+    }
+  }
+  // Group into two levels from the URL path, first-seen order preserved: first
+  // folder = `## section`, second folder = `### subsection` nested under it,
+  // root-level pages under the lead section. A collection's items already live
+  // under `collectionName/…`, so the folder segments double as its grouping.
+  const lead = config.sectionTitle || 'Pages'
+  const sections = new Map() // name -> { direct: [entry], subs: Map<name, [entry]> }
+  const sectionOf = (name) => {
+    if (!sections.has(name)) sections.set(name, { direct: [], subs: new Map() })
+    return sections.get(name)
+  }
+  for (const e of pageEntries) {
+    if (e.isIndex || isNoindex(e)) continue
+    const slash = e.url.lastIndexOf('/')
+    const segs = slash === -1 ? [] : e.url.slice(0, slash).split('/').filter(Boolean)
+    if (segs.length === 0) {
+      sectionOf(lead).direct.push(e)
+    } else if (segs.length === 1) {
+      sectionOf(humanizeSegment(segs[0])).direct.push(e)
+    } else {
+      const subs = sectionOf(humanizeSegment(segs[0])).subs
+      const key = humanizeSegment(segs[1])
+      if (!subs.has(key)) subs.set(key, [])
+      subs.get(key).push(e)
+    }
+  }
+  const linkLine = (e) => {
+    const link = `[${e.title || e.url}](${absUrl(e.url)})`
+    return e.description ? `- ${link}: ${e.description}\n` : `- ${link}\n`
+  }
+  // Collection buckets are chronological (newest first); other sections (docs)
+  // keep their file/order sequence.
+  const ordered = (entries) => {
+    if (!entries.length || !entries.every(e => e.collection != null)) return entries
+    return [...entries].sort((a, b) => new Date(b.date || 0) - new Date(a.date || 0))
+  }
+  for (const [name, group] of sections) {
+    out += `\n## ${name}\n\n`
+    for (const e of ordered(group.direct)) out += linkLine(e)
+    for (const [subName, entries] of group.subs) {
+      out += `\n### ${subName}\n\n`
+      for (const e of ordered(entries)) out += linkLine(e)
+    }
+  }
+  const outputPath = path.resolve(process.cwd(), outputDir, config.output)
+  fs.writeFileSync(outputPath, out)
+  log({ tag: 'indexer', text: 'Generated llms.txt:', link: path.relative(process.cwd(), outputPath), size: fileSize(outputPath) })
+}
+// Generates llms-full.txt: every page's full content concatenated, the
+// companion to llms.txt's link index (llmstxt.org). Enabled by `llms.full`
+// (string path, or `true` → the index filename with a `-full` suffix). The content comes from each
+// page's markdown SOURCE (entry._src), not entry.content — that's the rendered
+// HTML page (layout + chrome). Only markdown sources qualify: an .njk/.liquid
+// source is template code, not prose.
+// ponytail: raw markdown, so unrendered {% %} tags / shortcodes in a body leak
+// through verbatim. Upgrade path if that bites: render + HTML→markdown the body.
+export function generateLlmsFull(pageEntries, outputDir, siteUrl, config) {
+  config = normalizeConfig(config)
+  if (!config || !config.full) return
+  // Default filename pairs with the index: llms.txt → llms-full.txt, ai.txt →
+  // ai-full.txt (suffix the index's `output`, dir/ext preserved). A string
+  // `full` overrides the path outright.
+  let output = config.full
+  if (typeof output !== 'string') {
+    const p = path.parse(config.output || 'llms.txt')
+    output = path.join(p.dir, `${p.name}-full${p.ext}`)
+  }
+  const baseUrl = siteUrl ? siteUrl.replace(/\/+$/, '') : ''
+  const absUrl = (url) => baseUrl ? `${baseUrl}/${url}` : url
+  const isMarkdown = (src) => /\.(md|markdown)$/i.test(src || '')
+  const blocks = []
+  for (const e of pageEntries) {
+    if (e.isIndex || isNoindex(e) || !isMarkdown(e._src)) continue
+    const body = parseFrontMatter(e._src).content.trim()
+    if (!body) continue
+    const url = `URL: ${absUrl(e.url)}`
+    // If the body already opens with its own ATX H1, reuse it as the block
+    // heading and slot the URL beneath — otherwise the wrapper `# title` plus
+    // the body's H1 would emit two H1s per block (often exact duplicates).
+    const h1 = body.match(/^# .*(?:\r?\n|$)/)
+    blocks.push(h1
+      ? `${h1[0].trim()}\n${url}\n\n${body.slice(h1[0].length).trimStart()}`
+      : `# ${e.title || e.url}\n${url}\n\n${body}`)
+  }
+  if (!blocks.length) return
+  // Corpus header: names the file and its purpose so a whole-file ingest opens
+  // with context instead of the first page's H1. Reuses the index title/desc.
+  const name = config.title || 'this site'
+  let head = `# Full Documentation Archive for ${name}\n`
+  head += `\nThis file contains the complete Markdown documentation for ${name}.`
+  if (config.description) head += `\n\n> ${config.description}`
+  // Optional author-written preamble after the header — the full-file counterpart
+  // to `intro` for the index. Inserted verbatim; a missing file warns and skips.
+  if (config.fullIntro) {
+    try {
+      const body = fs.readFileSync(path.resolve(process.cwd(), config.fullIntro), 'utf-8').trim()
+      if (body) head += `\n\n${body}`
+    } catch {
+      log({ tag: 'indexer', warn: true, text: 'llms-full intro file not found:', link: config.fullIntro })
+    }
+  }
+  blocks.unshift(head)
+  const outputPath = path.resolve(process.cwd(), outputDir, output)
+  fs.mkdirSync(path.dirname(outputPath), { recursive: true })
+  fs.writeFileSync(outputPath, blocks.join('\n\n---\n\n') + '\n')
+  log({ tag: 'indexer', text: 'Generated llms-full.txt:', link: path.relative(process.cwd(), outputPath), size: fileSize(outputPath) })
+}
+// Generates a robots.txt. Defaults to allow-all with a `Sitemap:` line pointing
+// at the generated sitemap (absolute, needs `site.url`). The object form takes
+// `userAgent`, `allow`/`disallow` (string or array of paths) and `sitemap`
+// (an explicit URL, or `false` to omit the line). `sitemapOutput` is the
+// sitemap's filename, threaded in by generateIndexFiles.
+// ponytail: single user-agent group; add per-agent groups if a real need shows up.
+export function generateRobotsTxt(outputDir, siteUrl, config, sitemapOutput) {
+  config = normalizeConfig(config)
+  if (!config) return
+  const arr = (v) => v == null ? [] : (Array.isArray(v) ? v : [v])
+  const allow = arr(config.allow)
+  const disallow = arr(config.disallow)
+  const lines = [`User-agent: ${config.userAgent || '*'}`]
+  for (const p of allow) lines.push(`Allow: ${p}`)
+  for (const p of disallow) lines.push(`Disallow: ${p}`)
+  // Empty `Disallow:` is the canonical "allow everything" when nothing is set.
+  if (allow.length === 0 && disallow.length === 0) lines.push('Disallow:')
+  let sitemapUrl = null
+  if (config.sitemap === false) {
+    sitemapUrl = null
+  } else if (typeof config.sitemap === 'string') {
+    sitemapUrl = config.sitemap
+  } else if (sitemapOutput && siteUrl) {
+    sitemapUrl = `${siteUrl.replace(/\/+$/, '')}/${sitemapOutput}`
+  }
+  if (sitemapUrl) lines.push('', `Sitemap: ${sitemapUrl}`)
+  const outputPath = path.resolve(process.cwd(), outputDir, config.output)
+  fs.writeFileSync(outputPath, lines.join('\n') + '\n')
+  log({ tag: 'indexer', text: 'Generated robots.txt:', link: path.relative(process.cwd(), outputPath), size: fileSize(outputPath) })
+}
 function humanizeSegment(seg) {
   return seg
     .replace(/[-_]+/g, ' ')
@@ -333,8 +523,188 @@ export function generateNav(pageEntries, outputDir, config) {
   log({ tag: 'indexer', text: 'Generated nav:', link: path.relative(process.cwd(), outputPath), size: fileSize(outputPath) })
 }
+// RFC-822 date for RSS pubDate/lastBuildDate (toUTCString is RFC-822 compliant).
+function rssDate(d) {
+  const date = new Date(d)
+  return isNaN(date) ? '' : date.toUTCString()
+}
+// ISO-8601 (RFC-3339) date for Atom updated timestamps.
+function isoDate(d) {
+  const date = new Date(d)
+  return isNaN(date) ? new Date().toISOString() : date.toISOString()
+}
+function feedAuthorName(author) {
+  if (!author) return ''
+  return String(author.name || author)
+}
+// Normalizes the `feed` option into concrete per-collection specs. Accepts:
+//   true | "feed.xml"           → RSS for every collection
+//   { output, type, limit, … }  → those options for every collection
+//   { collection, … }           → one named collection
+//   [ {…}, {…} ]                → an explicit list
+// `output` containing a slash is a full path under the output dir; a bare
+// filename is placed in the collection's own folder (default "feed.xml").
+function resolveFeedSpecs(config, collectionNames, site) {
+  if (!config) return []
+  const raw = Array.isArray(config) ? config : [config]
+  const specs = []
+  for (let item of raw) {
+    if (item === true) item = {}
+    else if (typeof item === 'string') item = { output: item }
+    if (!item || typeof item !== 'object') continue
+    const targets = item.collection ? [item.collection] : collectionNames
+    for (const name of targets) {
+      const outName = item.output || 'feed.xml'
+      specs.push({
+        collection: name,
+        output: outName.includes('/') ? outName : `${name}/${outName}`,
+        type: item.type === 'atom' ? 'atom' : 'rss',
+        limit: Number(item.limit) > 0 ? Number(item.limit) : 20,
+        title: item.title || (site.title ? `${humanizeSegment(name)} | ${site.title}` : humanizeSegment(name)),
+        description: item.description || site.description || '',
+        author: feedAuthorName(item.author || site.author),
+        lang: item.lang || site.lang || '',
+        content: item.content === true // full article HTML in <content:encoded>
+      })
+    }
+  }
+  return specs
+}
+// Article-body HTML for a feed item's <content:encoded> / Atom <content>. Renders
+// the page's markdown SOURCE (entry._src) with the shared marked instance — the
+// same extensions the site uses — so it's clean article HTML, not entry.content
+// (the whole rendered page, layout + chrome). Markdown sources only; the render
+// cache is warm from the compile, so this is a lookup, not a re-render.
+// ponytail: raw markdown render, so unrendered {% %} tags / shortcodes in a body
+// leak; same ceiling as llms-full. Returns '' when there's no clean body to emit.
+function feedContentHtml(e) {
+  if (!e._src || !/\.(md|markdown)$/i.test(e._src)) return ''
+  return renderMarkdownCached(e._src, parseFrontMatter(e._src).content).trim()
+}
+// `]]>` would close the CDATA section early; split it so the payload survives.
+function cdata(html) {
+  return `<![CDATA[${html.replace(/]]>/g, ']]]]><![CDATA[>')}]]>`
+}
+function buildRssFeed(spec, items, { selfUrl, channelLink, abs }) {
+  let xml = '<?xml version="1.0" encoding="UTF-8"?>\n'
+  xml += '<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"'
+  xml += spec.content ? ' xmlns:content="http://purl.org/rss/1.0/modules/content/">\n' : '>\n'
+  xml += '<channel>\n'
+  xml += `  <title>${escapeXml(spec.title)}</title>\n`
+  xml += `  <link>${escapeXml(channelLink)}</link>\n`
+  if (spec.description) xml += `  <description>${escapeXml(spec.description)}</description>\n`
+  if (selfUrl) xml += `  <atom:link href="${escapeXml(selfUrl)}" rel="self" type="application/rss+xml"/>\n`
+  if (spec.lang) xml += `  <language>${escapeXml(spec.lang)}</language>\n`
+  if (items[0] && items[0].date) xml += `  <lastBuildDate>${rssDate(items[0].date)}</lastBuildDate>\n`
+  for (const e of items) {
+    const link = abs(e.url)
+    const desc = e.description || e.excerpt
+    xml += '  <item>\n'
+    xml += `    <title>${escapeXml(e.title || e.url)}</title>\n`
+    xml += `    <link>${escapeXml(link)}</link>\n`
+    xml += `    <guid isPermaLink="true">${escapeXml(link)}</guid>\n`
+    if (desc) xml += `    <description>${escapeXml(desc)}</description>\n`
+    if (e.date) xml += `    <pubDate>${rssDate(e.date)}</pubDate>\n`
+    if (spec.content) {
+      const html = feedContentHtml(e)
+      if (html) xml += `    <content:encoded>${cdata(html)}</content:encoded>\n`
+    }
+    xml += '  </item>\n'
+  }
+  xml += '</channel>\n</rss>\n'
+  return xml
+}
+function buildAtomFeed(spec, items, { selfUrl, channelLink, abs }) {
+  let xml = '<?xml version="1.0" encoding="UTF-8"?>\n'
+  xml += '<feed xmlns="http://www.w3.org/2005/Atom">\n'
+  xml += `  <title>${escapeXml(spec.title)}</title>\n`
+  if (spec.description) xml += `  <subtitle>${escapeXml(spec.description)}</subtitle>\n`
+  xml += `  <id>${escapeXml(selfUrl || channelLink)}</id>\n`
+  xml += `  <link href="${escapeXml(channelLink)}"/>\n`
+  if (selfUrl) xml += `  <link href="${escapeXml(selfUrl)}" rel="self" type="application/atom+xml"/>\n`
+  xml += `  <updated>${isoDate(items[0] && items[0].date)}</updated>\n`
+  if (spec.author) xml += `  <author><name>${escapeXml(spec.author)}</name></author>\n`
+  for (const e of items) {
+    const link = abs(e.url)
+    const desc = e.description || e.excerpt
+    const author = feedAuthorName(e.author)
+    xml += '  <entry>\n'
+    xml += `    <title>${escapeXml(e.title || e.url)}</title>\n`
+    xml += `    <link href="${escapeXml(link)}"/>\n`
+    xml += `    <id>${escapeXml(link)}</id>\n`
+    xml += `    <updated>${isoDate(e.date)}</updated>\n`
+    if (desc) xml += `    <summary>${escapeXml(desc)}</summary>\n`
+    if (author) xml += `    <author><name>${escapeXml(author)}</name></author>\n`
+    if (spec.content) {
+      const html = feedContentHtml(e)
+      if (html) xml += `    <content type="html">${cdata(html)}</content>\n`
+    }
+    xml += '  </entry>\n'
+  }
+  xml += '</feed>\n'
+  return xml
+}
+// Generates RSS/Atom feeds from collections — one file per resolved spec, its
+// items the collection's posts newest-first (by date), capped at `limit`.
+// Items carry description/excerpt by default; `content: true` on a spec adds
+// full article HTML (<content:encoded> / Atom <content>) via feedContentHtml,
+// which renders the markdown source rather than reusing the whole-page HTML.
+export function generateFeeds(pageEntries, outputDir, siteUrl, config, site = {}) {
+  if (!config) return
+  // Distinct collections present, first-seen order.
+  const collectionNames = []
+  const byCollection = new Map()
+  for (const e of pageEntries) {
+    if (e.isIndex || e.collection == null) continue
+    if (!byCollection.has(e.collection)) {
+      byCollection.set(e.collection, [])
+      collectionNames.push(e.collection)
+    }
+    byCollection.get(e.collection).push(e)
+  }
+  const baseUrl = siteUrl ? siteUrl.replace(/\/+$/, '') : ''
+  const abs = (u) => {
+    const rel = String(u == null ? '' : u).replace(/^\/+/, '')
+    if (/^https?:\/\//i.test(u)) return u
+    return baseUrl ? (rel ? `${baseUrl}/${rel}` : baseUrl) : `/${rel}`
+  }
+  for (const spec of resolveFeedSpecs(config, collectionNames, site)) {
+    const items = (byCollection.get(spec.collection) || [])
+      .filter(e => !isNoindex(e))
+      .sort((a, b) => new Date(b.date || 0) - new Date(a.date || 0))
+      .slice(0, spec.limit)
+    const ctx = { selfUrl: abs(spec.output), channelLink: abs(spec.collection), abs }
+    const xml = spec.type === 'atom' ? buildAtomFeed(spec, items, ctx) : buildRssFeed(spec, items, ctx)
+    const outputPath = path.resolve(process.cwd(), outputDir, spec.output)
+    fs.mkdirSync(path.dirname(outputPath), { recursive: true })
+    fs.writeFileSync(outputPath, xml)
+    log({ tag: 'indexer', text: 'Generated feed:', link: path.relative(process.cwd(), outputPath), size: fileSize(outputPath) })
+  }
+}
 export function generateIndexFiles(pageEntries, outputDir, siteUrl, config) {
   generateSearchIndex(pageEntries, outputDir, config.searchIndex)
   generateSitemap(pageEntries, outputDir, siteUrl, config.sitemap)
+  generateLlmsTxt(pageEntries, outputDir, siteUrl, config.llms)
+  generateLlmsFull(pageEntries, outputDir, siteUrl, config.llms)
+  const sitemapOutput = config.sitemap
+    ? (typeof config.sitemap === 'string' ? config.sitemap : config.sitemap.output)
+    : null
+  generateRobotsTxt(outputDir, siteUrl, config.robots, sitemapOutput)
   generateNav(pageEntries, outputDir, config.nav)
+  generateFeeds(pageEntries, outputDir, siteUrl, config.feed, config.site)
 }

package/lib/markups.js CHANGED Viewed

@@ -1,5 +1,5 @@
 import { pathExists, pathIsDirectory, readDataFile, mkDir, buildTime, toPosix } from './utils/helpers.js'
-import { replaceOutExtensions, getRelativePathPrefix, getPageUrlRelativeToOutput, parseFrontMatter, wordcount } from './markup/helpers.js'
+import { replaceOutExtensions, getRelativePathPrefix, getPageUrlRelativeToOutput, parseFrontMatter, wordcount, excerpt } from './markup/helpers.js'
 import { collectionAutoDiscovery, getCollectionDataBasedOnConfig, buildCollectionPaginationData, generateCollectionPaginationPages } from './markup/collections.js'
 import { generateIndexFiles, buildNavTree } from './markup/indexer.js'
 import NunjucksEngine from './markup/engines/nunjucks.js'
@@ -37,6 +37,9 @@ export default class Markups {
     this.includePaths = moduleConfig.includePaths || moduleConfig.options.includePaths || []
     this.searchIndexConfig = moduleConfig.options.searchIndex || moduleConfig.searchIndex
     this.sitemapConfig = moduleConfig.options.sitemap || moduleConfig.sitemap
+    this.llmsConfig = moduleConfig.options.llms || moduleConfig.llms
+    this.robotsConfig = moduleConfig.options.robots || moduleConfig.robots
+    this.feedConfig = moduleConfig.options.feed || moduleConfig.feed
     this.navConfig = moduleConfig.options.nav || moduleConfig.nav
     this.baseURL = moduleConfig.baseURL || moduleConfig.options.baseURL || null
@@ -107,6 +110,16 @@ export default class Markups {
     this.loadDataFiles(this.dataConfig)
   }
+  // Normalizes the llms config (string shorthand or object) and fills in the
+  // title/description from site data so generateLlmsTxt gets ready values.
+  resolveLlmsConfig() {
+    if (!this.llmsConfig) return undefined
+    const base = typeof this.llmsConfig === 'string' ? { output: this.llmsConfig } : { ...this.llmsConfig }
+    base.title = base.title || this.siteData.title
+    base.description = base.description || this.siteData.description
+    return base
+  }
   loadDataFiles(files) {
     if (!files) return
@@ -225,6 +238,7 @@ export default class Markups {
       context.page = frontMatterResult.frontMatter
       context.page.content = frontMatterResult.content
       context.page.wordcount = wordcount(frontMatterResult.content)
+      context.page.excerpt = excerpt(frontMatterResult.content)
     } catch (err) {
       log({ tag: 'error', text: 'Failed parsing front matter:', link: templateName })
       console.error(err)
@@ -502,7 +516,7 @@ export default class Markups {
       ...getCollectionDataBasedOnConfig(this.markupIn, this.collectionsConfig)
     }
-    const shouldIndex = this.searchIndexConfig || this.sitemapConfig || this.navConfig
+    const shouldIndex = this.searchIndexConfig || this.sitemapConfig || this.navConfig || this.feedConfig
     const pageEntries = shouldIndex ? [] : null
     buildCollectionPaginationData(collectionData)
@@ -541,7 +555,11 @@ export default class Markups {
       generateIndexFiles(pageEntries, this.markupOut, this.siteData.url, {
         searchIndex: this.searchIndexConfig,
         sitemap: this.sitemapConfig,
-        nav: this.navConfig
+        llms: this.resolveLlmsConfig(),
+        robots: this.robotsConfig,
+        nav: this.navConfig,
+        feed: this.feedConfig,
+        site: this.siteData
       })
     }
@@ -551,6 +569,20 @@ export default class Markups {
     this.lastPageEntries = pageEntries
   }
+  // Watch hook for .json/.yaml changes. Data extensions are ambiguous: a
+  // .json may be a data file feeding the globals, or engine-owned markup
+  // (Shopify templates/*.json, section-group JSON). Engines that can tell
+  // claim theirs via isMarkupSource() and get the incremental path;
+  // everything else reloads data globals and full-compiles, as before.
+  async compileDataChange(file) {
+    await this.init()
+    if (this.engine && typeof this.engine.isMarkupSource === 'function' &&
+      this.engine.isMarkupSource(path.resolve(process.cwd(), file))) {
+      return this.compileIncremental(file)
+    }
+    return this.reloadDataFiles().then(() => this.compile())
+  }
   // Watch-mode entry point: rebuild only the pages whose last render touched
   // the changed file. Falls back to a full compile() whenever the result
   // isn't provably identical to a full build — unsupported engine, no prior
@@ -593,6 +625,14 @@ export default class Markups {
       if (this.isCollectionIndexOverride(rel.split(path.sep), collectionData)) return this.compile()
     }
+    // A page's own dep set always contains itself, so an edited page is in
+    // `affected` (exact match, no glob needed). If it isn't, but the file IS a
+    // page source, this is a new page whose basename collides with a recorded
+    // dep — rendering only the collided pages would leave it unbuilt.
+    if (!affected.includes(abs) && this.getMarkupFiles(markupIn).some((f) => path.resolve(f) === abs)) {
+      return this.compile()
+    }
     const markupStart = performance.now()
     const newEntries = this.lastPageEntries ? [] : null
@@ -628,7 +668,11 @@ export default class Markups {
       generateIndexFiles(this.lastPageEntries, this.markupOut, this.siteData.url, {
         searchIndex: this.searchIndexConfig,
         sitemap: this.sitemapConfig,
-        nav: this.navConfig
+        llms: this.resolveLlmsConfig(),
+        robots: this.robotsConfig,
+        nav: this.navConfig,
+        feed: this.feedConfig,
+        site: this.siteData
       })
     }
@@ -640,6 +684,6 @@ export default class Markups {
 // Everything except the fields that change on every content edit — used to
 // detect front matter drift that would invalidate the shared nav global
 function entrySignature(entry) {
-  const { content, wordcount, ...rest } = entry
+  const { content, wordcount, excerpt, ...rest } = entry
   return JSON.stringify(rest)
 }

package/package.json CHANGED Viewed

@@ -1,7 +1,7 @@
 {
   "name": "poops",
   "description": "Straightforward, no-bullshit bundler for the web.",
-  "version": "1.5.2",
+  "version": "1.6.0",
   "license": "MIT",
   "type": "module",
   "main": "poops.js",

package/poops.js CHANGED Viewed

@@ -3,6 +3,7 @@
 import chokidar from 'chokidar'
 import connect from 'connect'
 import Copy from './lib/copy.js'
+import runExec from './lib/exec.js'
 import { pathExists, doesFileBelongToPath, pathContainsPathSegment, deriveWatchDirs } from './lib/utils/helpers.js'
 import http from 'node:http'
 import os from 'node:os'
@@ -98,6 +99,10 @@ function styleOutputs(config) {
       : path.join(entry.out, path.basename(entry.in).replace(/\.(sass|scss)$/i, '.css'))))
 }
+// Per-stage shell hooks (config.exec). Runs after a stage compiles in both
+// build and watch; see lib/exec.js. `hook(stage)` binds config + cwd here.
+const hook = (stage) => runExec(config, cwd, stage)
 function setupLiveReloadServer(config) {
   if (!config.livereload) return
   liveReloadServer = livereload.createServer({ port: config.livereload_port })
@@ -132,31 +137,33 @@ function setupWatchers(config, modules) {
     // exactly this file's entries; shared by change/add/unlink via rebuild.
     modules.markups.invalidate(file)
     if (/(\.m?jsx?|\.tsx?)$/i.test(file) && !isBuildOutput(file)) {
-      modules.scripts.compile().then(() => reload()).catch(err => console.error(err))
+      modules.scripts.compile().then(() => { hook('scripts'); reload() }).catch(err => console.error(err))
       if (modules.reactor.belongsToReactor(file)) {
         modules.reactor.compile().then(() => {
           if (modules.reactor.renderedChanged) {
             config.reactorData = modules.reactor.getRendered()
-            modules.markups.compile().then(() => modules.postcss.compile()).then(() => reload()).catch(err => console.error(err))
+            modules.markups.compile().then(() => modules.postcss.compile()).then(() => { hook('markup'); reload() }).catch(err => console.error(err))
           }
         }).catch(err => console.error(err))
       }
     }
     if (/(\.sass|\.scss|\.css)$/i.test(file) && !isBuildOutput(file)) {
       modules.styles.compile().then(() => modules.postcss.compile())
-        .then(() => styleOutputs(config).forEach((out) => reload(out)))
+        .then(() => { hook('styles'); styleOutputs(config).forEach((out) => reload(out)) })
         .catch(err => console.error(err))
     }
     if (/(\.html|\.xml|\.rss|\.atom|\.njk|\.liquid|\.md)$/i.test(file)) {
       // Incremental: re-render only the pages whose last render touched this
       // file; falls back to a full compile for anything it can't prove safe
       // (deletions, new files, collection members, engines without dep info).
-      modules.markups.compileIncremental(file).then(() => modules.postcss.compile()).then(() => reload()).catch(err => console.error(err))
+      modules.markups.compileIncremental(file).then(() => modules.postcss.compile()).then(() => { hook('markup'); reload() }).catch(err => console.error(err))
     }
     if (/(\.json|\.ya?ml)$/i.test(file)) {
-      modules.markups.reloadDataFiles().then(() => modules.markups.compile()).then(() => reload()).catch(err => console.error(err))
+      // Engine-owned markup with a data extension (Shopify templates/*.json)
+      // goes incremental; real data files reload globals + full compile.
+      modules.markups.compileDataChange(file).then(() => { hook('markup'); reload() }).catch(err => console.error(err))
     }
   }
@@ -172,13 +179,13 @@ function setupWatchers(config, modules) {
       modules.images.compile()
         .then(() => modules.markups.compile())
         .then(() => modules.postcss.compile())
-        .then(() => reload())
+        .then(() => { hook('images'); hook('markup'); reload() })
         .catch(err => console.error(err))
     }
     // A copied .css (e.g. the styles compiler's own output landing in a copy
     // source) stays a hot-swap; any other copied file needs a full reload.
     doesFileBelongToPath(file, config.copy) && modules.copy.execute()
-      .then(() => reload(/\.css$/i.test(file) ? file : undefined))
+      .then(() => { hook('copy'); reload(/\.css$/i.test(file) ? file : undefined) })
       .catch(err => console.error(err))
   }
@@ -205,7 +212,7 @@ function setupWatchers(config, modules) {
       modules.images.remove(file)
         .then(() => modules.markups.compile())
         .then(() => modules.postcss.compile())
-        .then(() => reload())
+        .then(() => { hook('images'); hook('markup'); reload() })
         .catch(err => console.error(err))
     }
     rebuild(file)
@@ -216,7 +223,7 @@ function setupWatchers(config, modules) {
     modules.markups.invalidate(dirPath) // prefix match drops every template under it
     modules.markups.removeOutput(dirPath)
     if (doesFileBelongToPath(dirPath, config.markup)) {
-      modules.markups.compile().then(() => modules.postcss.compile()).then(() => reload()).catch(err => console.error(err))
+      modules.markups.compile().then(() => modules.postcss.compile()).then(() => { hook('markup'); reload() }).catch(err => console.error(err))
     }
     modules.copy.unlink(dirPath, doesFileBelongToPath(dirPath, config.copy))
   }
@@ -257,12 +264,19 @@ async function poops() {
   await step(() => styles.compile())
   await step(() => reactor.compile())
+  failed = hook('reactor') || failed
   config.reactorData = reactor.getRendered()
   await step(() => scripts.compile())
+  failed = hook('scripts') || failed
   await step(() => images.compile()) // before markups: engines read the poops-images cache
+  failed = hook('images') || failed
   await step(() => markups.compile())
+  failed = hook('markup') || failed
   await step(() => postcss.compile())
+  failed = hook('styles') || failed // after PostCSS so the CSS is final
   await step(() => copy.execute())
+  failed = hook('copy') || failed
+  failed = hook('build') || failed
   if (build || (!config.watch && !config.livereload && !config.serve)) {
     process.exit(failed || hasLoggedErrors() ? 1 : 0)