@fullstackdatasolutions/articles 1.2.3 → 1.3.1

package/CHANGELOG.md

@@ -5,6 +5,52 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.3.1] - 2026-08-23
+
+### Fixed
+
+- **Markdown twins no longer declare themselves their own alternate.** `getArticleMarkdownResponse` spread `getArticleAiHeaders(article, config)` into the twin's response headers. That helper builds `Link: <{slug}.md>; rel="alternate"; type="text/markdown"`, which is correct on the *HTML* article page - it advertises that a markdown representation exists - but on the twin's own response it emitted a link from `/articles/x.md` to `/articles/x.md`, telling a client the alternate of the URL it just fetched was that same URL. A client following it round-trips to itself, and nothing in the response pointed back at the canonical HTML page the markdown represents, which is the one relationship an answer engine needs in order to attribute a citation to a real page. The twin now emits `Link: <{siteUrl}/articles/{slug}>; rel="canonical"` instead. The helper's other branch (`X-Robots-Tag: noai, noimageai`) was unreachable from this call site regardless, since `getArticleMarkdown` already returns `null` - and the route 404s - for anything not opted in, so dropping the spread loses no behavior. `getArticleAiHeaders` itself is unchanged and still correct for its documented use on HTML route handlers. Backward compatible: the header key is the same, only its target and relation change.
+
+## [1.3.0] - 2026-08-22
+
+### Added
+
+- **`ArticlesConfig.aiCrawlDefault`** - default `aiCrawl` value for articles whose frontmatter omits the key. Defaults to `false`, so every existing site keeps the original "blocked unless the article opts in" behavior byte-for-byte. Set it to `true` on a site whose goal is being cited by answer engines to opt the entire corpus in at once; a per-article `aiCrawl: false` still wins and keeps that article blocked. `getArticleMarkdown` and `getAiRobotsTxtRules` now accept an optional `config` so the resolved value reaches them (both were previously calling `getArticleSummary`/`getAllArticles` with no config, which also meant degraded author resolution in the markdown path).
+- **Attribution header on article markdown twins.** `matter()` strips every frontmatter field before the twin is served, so a model fetching `/articles/[slug].md` previously received an anonymous body - no title, date, author, site, or link back to the canonical page, and therefore nothing to attribute a citation to. `getArticleMarkdownResponse` now prefixes the body with the title, excerpt, canonical source URL, published/updated dates, author, and site name. The `# {title}` line is skipped when the body already opens with the same H1, so the common "body repeats the title" layout does not end up with two. Set `ArticlesConfig.markdownTwinHeader: false` to serve the bare body as before. The header builder is exported as `buildMarkdownTwinHeader` for apps that assemble their own responses.
+- **`generateLlmsTxt(articles, config)`** - builds an `llms.txt` index (https://llmstxt.org) listing only `aiCrawl` articles, grouped by category, linking to each article's `.md` twin rather than its HTML page. Wire it up as `app/llms.txt/route.ts` in the consuming app.
+- **`generateLlmsFullTxt(articles, config)`** - builds `llms-full.txt`: every opted-in article's full markdown twin, attribution headers included, concatenated into one document. Larger and slower than `generateLlmsTxt`; generate it in a route handler or at build time, not per request.
+
+- **`ArticlesConfig.organization` plus `OrganizationSchema` and `WebSiteSchema`.** Every article previously emitted its own inline `{'@type':'Organization', name: siteName}` publisher stub with no `@id`, so nothing tied articles, authors, and the site together into one entity - each page read as an orphan. Setting `organization` emits a single `Organization` (or `Person`, for a personal brand) node at `{siteUrl}/#organization` and a `WebSite` node at `{siteUrl}/#website` from the root layout, and switches every article's `publisher` to a reference to that `@id`. Authors now carry `@id: {authorUrl}#person` in both the article and author-page `Person` schemas, so all of an author's articles resolve to one entity rather than N name-matched duplicates. `WebSite` emits a `SearchAction` only when `organization.searchUrlTemplate` is set - the library's own search is client-side with no crawlable results URL, so declaring one unconditionally would advertise an endpoint that does not exist. Omitting `organization` keeps the previous inline publisher stub.
+- **`AuthorProfile.knowsAbout`** and a fix to the author-page `Person` schema, which was emitting `knowsAbout: config.siteName` - a publisher name where schema.org expects subject matter, which told a consumer nothing. It now derives from the categories of the author's own published articles, or uses `AuthorProfile.knowsAbout` when set. Unlike `credentials`/`proof` (still excluded as unverifiable self-reported claims), subject matter is verifiable from the corpus itself. The `Person` `@id`/`url`/`mainEntityOfPage` are now consistently absolute; `@id` previously would have been built from `getAuthorUrl`'s relative path, which cannot resolve to the same node across pages.
+- **`Article.answer` and the `ArticleAnswer` component.** A direct, self-contained answer to the article's core question - rendered as a callout above the body, emitted as the Article schema's `abstract`, placed at the top of the markdown twin, and used as the `acceptedAnswer` when `articleType` is `'QAPage'` (folded into the main schema's `mainEntity`, since `articleType` already retypes that node and a second `QAPage` script would present the same page as two competing entities). Answer-first passages are the format most likely to be lifted verbatim, and nothing in the package previously produced one.
+- **Article schema entity and trust fields**: `inLanguage` (from the new `ArticlesConfig.language`, default `'en'`), `isAccessibleForFree` (default `true`, set `ArticlesConfig.isAccessibleForFree: false` for paywalled content), `about` (from `Article.about` - entity references with `sameAs`, which is what lets a consumer resolve a topic rather than guess from a string), `citation` (from `Article.citation`), and `speakable` (from `ArticlesConfig.speakableSelectors`, omitted by default since the correct selectors depend on the consuming app's markup).
+- **`ArticlesConfig.deriveFaqFromHeadings`** and the exported `deriveFaqFromHeadings(markdown)`. Derives `FAQPage` entries from question-shaped `##` headings and the paragraph beneath each. Off by default and deliberately conservative: it requires a closed list of interrogative openers, so a rhetorical heading like "Sound familiar?" is not promoted into a published Q&A pair. Explicit `faq` frontmatter always wins.
+- **`generateRssFeed(articles, config, { fullContent: true })`** adds `<content:encoded>` with each article's rendered HTML, and the feed now declares the `content` namespace. An excerpt-only feed gives an ingestion pipeline nothing to work with. Off by default; it needs articles loaded with `htmlContent`, and articles without it are emitted unchanged. The channel `<language>` now follows `ArticlesConfig.language` instead of a hardcoded `en`.
+- **Five answer-engine readiness rules in `validateArticles`**, all warnings: `no-answer` (no `answer`, no `faq`, no question-shaped `h2`), `thin-content` (under 300 words), `missing-about` (no entity references), `stale-content` (not updated in 18 months - the clock is injectable via `options.now` so CI runs are deterministic), and `orphan-article` (no other article's body links to it, checked only when articles are loaded with their `content`).
+
+- **Markdown twins for listing surfaces.** `getCategoryMarkdown`, `getAuthorMarkdown`, and `getSeriesMarkdown` build `.md` twins for `/articles/category/[category]`, `/articles/authors/[author]`, and `/articles/series/[series]`. Article twins alone leave a model with a flat bag of pages and no map; these are the surfaces that answer "what does this site cover, and who writes it". The author twin also carries the bio, promise, `servesWho`, `knowsAbout`, `credentials`, `proof`, and `originStory` that previously existed only inside React components - the "why trust them" context asked for before anything an author wrote gets cited (prose can attribute a stated claim without asserting it as fact, which is why `credentials` appear here but still not in JSON-LD). `getMarkdownTwinResponse` dispatches by slug prefix, so the existing single `/articles/:path*.md` rewrite serves all four kinds with no new app routes - without it, `/articles/category/campaigns.md` matched that rewrite, hit the article handler, and 404'd. `generateLlmsTxt` now links these under a `## Collections` section.
+- **`ArticlesConfig.onAiCrawl`** plus `matchAiCrawler` and the exported `AI_CRAWLERS` list. `getMarkdownTwinResponse` is the one choke point every AI-crawler markdown fetch passes through, so it is the only place a site can measure whether its AI-readable content is actually being read, and by which bot. The payload is the slug, the matched crawler name, and the raw user agent - no PII. A throwing handler is reported through `setArticlesErrorHandler` and never turns a served article into a 500. The same `AI_CRAWLERS` list now drives `getAiRobotsTxtRules`, so the robots rules and the telemetry can't disagree about what counts as an AI crawler; it also picks up agents added since the original list (`OAI-SearchBot`, `Claude-SearchBot`, `Perplexity-User`, `Applebot-Extended`, `Bytespider`, `Amazonbot`, `meta-externalagent`, `cohere-ai`, `DuckAssistBot`, `MistralAI-User`).
+- **`ArticlesConfig.entities`** - a shared entity vocabulary keyed by slug. An `about` entry may now be a bare string that resolves through the registry, so a corpus shares one canonical name/`sameAs` pair per topic instead of every article spelling it out, differently. The whole point of `about` is that a consumer can resolve one entity across a corpus, which "Pathfinder", "pathfinder", and "PF2e" as three free-text strings defeats. Unregistered strings still work as plain names; `validateArticles` warns (`unknown-entity`) rather than dropping them.
+- **`ArticlesConfig.lastmodFallback`** (`'published'` | `'none'` | `'fileMtime'`, default `'published'`). An article edited three times with no `lastmod` in frontmatter reported its original publish date as `dateModified` forever - a silent freshness lie. `'none'` omits `dateModified` instead of repeating a stale date; `'fileMtime'` reads the article file's modification time, which is accurate locally but reports the whole corpus as updated today on a CI runner that clones fresh, so it is opt-in and documented as such.
+- **`AuthorProfile.identityUrl`**, **`AuthorProfile.sameAs`**, and **`organization.parentOrganization`** for authors and sites that span a network. `identityUrl` is the one canonical URL a Person `@id` derives from, set identically on every site an author publishes on, so all of it resolves to a single entity rather than one per site. It is deliberately separate from `url`: that field also drives byline link targets, the author page's Open Graph URL, and its `CollectionPage` URL, so pointing it at another domain would send readers off-site and hand this site's author page a canonical belonging to a different one. `identityUrl` changes nothing a reader sees. Explicit `sameAs` URLs merge with the derived social links (deduped), and `parentOrganization` links a site to its umbrella entity. Relatedly, the author page's Person `mainEntityOfPage` is now always this site's own author page - it states where the entity is described, so it must never point at another site, which the previous `author.url ?? default` expression allowed.
+- **Comment structured data.** `ArticleSEO` accepts a `comments` prop and emits `commentCount` plus `comment` entries. Genuine discussion is a quality signal that was previously invisible to any consumer. Deleted comments and replies are excluded, and only the author name, body, and timestamp are emitted - never the commenter's id.
+
+- **`ArticlesConfig.titleTemplate`** (default `'{title} | {siteName}'`, supporting `{title}` and `{siteName}`). Every page title previously hardcoded a `| siteName` suffix with no way to opt out. Google truncates a result title around 60 characters, so a site name spends that budget on every page whether or not anyone searches for the brand - on one site in this repo's network the suffix pushed 74% of titles over the limit while the brand itself drew 11 impressions in 90 days. Set `'{title}'` to drop it. Applied through the new `formatPageTitle` helper so article, category, series, and author titles can never drift apart on how the suffix is appended.
+
+### Fixed
+
+- **`validateArticles` measured the wrong strings for title and description length.** `checkDiscoveryFieldLengths` only checked `searchTitle`/`searchDescription`, and those are optional overrides most sites never set - so a corpus where every rendered title was over-length produced no warnings at all, because the field being measured was empty. It now measures what a search result actually renders: `titleTemplate` applied to `searchTitle ?? title`, and `searchDescription ?? excerpt`. The `search-title-too-long`/`search-description-too-long` codes are replaced by `effective-title-too-long`/`effective-description-too-long`; the social-field checks are unchanged.
+- **The Article schema's `image` was a relative URL.** `featuredImage` is stored site-relative (`/articles/<slug>/hero.jpg`), and Open Graph already absolutized it through `resolveImageUrl` - JSON-LD did not, so every article published an `ImageObject.url` that consumers of structured data reject. Now absolutized against `config.siteUrl`, falling back to the origin of `articleUrl` when no config is passed. No `width`/`height` added: the package never measures the file, and asserting dimensions it has not read would be a guess in structured data.
+- **Category sitemap entries stamped `lastModified: new Date()`**, telling every crawl that every category changed today - the exact freshness signal a sitemap exists to carry, inverted. Now derived from the newest article in the category. Author entries, which carried no `lastModified` at all, are derived the same way.
+- **Series pages were missing from the sitemap.** `/articles/series/[series]` routes have static params, metadata, and (as of this release) a markdown twin, but nothing listed them. Now emitted for every distinct `seriesSlug`, dated from the newest article in the series.
+- **Paginated listing routes were missing from the sitemap** in `listingPagination: 'pages'` mode. Their canonical/prev/next metadata was already correct, but no sitemap declared they existed. Now emitted from page 2 upward (page 1 is the listing URL itself) for the index, each category, and each author.
+
+### Changed
+
+- `getArticleMarkdownResponse` output now includes the attribution header by default (see above).
+- `createArticleMarkdownHandler` now calls `getMarkdownTwinResponse` (article twins behave identically; category/author/series paths that previously 404'd now resolve) and passes the request through so `onAiCrawl` can read the user agent.
+- Article and author `Person` JSON-LD now carry an `@id` when an author URL is resolvable, and the Article schema always emits `inLanguage` and `isAccessibleForFree`. These are additive fields on existing nodes; a site that sets none of the new config options sees no other output change.
+
 ## [1.2.3] - 2026-08-21
 
 ### Fixed

package/README.md

@@ -480,6 +480,17 @@ Use this if you don't need a custom API base path. For custom paths, use `create
 | `linkTargetStrategy`   | `'external-new-tab' \| 'all-new-tab' \| 'same-tab'` | `'external-new-tab'` | Controls article body links. Internal links open in the same window by default; external `http`/`https` links open in a new window.                 |
 | `mdxComponents`        | `Record<string, ComponentType<never>>`              | -                    | Components exposed to article `.mdx` bodies by tag name. Merged with the built-in MDX image override.                                               |
 | `description`          | `string`                                            | —                    | Short description used as the RSS feed channel description. Falls back to `siteName` if omitted.                                                    |
+| `entities`             | `Record<string, EntityReference>`                    | —                    | Shared entity vocabulary. Article `about` entries may reference a key instead of repeating a name/`sameAs` pair.                                     |
+| `lastmodFallback`      | `'published' \| 'none' \| 'fileMtime'`               | `'published'`        | Where `lastmod` comes from when frontmatter omits it. See [Freshness](#freshness).                                                                   |
+| `onAiCrawl`            | `(event: AiCrawlEvent) => void`                     | —                    | Fired when an AI crawler fetches a markdown twin. The only place to measure whether AI-readable content is being read.                               |
+| `organization`         | `OrganizationConfig`                                | —                    | Publishing entity behind the site. Set it to emit `OrganizationSchema`/`WebSiteSchema` and have every article reference one `@id`.                   |
+| `titleTemplate`        | `string`                                            | `'{title} \| {siteName}'` | Page `<title>` template. Supports `{title}` and `{siteName}`. Set `'{title}'` to drop the site-name suffix - see [Title length](#title-length).             |
+| `language`             | `string`                                            | `'en'`               | BCP 47 tag. Emitted as Article `inLanguage` and the RSS channel `<language>`.                                                                        |
+| `isAccessibleForFree`  | `boolean`                                           | `true`               | Set `false` for paywalled content. Emitted as Article `isAccessibleForFree`.                                                                         |
+| `speakableSelectors`   | `string[]`                                          | —                    | CSS selectors for the Article `speakable` field. Omitted by default - correct selectors depend on your markup.                                       |
+| `deriveFaqFromHeadings`| `boolean`                                           | `false`              | Derive `FAQPage` entries from question-shaped `##` headings. Explicit `faq` frontmatter always wins.                                                 |
+| `aiCrawlDefault`       | `boolean`                                           | `false`              | Default `aiCrawl` for articles whose frontmatter omits it. Set `true` to opt the whole corpus in; per-article `aiCrawl: false` still wins.            |
+| `markdownTwinHeader`   | `boolean`                                           | `true`               | Prefix markdown twins with title, excerpt, canonical source URL, dates, author, and site name. Set `false` to serve the bare body.                   |
 | `listingPagination`    | `'load-more' \| 'pages'`                            | `'load-more'`        | `'load-more'` keeps the client-only "Load more" button (no URL change). `'pages'` opts every listing surface into real, crawlable paginated routes - see the [Pagination](#pagination) section.  |
 
 **Default layout order:** `['hero', 'search', 'featured', 'latest', 'categories']`
@@ -1113,6 +1124,25 @@ import { CtaViewTracker } from '@fullstackdatasolutions/articles'
 
 `validateArticles(articles, config)` (pure, no `fs`) and `validateAllArticles(config)` (loads via `getAllArticles` first) check required frontmatter, unique canonical URLs, valid author references, series order/slug collisions, `Path` article references (missing or draft), unsafe URL schemes (`javascript:`/`data:`/`vbscript:` in `Path.nextAction.href`/`AuthorProfile.primaryCta.href`), and discovery field length limits - returning `{ ok, errors, warnings }`. Errors are broken-reader-journey issues (fail your build/CI on them); warnings are optional-field gaps (missing excerpt/date, over-length search/social fields, category slug collisions).
 
+**Answer-engine readiness warnings:**
+
+| Code | Fires when |
+| --- | --- |
+| `no-answer` | No `answer`, no `faq`, and no question-shaped `##` heading - nothing for an answer engine to lift. |
+| `thin-content` | `wordCount` under 300. |
+| `missing-about` | No `about` entity references. |
+| `stale-content` | `lastmod` (or `date`) older than 18 months. |
+| `orphan-article` | No other article's body links to it. |
+| `unknown-entity` | An `about` entry is not in `config.entities` and has no `sameAs`. |
+| `effective-title-too-long` | The rendered `<title>`, suffix included, exceeds 60 characters. |
+| `effective-description-too-long` | The rendered meta description exceeds 160 characters. |
+
+`orphan-article` only runs when articles are loaded with their `content` (i.e. via `getArticleMetadata`, not `getAllArticles`' summaries). Pass `{ now }` as a third argument to both functions to pin the clock the `stale-content` check uses, so CI runs are deterministic:
+
+```ts
+const result = validateArticles(articles, siteConfig, { now: new Date('2026-08-22') })
+```
+
 ```ts
 // scripts/validate-articles.ts (your app)
 import { validateAllArticles } from '@fullstackdatasolutions/articles/server'
@@ -1350,6 +1380,100 @@ export async function GET() {
 - Uses `generateRssFeed(articles, siteConfig)` from the package so RSS escaping, authors, descriptions, and feed URLs stay consistent across consuming apps
 - Uses `siteConfig.description` for the channel description field (falls back to `siteName` if omitted)
 - The `<link rel="alternate" type="application/rss+xml">` tag is added automatically to the articles index `<head>` via `generateArticlesIndexMetadata`
+- The channel `<language>` follows `siteConfig.language` (default `'en'`)
+
+**Full-text feeds.** An excerpt-only feed gives an ingestion pipeline nothing to work with. Pass `{ fullContent: true }` to add `<content:encoded>` with each article's rendered HTML:
+
+```ts
+const articles = await Promise.all(
+  (await getAllArticles(siteConfig)).map((a) => getArticleMetadata(a.slug, siteConfig))
+)
+const xml = generateRssFeed(articles.filter(Boolean), siteConfig, { fullContent: true })
+```
+
+It needs `htmlContent`, which `getAllArticles`' summaries do not carry - hence the `getArticleMetadata` pass above. Articles without it are emitted unchanged, no element.
+
+### Site entity graph
+
+Without `organization`, every article carries its own inline `{'@type':'Organization', name: siteName}` publisher stub with no `@id` - nothing ties articles, authors, and the site into a single entity. Set it, render the two schema components once in the root layout, and every article's `publisher` becomes a reference to one node instead:
+
+```ts
+// config/articles.ts
+export const siteConfig: ArticlesConfig = {
+  siteUrl: 'https://yoursite.com',
+  siteName: 'Your Site',
+  organization: {
+    type: 'Organization', // or 'Person' for a personal brand
+    logo: '/logo.png',
+    sameAs: [
+      'https://www.linkedin.com/company/yoursite',
+      'https://github.com/yoursite',
+    ],
+  },
+}
+```
+
+```tsx
+// app/layout.tsx
+import { OrganizationSchema, WebSiteSchema } from '@fullstackdatasolutions/articles'
+import { siteConfig } from '@/config/articles'
+
+export default function RootLayout({ children }: { children: React.ReactNode }) {
+  return (
+    <html lang="en">
+      <body>
+        <OrganizationSchema config={siteConfig} />
+        <WebSiteSchema config={siteConfig} />
+        {children}
+      </body>
+    </html>
+  )
+}
+```
+
+This produces stable identifiers - `{siteUrl}/#organization`, `{siteUrl}/#website`, and `{authorUrl}#person` for each author - so every article by an author resolves to the same `Person` rather than N name-matched duplicates. `getOrganizationId`, `getWebSiteId`, and `getPersonId` are exported if you need to reference the same nodes from your own schema.
+
+`WebSiteSchema` emits a `SearchAction` only when you set `organization.searchUrlTemplate` (e.g. `'/search?q={search_term_string}'`). The library's own search is client-side with no crawlable results URL, so it is not declared for you.
+
+### Answer-first content
+
+`answer` is a direct, self-contained answer to the article's core question - 40 to 60 words. It is the passage most likely to be lifted verbatim by an answer engine, so the package surfaces it in four places at once: as a reader-facing callout, as the Article schema's `abstract`, at the top of the markdown twin, and as the `acceptedAnswer` when `articleType: QAPage`.
+
+```markdown
+---
+title: How does initiative work?
+excerpt: Turn order in combat, explained.
+answer: Every combatant rolls a d20 and adds their initiative modifier. Highest result acts first, and the order repeats each round until combat ends.
+articleType: QAPage
+about:
+  - name: Pathfinder
+    sameAs: https://www.wikidata.org/wiki/Q1194077
+  - Combat
+citation:
+  - name: Core Rulebook
+    url: https://example.com/crb
+---
+```
+
+```tsx
+import { ArticleAnswer } from '@fullstackdatasolutions/articles'
+
+<ArticleAnswer article={article} />
+```
+
+`ArticleAnswer` returns `null` when the article has no `answer`, so it is safe to render unconditionally. `about` entries carry a `sameAs` identifier so a consumer can resolve the actual entity rather than guess from a string; `citation` lists outbound sources.
+
+### Derived FAQ
+
+Set `deriveFaqFromHeadings: true` to build `FAQPage` entries from question-shaped `##` headings and the paragraph beneath each:
+
+```markdown
+## What is initiative?
+
+Turn order for combat. Everyone rolls once at the start.
+```
+
+Off by default, and deliberately conservative - a heading must both end in `?` and open with an interrogative, so "Sound familiar?" is not promoted into a published Q&A pair. Explicit `faq` frontmatter always wins over derived entries. `deriveFaqFromHeadings(markdown)` is exported if you want to inspect what it would produce.
 
 ### AI markdown twins
 
@@ -1365,6 +1489,36 @@ aiCrawl: true
 Markdown body...
 ```
 
+To opt in an entire corpus at once, set `aiCrawlDefault: true` in `ArticlesConfig` instead of adding the flag to every file. Frontmatter still wins - `aiCrawl: false` keeps that one article blocked.
+
+```ts
+export const siteConfig: ArticlesConfig = {
+  siteUrl: 'https://yoursite.com',
+  siteName: 'Your Site',
+  aiCrawlDefault: true,
+}
+```
+
+**Attribution header.** Frontmatter is stripped before the twin is served, so the markdown response is prefixed with the article's title, excerpt, canonical source URL, dates, author, and site name - the context a model needs to cite the article rather than reproduce it anonymously:
+
+```markdown
+# Running Your First Session
+
+> A short summary.
+
+Source: https://yoursite.com/articles/guides/first-session
+Published: 2025-01-01
+Updated: 2025-06-01
+Author: Jane Doe
+Site: Your Site
+
+---
+
+Markdown body...
+```
+
+The `# {title}` line is skipped when the body already opens with the same H1. Set `markdownTwinHeader: false` to serve the bare body instead. `buildMarkdownTwinHeader(article, config, body)` is exported from `./server` for apps that build their own responses.
+
 To expose markdown at public URLs like `/articles/my-post.md`, use the markdown handler factories above:
 
 - **Next.js 16+** — Use `createArticleMarkdownHandler()` + `rewriteArticleMarkdown()` in proxy.ts (see Step 6 above).
@@ -1372,6 +1526,164 @@ To expose markdown at public URLs like `/articles/my-post.md`, use the markdown
 
 For static generation, call `getArticleMarkdown(slug)` during your build and write the returned content beside your generated HTML; it returns `null` unless `aiCrawl: true`.
 
+### Listing twins
+
+Article twins alone leave a model with a flat bag of pages and no map. The category, author, and series surfaces are what answer "what does this site cover, and who writes it" - and they get `.md` twins from the same handler you already wired up in Step 6:
+
+| URL | Served by |
+| --- | --- |
+| `/articles/my-post.md` | article twin |
+| `/articles/category/campaigns.md` | `getCategoryMarkdown` |
+| `/articles/authors/jane-doe.md` | `getAuthorMarkdown` |
+| `/articles/series/new-gm.md` | `getSeriesMarkdown` |
+
+`createArticleMarkdownHandler` dispatches on the slug prefix, so no extra routes are needed. If you build responses yourself, call `getMarkdownTwinResponse(slug, config, { headers })` instead of `getArticleMarkdownResponse`.
+
+The author twin carries the bio, `promise`, `servesWho`, `knowsAbout`, `credentials`, `proof`, and `originStory` from `AuthorProfile` - the "who is this and why trust them" context that otherwise lives only inside React components, and the exact question asked before anything they wrote gets cited.
+
+### Measuring AI crawler traffic
+
+Markdown twin requests all pass through one function, which makes it the only place a site can see whether any of this is working:
+
+```ts
+// config/articles.ts
+export const siteConfig: ArticlesConfig = {
+  siteUrl: 'https://yoursite.com',
+  siteName: 'Your Site',
+  aiCrawlDefault: true,
+  onAiCrawl: ({ slug, crawler, userAgent }) => {
+    posthog.capture('ai_crawl', { slug, crawler, userAgent })
+  },
+}
+```
+
+`crawler` is a name from the exported `AI_CRAWLERS` list (GPTBot, ClaudeBot, PerplexityBot, CCBot, OAI-SearchBot, Bytespider, and others) or `'unknown'`. The payload carries no PII. A handler that throws is reported through `setArticlesErrorHandler` and never breaks the response. `matchAiCrawler(userAgent)` is exported if you need the same classification elsewhere.
+
+The same list drives `getAiRobotsTxtRules`, so your robots rules and your telemetry can never disagree about what counts as an AI crawler.
+
+### Shared entity vocabulary
+
+Free-text `about` names fragment fast - "Pathfinder", "pathfinder", and "PF2e" become three entities across a corpus, which defeats the point of the field. Register them once:
+
+```ts
+export const siteConfig: ArticlesConfig = {
+  entities: {
+    pathfinder: { name: 'Pathfinder', sameAs: 'https://www.wikidata.org/wiki/Q1194077' },
+    'dnd-5e': { name: 'Dungeons & Dragons 5th Edition', sameAs: 'https://www.wikidata.org/wiki/Q1379818' },
+  },
+}
+```
+
+```markdown
+---
+about:
+  - pathfinder
+  - name: Session Zero
+---
+```
+
+A bare string resolves through the registry; an inline object still works unchanged. Unregistered strings render as plain names and `validateArticles` warns (`unknown-entity`).
+
+### Title length
+
+Google truncates a result title around 60 characters. The default template appends your site name to every page, which spends that budget whether or not anyone searches for your brand:
+
+```ts
+export const siteConfig: ArticlesConfig = {
+  siteUrl: 'https://yoursite.com',
+  siteName: 'Your Site',
+  titleTemplate: '{title}', // drop the suffix entirely
+}
+```
+
+`{title}` and `{siteName}` are both available, so `'{siteName}: {title}'` and any other arrangement work too. The template applies to article, category, series, and author pages alike.
+
+Check whether this is costing you: `validateArticles` warns with `effective-title-too-long` when the rendered title exceeds 60 characters and `effective-description-too-long` when the rendered meta description exceeds 160. Both measure what a search result actually shows - `titleTemplate` applied to `searchTitle ?? title`, and `searchDescription ?? excerpt` - not just the optional override fields.
+
+### Freshness
+
+`lastmod` in frontmatter always wins. `lastmodFallback` decides what an article that never declared one reports as `dateModified`:
+
+| Value | Behavior |
+| --- | --- |
+| `'published'` (default) | Reuse the publish date. An article edited three times reports its original date forever. |
+| `'none'` | Omit `dateModified` rather than repeat a stale date. |
+| `'fileMtime'` | Read the article file's mtime. |
+
+`'fileMtime'` is accurate locally, but a CI runner that clones fresh sets every file's mtime to the checkout time - which would report your whole corpus as updated today. Only use it where the build preserves mtimes.
+
+### Cross-site author identity
+
+For an author publishing across several sites, set `identityUrl` to **one** canonical URL, identical on every site. The Person `@id` derives from it, so all of them resolve to a single entity instead of one per site. List the other sites' author pages in `sameAs`:
+
+```ts
+authors: {
+  'jane-doe': {
+    name: 'Jane Doe',
+    slug: 'jane-doe',
+    bio: 'Writes about campaigns.',
+    identityUrl: 'https://main-site.com/about', // identical on every site
+    sameAs: [
+      'https://second-site.com/articles/authors/jane-doe',
+      'https://third-site.com/articles/authors/jane-doe',
+    ],
+  },
+}
+```
+
+**Use `identityUrl`, not `url`.** `url` also drives byline link targets, the author page's Open Graph URL, and its `CollectionPage` URL - pointing it at another domain sends your readers off-site and hands this site's author page a canonical belonging to a different one. `identityUrl` is used only for the `@id` and changes nothing a reader sees. It falls back to `url`, then to this site's own author page, so sites that configure neither keep a per-site identity.
+
+Explicit `sameAs` merges with the links derived from `social` and is deduped. Pair it with `organization.parentOrganization` to link the sites themselves under one publisher:
+
+```ts
+organization: {
+  parentOrganization: { name: 'Example Network', url: 'https://network.example' },
+}
+```
+
+### Comments in structured data
+
+Pass loaded comments to `ArticleSEO` to emit `commentCount` and `comment` entries - real discussion is a quality signal that is otherwise invisible:
+
+```tsx
+<ArticleSEO article={article} articleUrl={articleUrl} siteName={siteConfig.siteName} comments={comments} />
+```
+
+Deleted comments and replies are excluded; only the author name, body, and timestamp are emitted - never the commenter's id.
+
+### llms.txt
+
+`generateLlmsTxt` builds an [llms.txt](https://llmstxt.org) index of every opted-in article, grouped by category and linked to its `.md` twin rather than its HTML page.
+
+```ts
+// app/llms.txt/route.ts
+import { generateLlmsTxt, getAllArticles } from '@fullstackdatasolutions/articles/server'
+import { siteConfig } from '@/config/articles'
+
+export async function GET() {
+  const articles = await getAllArticles(siteConfig)
+  return new Response(generateLlmsTxt(articles, siteConfig), {
+    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
+  })
+}
+```
+
+`generateLlmsFullTxt(articles, config)` returns the same corpus as one document with every article's full markdown body (attribution headers included). It reads every article file, so serve it from a cached or statically generated route rather than on each request.
+
+```ts
+// app/llms-full.txt/route.ts
+export const dynamic = 'force-static'
+
+export async function GET() {
+  const articles = await getAllArticles(siteConfig)
+  return new Response(await generateLlmsFullTxt(articles, siteConfig), {
+    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
+  })
+}
+```
+
+Both list only articles resolved to `aiCrawl: true` - via frontmatter or `aiCrawlDefault`.
+
 **robots.txt - block AI crawlers from non-aiCrawl articles**
 
 Wire `getAiRobotsTxtRules()` into your robots.txt route. It returns disallow rules for known AI crawler agents (GPTBot, Claude-Web, etc.) targeting any article that does not have `aiCrawl: true`. Without this step, AI crawlers can still reach all articles regardless of the frontmatter flag.
@@ -1383,7 +1695,7 @@ import { getAiRobotsTxtRules } from '@fullstackdatasolutions/articles/server'
 
 export async function GET() {
   const siteUrl = 'https://yoursite.com'
-  const aiRules = await getAiRobotsTxtRules()
+  const aiRules = await getAiRobotsTxtRules(siteConfig)
 
   const robotsTxt = `User-agent: *
 Allow: /