@fullstackdatasolutions/articles 1.3.1 → 1.4.0

package/CHANGELOG.md

@@ -5,6 +5,16 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.4.0] - 2026-08-23
+
+### Added
+
+- **`ArticlesConfig.listingTwinNoindex`** - set `false` to stop serving the `category/`, `authors/`, and `series/` listing twins with `X-Robots-Tag: noindex`. Defaults to `true`. Only the listing twins are configurable, because a listing twin is a generated index of a category or author rather than a copy of one page, so a site may have a legitimate reason to let it rank. Article twins are not configurable (see below).
+
+### Changed
+
+- **Markdown twins are served `X-Robots-Tag: noindex`.** A twin is a second representation of a page that is already indexed, so on a large corpus it silently doubles the crawlable URL count with near-duplicate content - a site with 1,300 articles publishes 1,300 more URLs whose text is the same article. Nothing previously kept them out of the index: the responses carried no robots directive, `robots.txt` has no reason to disallow them, and `Cache-Control: public` invites caching. Article twins are now **always** `noindex` - an article twin is the same text as exactly one HTML article, so indexing it can only ever split that article's own signal across two URLs, and there is no configuration under which that is desirable. Listing twins default to `noindex` and can be opted out via `listingTwinNoindex`. In both cases `noindex` rather than a `robots.txt` disallow, because AI crawlers still need to *fetch* these - a disallow would block the fetch that is the entire point of publishing them, while `noindex` only removes them from search results. This belongs in the package rather than the consuming app because `/articles/[slug].md` is typically a rewrite to a route handler, so a host-level header rule keyed on `.md` is matched against the pre-rewrite path and never fires - a footgun worth closing here once instead of in every consuming site. No API removed; sites currently relying on twins appearing in search results will see them drop out, which was not a supported use.
+
 ## [1.3.1] - 2026-08-23
 
 ### Fixed

package/README.md

@@ -1499,6 +1499,18 @@ export const siteConfig: ArticlesConfig = {
 }
 ```
 
+**Response headers.** Twins are served with `Link: <html-url>; rel="canonical"` pointing back at the article they represent, and `X-Robots-Tag: noindex` so they do not compete with that article in a search index - on a large corpus they would otherwise double the crawlable URL count with near-duplicate content. `noindex` rather than a `robots.txt` disallow, because AI crawlers still need to fetch them; a disallow would block the fetch that is the point of publishing them. Do not try to set this from the consuming app: `/articles/[slug].md` is normally a rewrite, so a host-level header rule keyed on `.md` is matched against the pre-rewrite path and never fires.
+
+Article twins are always `noindex` - an article twin is the same text as exactly one HTML article, so indexing it can only split that article's own signal across two URLs. The `category/`, `authors/`, and `series/` listing twins default to `noindex` but can be opted out, since a listing twin is a generated index rather than a copy of one page:
+
+```ts
+export const siteConfig: ArticlesConfig = {
+  siteUrl: 'https://yoursite.com',
+  siteName: 'Your Site',
+  listingTwinNoindex: false, // let category/author/series twins rank
+}
+```
+
 **Attribution header.** Frontmatter is stripped before the twin is served, so the markdown response is prefixed with the article's title, excerpt, canonical source URL, dates, author, and site name - the context a model needs to cite the article rather than reproduce it anonymously:
 
 ```markdown