@ultimat3/seo 27.8.1 → 27.9.0

package/CLAUDE.md

@@ -46,6 +46,15 @@ Tier 1. May import `@ultimat3/core`, `@ultimat3/schema`, `@ultimat3/i18n`. Nothi
   which costs the alternates that WERE right. `defaultLocaleUrl` answers `undefined` unless the
   default locale's own URL is among the ones this route emits — the same explicit-fallback shape
   `meta.ts`'s `hreflangSet` takes. Never re-derive it from `path`.
+- **No `sections`, no byte changes.** The unsectioned sitemap is pinned by hash in
+  `sitemap-sections.test.ts`. Sorting, a child's own index `<lastmod>` and `<name>-<n>` naming
+  belong to the sectioned path only; a fix that must reach both paths re-pins the hash on purpose.
+- **A page's `lastmod` never reads the ambient zone, and a future one is clamped.**
+  `sitemapDatetime` refuses a zoneless time before `Date.parse` can read it in the process's zone.
+  Refusing a future date would fail a whole enumeration for one skewed row; clamping to `now` is
+  the true upper bound. A route's `lastmod` string is the caller's and is written as given.
+- **The hreflang cluster, `x-default` and the `<loc>`s read ONE trimmed list** — the page's
+  `locales`, minus what `disallow` blocks. Never trim one of the three alone.
 - **Errors name the file, not the URL.** `RouteRecord.file` is in every cause and every fix; an agent must be able to open the source without guessing.
 - **Fail closed, and core reads the key.** `isIndexable()` is `environment === 'production'` and
   nothing else — `staging`, a laptop, a typo and an unset variable all disallow. `ULTIMATE_ENV` has

package/README.md

@@ -18,6 +18,7 @@ enforced where they are built: each builder throws.
 | `X_SEO_CANONICAL_MISMATCH` | `meta.canonical` does not resolve to the route's own URL | a wrong canonical de-indexes the page in favour of another |
 | `X_LD_INVALID` | a JSON-LD node missing a required schema.org field | invalid structured data drops the rich result silently |
 | `X_SITEMAP_TOO_LARGE` | the sitemap index exceeds 50,000 files | past the protocol limit the whole sitemap is discarded |
+| `X_SITEMAP_ENTRY_INVALID` | a page's `lastmod` is not a `Date` or ISO 8601 with a zone, or its `locales` names a locale the sitemap is not built for | a wrong date teaches a crawler to distrust every date in the file; a typo'd locale drops the page from one silently |
 
 Performance budgets are **not** here: `x verify`'s `budgets` step is `@ultimat3/cli`'s
 `checkBudgets`, over the route manifest and the build's own stats, and it throws
@@ -47,8 +48,11 @@ a job boundary the class is gone and the `code` is what survives — match on th
 | `meta.ts` | the metadata model, `renderMeta()` → head tags: title template, canonical, robots, `og:*`, `twitter:*`, hreflang + `x-default`, `theme-color` per colour scheme. **`robots: { index: false }` withdraws every `og:*`, `article:*` and `twitter:*` tag** — declared or derived; there is no `social` switch |
 | `validate.ts` | the build gate — `validateMeta()` (`--json`-shaped) and `assertMeta()` |
 | `ld.ts` | typed JSON-LD builders; required fields are required in the **input type** |
-| `sitemap.ts` | `buildSitemap()` from the route table + each route's `prerender()`, per-locale alternates, automatic index splitting past 50k — the index stays `/sitemap.xml`, the parts are `/sitemaps/<n>.xml` (`SITEMAP_PARTS_DIR`) |
-| `robots.ts` | `buildRobots()`, environment-aware and fail-closed; `disallow` reaches **every** group, and a `User-agent: *` group is emitted for it when none is declared |
+| `sitemap.ts` | `buildSitemap()` from the route table + each route's `prerender()`, per-locale alternates. One urlset at `/sitemap.xml`; past a cap (`SITEMAP_MAX_URLS` 50,000, `SITEMAP_MAX_BYTES` 50 MB) the index stays `/sitemap.xml` and the parts are `/sitemaps/<n>.xml` (`SITEMAP_PARTS_DIR`). With `sections` it is always an index of `/sitemaps/<name>-<n>.xml` — see below |
+| `sitemap-sections.ts` | `sectionOf(path, sections)`: which named section a page files under (`/blog/**`, `/blog/*`, exact), first match wins, else `pages` |
+| `sitemap-datetime.ts` | `sitemapDatetime(value, now)`: a page's `lastmod` as W3C Datetime in UTC, or `undefined` — a zoneless time is not a date here |
+| `sitemap-xml.ts` | the `<urlset>` and `<sitemapindex>` XML, and `paginate()` — files that each fit both caps |
+| `robots.ts` | `buildRobots()`, environment-aware and fail-closed; `disallow` reaches **every** group, and a `User-agent: *` group is emitted for it when none is declared |. `disallowMatcher(rules)` reads those rules as a crawler does — what `buildSitemap({ disallow })` leaves out
 | `rss.ts` | `buildFeed()` → RSS 2.0 + Atom + JSON Feed from one item list. Channel `author`/`copyright`/`icon` → Atom `<author>`/`<rights>`/`<icon>`; an item `author` → Atom `<author>`, RSS `<author>` when it has an email, `<dc:creator>` when it does not; an item `image` → Atom `<link rel="enclosure">`, RSS `<media:content medium="image">`. Extra RSS namespaces are declared only when used |
 | `feed-dates.ts` | the one place a feed timestamp is parsed or formatted — an item date that will not parse is *absent*, never `Invalid Date` and never a crash |
 | `images.ts` | `srcset` widths (never past the intrinsic width or `MAX_IMAGE_WIDTH`, 8192), modern formats before the original — `DEFAULT_FORMATS` is what the built-in driver encodes (`webp`); pass `formats: FORMAT_ORDER` for AVIF behind a CDN driver — inlined intrinsic dimensions, and `parseImageQuery()` — reads a minted URL back into a transform request |
@@ -64,6 +68,33 @@ ld.Article({ headline: 'Ship it', author: { name: 'Ada' } });
 A missing `datePublished` is a compile error, not a Search Console warning three
 weeks later.
 
+## Sitemap: pages, sections, caps
+
+```ts
+import { buildSitemap, type RouteRecord } from '@ultimat3/seo';
+
+declare const routes: readonly RouteRecord[];
+
+await buildSitemap(routes, {
+  baseUrl: 'https://notificado.co',
+  locales: ['es-co', 'en'],
+  defaultLocale: 'es-co',
+  maxUrls: 1000,                                      // page size; default 50,000
+  sections: [{ name: 'blog', match: '/blog/**' }],    // omitted: one urlset
+  disallow: ['/panel'],                               // robots.txt rules: never listed
+});
+// → index /sitemap.xml · files /sitemaps/pages-1.xml, /sitemaps/blog-1.xml, /sitemaps/blog-2.xml …
+```
+
+| Rule | |
+|---|---|
+| a page | `RouteRecord.prerender()` answers `'/blog/a'` or a `SitemapPage` — `{ path, lastmod?, locales? }`. `expandRoutePages()` reads both; `expandRoute()` still answers paths |
+| `lastmod` | a `Date` or ISO 8601 with a zone, written in UTC. No zone, or no date: `X_SITEMAP_ENTRY_INVALID`. After `now`: clamped to `now` — one row from a clock a second ahead must not fail every page, and a future date is one a crawler stops believing |
+| `locales` | per page, else `RouteRecord.locales`, else all. One `<url>` per listed locale, hreflang among those only; `x-default` only when the default is listed; a page left in one locale of several carries no alternates; `[]` lists it nowhere. An unknown name is `X_SITEMAP_ENTRY_INVALID` |
+| sections | matched on the unprefixed path, so a page and its locale copies share a file. `pages` (reserved) first, then declaration order; inside one, sorted by path. Numbered from 1; an empty one writes nothing; the index `<lastmod>` of a child is the newest inside it (`SitemapFile.lastmod`) |
+| no sections | byte-for-byte the output before sections existed — `sitemap-sections.test.ts` pins the hash |
+| names | `sections` and `maxUrls` are refused by `@ultimat3/core`'s `sitemapSectionIssues`, the validator `defineConfig` runs for `seo.sitemap` |
+
 ## robots.txt is fail-closed
 
 Only the literal string `production` in `ULTIMATE_ENV` / `NODE_ENV` opts a deploy

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/seo",
-  "version": "27.8.1",
+  "version": "27.9.0",
   "description": "Enforced SEO: typed meta, JSON-LD, sitemap, robots, feeds, responsive images, perf budgets",
   "license": "MIT",
   "type": "module",
@@ -32,7 +32,7 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "27.8.1",
-    "@ultimat3/schema": "27.8.1"
+    "@ultimat3/core": "27.9.0",
+    "@ultimat3/schema": "27.9.0"
   }
 }

package/src/errors.ts

@@ -19,6 +19,7 @@ export const SEO_ERROR_CODES = {
   sitemapTooLarge: 'X_SITEMAP_TOO_LARGE',
   imageQueryInvalid: 'X_IMAGE_QUERY_INVALID',
   linkInvalid: 'X_SEO_LINK_INVALID',
+  sitemapEntryInvalid: 'X_SITEMAP_ENTRY_INVALID',
 } as const;
 
 export type SeoErrorCode = (typeof SEO_ERROR_CODES)[keyof typeof SEO_ERROR_CODES];
@@ -42,6 +43,9 @@ registerErrorCodes({
   X_SITEMAP_TOO_LARGE: { title: 'sitemap exceeds the 50,000-entry protocol limit' },
   X_IMAGE_QUERY_INVALID: { title: 'an image transform query parameter is present but unusable' },
   X_SEO_LINK_INVALID: { title: 'a route meta link is unsafe or would fetch nothing' },
+  X_SITEMAP_ENTRY_INVALID: {
+    title: 'a sitemap entry carries a lastmod or a locale the sitemap cannot write',
+  },
 });
 
 export interface SeoErrorInit {
@@ -148,6 +152,19 @@ export function sitemapTooLarge(count: number, max: number): SeoError {
   });
 }
 
+/**
+ * A page a sitemap was asked to list with a `lastmod` or a locale it cannot write. `file` is the
+ * route's source — where the `prerender()` that answered the entry lives.
+ */
+export function sitemapEntryInvalid(file: string, path: string, problem: string): SeoError {
+  return new SeoError({
+    code: SEO_ERROR_CODES.sitemapEntryInvalid,
+    cause: `${file} lists "${path}" in the sitemap with ${problem}`,
+    fix: `x verify --json   # after prerender() in ${file} answers lastmod as a Date or an ISO 8601 string with a zone (row.updatedAt), and locales naming only locales the app routes`,
+    meta: { file, path, problem },
+  });
+}
+
 /**
  * `parseImageQuery`'s only refusal: a `?w=`/`?q=`/`?f=` value present but not usable — serving
  * the untransformed original against a URL that asked for a size would be the layout shift this

package/src/index.ts

@@ -13,6 +13,7 @@ export {
   notImplementedDriver,
   SEO_ERROR_CODES,
   SeoError,
+  sitemapEntryInvalid,
   sitemapTooLarge,
 } from './errors';
 export type {
@@ -98,9 +99,9 @@ export {
   TITLE_MAX_LENGTH,
 } from './meta';
 export type { RobotsConfig, RobotsGroup } from './robots';
-export { buildRobots, isIndexable } from './robots';
-export type { ChangeFreq, RouteRecord, Surface } from './routes';
-export { expandRoute, indexableRoutes, isDynamic } from './routes';
+export { buildRobots, disallowMatcher, isIndexable } from './robots';
+export type { ChangeFreq, RouteRecord, SitemapPage, Surface } from './routes';
+export { expandRoute, expandRoutePages, indexableRoutes, isDynamic } from './routes';
 export type { BuildFeedOptions, Feed, FeedAuthor, FeedChannel, FeedItem } from './rss';
 export { buildFeed } from './rss';
 export type {
@@ -114,10 +115,14 @@ export {
   buildSitemap,
   chunk,
   SITEMAP_INDEX_MAX_FILES,
+  SITEMAP_MAX_BYTES,
   SITEMAP_MAX_URLS,
   SITEMAP_PARTS_DIR,
   sitemapUrls,
 } from './sitemap';
+export { sitemapDatetime } from './sitemap-datetime';
+export type { SitemapSection } from './sitemap-sections';
+export { sectionOf } from './sitemap-sections';
 export type { MetaIssue, MetaValidationReport, ValidateMetaOptions } from './validate';
 export { assertMeta, validateMeta } from './validate';
 export { absoluteUrl, escapeXml } from './xml';

package/src/robots.ts

@@ -96,3 +96,27 @@ export function buildRobots(config: RobotsConfig): string {
 
   return `${lines.join('\n').trimEnd()}\n`;
 }
+
+/**
+ * `rule` as a crawler reads it: a PREFIX of the path, `*` standing for any run of characters and a
+ * trailing `$` anchoring the end. An empty rule disallows nothing.
+ */
+function disallowPattern(rule: string): RegExp | undefined {
+  if (rule === '') return undefined;
+  const anchored = rule.endsWith('$');
+  const body = (anchored ? rule.slice(0, -1) : rule)
+    .split('*')
+    .map((literal) => literal.replace(/[.+?^${}()|[\]\\]/g, '\\$&'))
+    .join('.*');
+  return new RegExp(`^${body}${anchored ? '$' : ''}`);
+}
+
+/**
+ * A test for "do these `Disallow:` rules keep a crawler out of this path". What the sitemap asks
+ * before it lists a URL: one that `robots.txt` blocks is reported as "submitted URL blocked by
+ * robots.txt", and the two files are written from one config.
+ */
+export function disallowMatcher(rules: readonly string[]): (path: string) => boolean {
+  const patterns = rules.flatMap((rule) => disallowPattern(rule) ?? []);
+  return (path) => patterns.some((pattern) => pattern.test(path));
+}

package/src/routes.ts

@@ -10,6 +10,25 @@ export type Surface = 'site' | 'app' | 'api';
 
 export type ChangeFreq = 'always' | 'hourly' | 'daily' | 'weekly' | 'monthly' | 'yearly' | 'never';
 
+/**
+ * One concrete page of a route, as a sitemap lists it. A bare string is the same thing with nothing
+ * to add: `'/blog/a'` is `{ path: '/blog/a' }`.
+ */
+export interface SitemapPage {
+  /** The concrete path, WITHOUT a locale prefix: `/blog/a`. */
+  readonly path: string;
+  /**
+   * When this page last changed: a `Date`, or ISO 8601 with an explicit zone (`2026-10-01`,
+   * `2026-10-01T15:04:05Z`). Written as W3C Datetime in UTC; wins over the route's own `lastmod`.
+   */
+  readonly lastmod?: string | Date;
+  /**
+   * The locales this page is LISTED in — one `<url>` each, cross-linked with hreflang among
+   * themselves only. Omitted: every locale the sitemap is built for. Empty: listed nowhere.
+   */
+  readonly locales?: readonly string[];
+}
+
 export interface RouteRecord {
   /** URL pattern, e.g. `/blog/:slug`. */
   path: string;
@@ -18,8 +37,13 @@ export interface RouteRecord {
   surface: Surface;
   render: RenderMode;
   meta?: RouteMeta;
-  /** Concrete paths for a dynamic route, from `defineRoute({ prerender })`. */
-  prerender?: () => readonly string[] | Promise<readonly string[]>;
+  /** The route's concrete pages, from `defineRoute({ prerender })` — required for a dynamic one. */
+  prerender?: () => readonly (string | SitemapPage)[] | Promise<readonly (string | SitemapPage)[]>;
+  /**
+   * The locales every page of this route is listed in, unless the page names its own
+   * (`SitemapPage.locales`). Omitted: every locale the sitemap is built for.
+   */
+  locales?: readonly string[];
   /** Keep out of the sitemap and emit `noindex`. */
   noindex?: boolean;
   /**
@@ -47,9 +71,20 @@ export function indexableRoutes(routes: readonly RouteRecord[]): readonly RouteR
   );
 }
 
+/**
+ * Every concrete page a route resolves to, each in the one object form. A route that declares
+ * `prerender()` is the pages it answers — with or without params: a page with none may still say
+ * its own `lastmod` and `locales` there. Without one, a static path is its one page and a dynamic
+ * one is none.
+ */
+export async function expandRoutePages(route: RouteRecord): Promise<readonly SitemapPage[]> {
+  if (route.prerender === undefined) return isDynamic(route.path) ? [] : [{ path: route.path }];
+  return (await route.prerender()).map((page) =>
+    typeof page === 'string' ? { path: page } : page,
+  );
+}
+
 /** Every concrete URL a route resolves to, expanding `prerender()`. */
 export async function expandRoute(route: RouteRecord): Promise<readonly string[]> {
-  if (!isDynamic(route.path)) return [route.path];
-  if (route.prerender === undefined) return [];
-  return await route.prerender();
+  return (await expandRoutePages(route)).map((page) => page.path);
 }

package/src/sitemap-datetime.ts

@@ -0,0 +1,33 @@
+// Single responsibility: one page's `lastmod`, as the W3C Datetime a sitemap writes — or the
+// answer that it is not a date. No ambient zone is ever read: a time without one is refused.
+
+/**
+ * ISO 8601 as a sitemap accepts it, with the zone REQUIRED on a time. `Date.parse` alone is not the
+ * test: it reads `2026-10-01T10:00:00` and `October 1, 2026` in the process's own zone, so one row
+ * would be two instants on a laptop in Bogotá and in a container on UTC.
+ */
+const ISO_ZONED =
+  /^\d{4}-\d{2}-\d{2}(?:T\d{2}:\d{2}(?::\d{2}(?:\.\d{1,9})?)?(?:Z|[+-]\d{2}:\d{2}))?$/;
+
+/** What a refusal quotes back, so the fix names the forms that are accepted. */
+export const SITEMAP_DATETIME_FORMS =
+  'a Date or an ISO 8601 date with an explicit zone (2026-10-01, 2026-10-01T15:04:05Z, 2026-10-01T10:04:05-05:00)';
+
+/**
+ * `value` in UTC (`2026-10-01T15:04:05.000Z`), or `undefined` when it names no instant.
+ *
+ * A moment after `now` is CLAMPED to `now`, never refused and never written as it stands. Written,
+ * it is a date a crawler learns to distrust for the whole file. Refused, one row stamped by a
+ * database a second ahead of this process fails the enumeration for every page. Clamped, it says
+ * the one thing that is certainly true of a change that already happened: no later than now.
+ */
+export function sitemapDatetime(value: unknown, now: Date): string | undefined {
+  const time =
+    value instanceof Date
+      ? value.getTime()
+      : typeof value === 'string' && ISO_ZONED.test(value)
+        ? Date.parse(value)
+        : Number.NaN;
+  if (!Number.isFinite(time)) return undefined;
+  return new Date(Math.min(time, now.getTime())).toISOString();
+}

package/src/sitemap-sections.ts

@@ -0,0 +1,50 @@
+// Single responsibility: which named section of a sitemap index a page files under. The shape and
+// its refusals are `@ultimat3/core`'s (`seo.sitemap.sections`), so `defineConfig` and a direct
+// `buildSitemap` call reject the same things in the same words.
+
+import {
+  assert,
+  SITEMAP_DEFAULT_SECTION,
+  type SitemapSectionConfig,
+  sectionPatterns,
+  sitemapSectionIssues,
+} from '@ultimat3/core';
+
+export type SitemapSection = SitemapSectionConfig;
+
+/** A page's path as a pattern reads it: no query, no fragment, no trailing slash. */
+function segmentsOf(path: string): readonly string[] {
+  const bare = path.replace(/[?#].*$/, '').replace(/\/+$/, '');
+  return bare === '' ? [] : bare.slice(1).split('/');
+}
+
+function matches(pattern: string, page: readonly string[]): boolean {
+  const wanted = segmentsOf(pattern);
+  const subtree = wanted.at(-1) === '**';
+  const fixed = subtree ? wanted.slice(0, -1) : wanted;
+  // `/blog/**` is `/blog` and everything below it: a section's own landing page files with it.
+  if (subtree ? page.length < fixed.length : page.length !== fixed.length) return false;
+  return fixed.every((segment, index) => segment === '*' || segment === page[index]);
+}
+
+/**
+ * The section `path` files under: the first one, in declaration order, with a pattern that matches
+ * — else the default. `path` is UNPREFIXED, so a page and its locale copies share one file.
+ */
+export function sectionOf(path: string, sections: readonly SitemapSection[]): string {
+  const page = segmentsOf(path);
+  for (const section of sections) {
+    if (sectionPatterns(section).some((pattern) => matches(pattern, page))) return section.name;
+  }
+  return SITEMAP_DEFAULT_SECTION;
+}
+
+/** Refuses what `defineConfig` refuses, for a caller that built the options by hand. */
+export function assertSections(sections: readonly SitemapSection[], pageSize: number): void {
+  const issues = sitemapSectionIssues(sections, pageSize);
+  assert(
+    issues.length === 0,
+    issues.join('; '),
+    "pass sections as [{ name: 'blog', match: '/blog/**' }] — a lower-case slug and a path pattern — and maxUrls from 1 to 50000",
+  );
+}

package/src/sitemap-xml.ts

@@ -0,0 +1,134 @@
+// Single responsibility: the XML of a sitemap — one `<urlset>`, one `<sitemapindex>` — and the
+// split of a list of URLs into files that each fit BOTH protocol caps, entries and bytes.
+
+import { assert } from '@ultimat3/core';
+import type { ChangeFreq } from './routes';
+import { attributes, escapeXml } from './xml';
+
+export interface SitemapAlternate {
+  hreflang: string;
+  href: string;
+}
+
+export interface SitemapUrl {
+  loc: string;
+  lastmod?: string;
+  changefreq?: ChangeFreq;
+  priority?: number;
+  alternates?: readonly SitemapAlternate[];
+}
+
+const HEAD = '<?xml version="1.0" encoding="UTF-8"?>\n';
+const SITEMAP_NS = ' xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"';
+const XHTML_NS = ' xmlns:xhtml="http://www.w3.org/1999/xhtml"';
+/** The envelope around the `<url>` entries, at its largest: every byte of it is ASCII. */
+const ENVELOPE_BYTES = `${HEAD}<urlset${SITEMAP_NS}${XHTML_NS}>\n\n</urlset>\n`.length;
+
+function urlXml(url: SitemapUrl): string {
+  const parts = [`    <loc>${escapeXml(url.loc)}</loc>`];
+  if (url.lastmod !== undefined) parts.push(`    <lastmod>${escapeXml(url.lastmod)}</lastmod>`);
+  if (url.changefreq !== undefined) parts.push(`    <changefreq>${url.changefreq}</changefreq>`);
+  if (url.priority !== undefined) {
+    parts.push(`    <priority>${url.priority.toFixed(1)}</priority>`);
+  }
+  for (const alternate of url.alternates ?? []) {
+    parts.push(
+      `    <xhtml:link${attributes({ rel: 'alternate', hreflang: alternate.hreflang, href: alternate.href })}/>`,
+    );
+  }
+  return `  <url>\n${parts.join('\n')}\n  </url>`;
+}
+
+export function urlSetXml(urls: readonly SitemapUrl[]): string {
+  const needsXhtml = urls.some((url) => (url.alternates?.length ?? 0) > 0);
+  const ns = needsXhtml ? `${SITEMAP_NS}${XHTML_NS}` : SITEMAP_NS;
+  return `${HEAD}<urlset${ns}>\n${urls.map(urlXml).join('\n')}\n</urlset>\n`;
+}
+
+export interface SitemapIndexEntry {
+  /** Absolute. */
+  readonly loc: string;
+  readonly lastmod?: string | undefined;
+}
+
+export function indexXml(entries: readonly SitemapIndexEntry[]): string {
+  const body = entries
+    .map((entry) => {
+      const lastmod =
+        entry.lastmod === undefined ? '' : `\n    <lastmod>${escapeXml(entry.lastmod)}</lastmod>`;
+      return `  <sitemap>\n    <loc>${escapeXml(entry.loc)}</loc>${lastmod}\n  </sitemap>`;
+    })
+    .join('\n');
+  return `${HEAD}<sitemapindex${SITEMAP_NS}>\n${body}\n</sitemapindex>\n`;
+}
+
+export function chunk<T>(items: readonly T[], size: number): T[][] {
+  // The loop advances by `size`, so a non-positive one never moves the cursor: `maxUrls: 0` in a
+  // route config turned a build into an infinite loop allocating empty slices until the box ran
+  // out of memory. A fractional size is refused for a quieter reason — `slice` truncates it, so
+  // the groups silently stop being the size that was asked for.
+  assert(
+    Number.isSafeInteger(size) && size > 0,
+    `a chunk size must be a positive integer, got ${String(size)}: a non-positive step never advances and the loop cannot end`,
+    'pass a positive integer — buildSitemap(routes, { baseUrl, maxUrls: 50000 }), the sitemaps.org bound SITEMAP_MAX_URLS already carries',
+  );
+  const out: T[][] = [];
+  for (let index = 0; index < items.length; index += size) {
+    out.push(items.slice(index, index + size));
+  }
+  return out;
+}
+
+const encoder = new TextEncoder();
+const bytesOf = (text: string): number => encoder.encode(text).length;
+
+/** One group of at most `maxUrls`, cut again wherever its XML would pass `maxBytes`. */
+function splitByBytes(urls: readonly SitemapUrl[], maxBytes: number): SitemapUrl[][] {
+  const out: SitemapUrl[][] = [];
+  let current: SitemapUrl[] = [];
+  let size = ENVELOPE_BYTES;
+  for (const url of urls) {
+    const cost = bytesOf(urlXml(url)) + 1;
+    // A lone URL larger than the cap still gets a file: there is no smaller unit to write.
+    if (current.length > 0 && size + cost > maxBytes) {
+      out.push(current);
+      current = [];
+      size = ENVELOPE_BYTES;
+    }
+    current.push(url);
+    size += cost;
+  }
+  if (current.length > 0) out.push(current);
+  return out;
+}
+
+export interface SitemapPageXml {
+  readonly urls: readonly SitemapUrl[];
+  readonly xml: string;
+}
+
+/**
+ * `urls` as the files that hold them: at most `maxUrls` entries and at most `maxBytes` of XML each,
+ * in order. No URL at all is ONE empty urlset — `/sitemap.xml` always answers a document.
+ */
+export function paginate(
+  urls: readonly SitemapUrl[],
+  maxUrls: number,
+  maxBytes: number,
+): readonly SitemapPageXml[] {
+  if (urls.length === 0) return [{ urls, xml: urlSetXml(urls) }];
+  const pages: SitemapPageXml[] = [];
+  for (const group of chunk(urls, maxUrls)) {
+    const xml = urlSetXml(group);
+    // UTF-8 is at most three bytes per UTF-16 unit: a file this short cannot pass the cap, and the
+    // common case never pays for an encode.
+    if (xml.length * 3 <= maxBytes || bytesOf(xml) <= maxBytes) {
+      pages.push({ urls: group, xml });
+      continue;
+    }
+    for (const part of splitByBytes(group, maxBytes)) {
+      pages.push({ urls: part, xml: urlSetXml(part) });
+    }
+  }
+  return pages;
+}

package/src/sitemap.ts

@@ -1,47 +1,51 @@
-// Sitemap generation from the route table. Dynamic routes contribute the URLs
-// their `prerender()` enumerates, so the sitemap can never drift from what the
-// build actually produced. Splits into an index past the 50,000-URL protocol cap.
+// Sitemap generation from the route table. Dynamic routes contribute the pages their `prerender()`
+// enumerates, so the sitemap can never drift from what the build actually produced. One urlset —
+// split into an index past a protocol cap — or, with `sections`, an index of named, paginated parts.
 
-import { assert } from '@ultimat3/core';
-import { sitemapTooLarge } from './errors';
+import { assert, SITEMAP_DEFAULT_SECTION } from '@ultimat3/core';
+import { sitemapEntryInvalid, sitemapTooLarge } from './errors';
 import { hreflangTag } from './locale-tags';
-import { type ChangeFreq, expandRoute, indexableRoutes, type RouteRecord } from './routes';
-import { absoluteUrl, attributes, escapeXml } from './xml';
+import { disallowMatcher } from './robots';
+import { expandRoutePages, indexableRoutes, type RouteRecord, type SitemapPage } from './routes';
+import { SITEMAP_DATETIME_FORMS, sitemapDatetime } from './sitemap-datetime';
+import { assertSections, type SitemapSection, sectionOf } from './sitemap-sections';
+import {
+  indexXml,
+  paginate,
+  type SitemapAlternate,
+  type SitemapPageXml,
+  type SitemapUrl,
+} from './sitemap-xml';
+import { absoluteUrl } from './xml';
 
-/** Both limits are from the sitemaps.org protocol. */
+export type { SitemapAlternate, SitemapUrl } from './sitemap-xml';
+export { chunk } from './sitemap-xml';
+
+/** All three limits are from the sitemaps.org protocol; the byte one is of the UNCOMPRESSED file. */
 export const SITEMAP_MAX_URLS = 50_000;
 export const SITEMAP_INDEX_MAX_FILES = 50_000;
+export const SITEMAP_MAX_BYTES = 52_428_800;
 
 /**
- * Where the parts of a split sitemap live: `/sitemaps/1.xml`, `/sitemaps/2.xml`, … — one directory,
- * so a running server answers every part from ONE route (`/sitemaps/:file`). `/sitemap-N.xml` needed
- * a parameter inside a segment, which no router here has, and every part 404'd from the web role.
+ * Where the parts of a sitemap index live: `/sitemaps/1.xml`, … for a split, `/sitemaps/blog-1.xml`,
+ * … for a section — one directory, so a running server answers every part from ONE route
+ * (`/sitemaps/:file`). `/sitemap-N.xml` needed a parameter inside a segment, which no router here
+ * has, and every part 404'd from the web role.
  */
 export const SITEMAP_PARTS_DIR = '/sitemaps';
 
-export interface SitemapAlternate {
-  hreflang: string;
-  href: string;
-}
-
-export interface SitemapUrl {
-  loc: string;
-  lastmod?: string;
-  changefreq?: ChangeFreq;
-  priority?: number;
-  alternates?: readonly SitemapAlternate[];
-}
-
 export interface SitemapFile {
-  /** Path relative to the site root: `/sitemap.xml`, or a part, `/sitemaps/1.xml`. */
+  /** Path relative to the site root: `/sitemap.xml`, or a part, `/sitemaps/blog-1.xml`. */
   path: string;
   xml: string;
   urlCount: number;
+  /** The newest `<lastmod>` inside a urlset — what the index says of it. Absent when none has one. */
+  lastmod?: string;
 }
 
 export interface SitemapResult {
   readonly files: readonly SitemapFile[];
-  /** Present only when the URLs did not fit in a single file. */
+  /** Present when `/sitemap.xml` is an index: sections were declared, or one file did not fit. */
   readonly index: SitemapFile | undefined;
   readonly urlCount: number;
 }
@@ -57,8 +61,23 @@ export interface BuildSitemapOptions {
    * emitted at all** — there is no unprefixed URL in the sitemap for it to name.
    */
   defaultLocale?: string;
+  /** `<url>` entries per file — the page size of a section. Defaults to the protocol's 50,000. */
   maxUrls?: number;
+  /** Bytes of XML per file. Defaults to the protocol's 50 MB; a test passes less. */
+  maxBytes?: number;
   lastmod?: string;
+  /**
+   * Named parts. Declared, `/sitemap.xml` is ALWAYS an index — of `/sitemaps/<name>-<n>.xml`, the
+   * pages no section matches under `pages` — whatever the URL count. Omitted or empty: one urlset.
+   */
+  sections?: readonly SitemapSection[];
+  /**
+   * `robots.txt` `Disallow:` rules. A URL one of them matches is not listed, and leaves its
+   * hreflang cluster: a crawler told to stay out of a URL must not be handed it to fetch.
+   */
+  disallow?: readonly string[];
+  /** The moment a page's `lastmod` may not be later than. Omitted: when the sitemap is built. */
+  now?: Date;
 }
 
 function localize(path: string, locale: string, options: BuildSitemapOptions): string {
@@ -85,29 +104,83 @@ function defaultLocaleUrl(
   return localised.find((entry) => entry.locale === options.defaultLocale)?.path;
 }
 
-/** Every concrete URL the route table produces, with per-locale alternates. */
-export async function sitemapUrls(
+/** One page and every `<url>` it is listed as — its locale copies, which file and move together. */
+interface Cluster {
+  /** The page's UNPREFIXED path: what a section matches and what a section sorts by. */
+  readonly page: string;
+  readonly urls: readonly SitemapUrl[];
+}
+
+/**
+ * The locales `page` is listed in, in the SITEMAP's order. A name outside `locales` is refused: a
+ * typo (`en-us` for `en`) would otherwise drop the page from a locale without a word.
+ */
+function listedLocales(
+  route: RouteRecord,
+  page: SitemapPage,
+  locales: readonly string[],
+): readonly string[] {
+  const wanted = page.locales ?? route.locales;
+  if (wanted === undefined) return locales;
+  const unknown = wanted.find((locale) => !locales.includes(locale));
+  if (unknown !== undefined) {
+    throw sitemapEntryInvalid(
+      route.file,
+      page.path,
+      `the locale "${unknown}", which is not one of the routed locales (${locales.join(', ')})`,
+    );
+  }
+  return locales.filter((locale) => wanted.includes(locale));
+}
+
+function lastmodOf(
+  route: RouteRecord,
+  page: SitemapPage,
+  options: BuildSitemapOptions,
+  now: Date,
+): string | undefined {
+  if (page.lastmod === undefined) return route.lastmod ?? options.lastmod;
+  const lastmod = sitemapDatetime(page.lastmod, now);
+  if (lastmod !== undefined) return lastmod;
+  const written =
+    page.lastmod instanceof Date ? 'an invalid Date' : JSON.stringify(String(page.lastmod));
+  throw sitemapEntryInvalid(
+    route.file,
+    page.path,
+    `the lastmod ${written}, which is not ${SITEMAP_DATETIME_FORMS}`,
+  );
+}
+
+async function sitemapClusters(
   routes: readonly RouteRecord[],
   options: BuildSitemapOptions,
-): Promise<readonly SitemapUrl[]> {
-  const urls: SitemapUrl[] = [];
+): Promise<readonly Cluster[]> {
+  const clusters: Cluster[] = [];
   const locales = options.locales ?? [];
+  const disallowed = disallowMatcher(options.disallow ?? []);
+  const now = options.now ?? new Date();
 
   for (const route of indexableRoutes(routes)) {
-    for (const path of await expandRoute(route)) {
+    for (const page of await expandRoutePages(route)) {
+      const path = page.path;
       // Localised once, then read three times — the alternates, the `x-default` candidate and the
       // `<loc>`s below are three questions with one answer, and they drifted apart when each
-      // computed its own.
-      const localised = locales.map((locale) => ({
-        locale,
-        path: localize(path, locale, options),
-      }));
-      const alternates: SitemapAlternate[] = localised.map((entry) => ({
-        // BCP 47 region form (`es-CO`), the spelling `renderMeta` puts in the head: one cluster,
-        // one vocabulary, whether a crawler reads the page or the sitemap.
-        hreflang: hreflangTag(entry.locale),
-        href: absoluteUrl(options.baseUrl, entry.path),
-      }));
+      // computed its own. Trimmed HERE, to the locales the page is listed in and the URLs a
+      // crawler may fetch, so all three read the same trimmed list.
+      const localised = listedLocales(route, page, locales)
+        .map((locale) => ({ locale, path: localize(path, locale, options) }))
+        .filter((entry) => !disallowed(entry.path));
+      // A page left in ONE locale of several has no alternate to name: the cluster is the page.
+      // (A sitemap built for one locale keeps its cluster of one — it always wrote it.)
+      const alone = localised.length === 1 && locales.length > 1;
+      const alternates: SitemapAlternate[] = alone
+        ? []
+        : localised.map((entry) => ({
+            // BCP 47 region form (`es-CO`), the spelling `renderMeta` puts in the head: one cluster,
+            // one vocabulary, whether a crawler reads the page or the sitemap.
+            hreflang: hreflangTag(entry.locale),
+            href: absoluteUrl(options.baseUrl, entry.path),
+          }));
       // `x-default` only when it names a URL THIS sitemap lists. It was the bare `path` for every
       // route: with `locales` set and no `defaultLocale`, every `<loc>` is prefixed and the
       // unprefixed path is one the sitemap never mentions — an hreflang cluster pointing at a URL
@@ -119,77 +192,81 @@ export async function sitemapUrls(
         alternates.push({ hreflang: 'x-default', href: absoluteUrl(options.baseUrl, fallback) });
       }
 
-      const lastmod = route.lastmod ?? options.lastmod;
-      const emitFor = locales.length === 0 ? [path] : localised.map((entry) => entry.path);
-      for (const emitted of emitFor) {
-        urls.push({
-          loc: absoluteUrl(options.baseUrl, emitted),
-          ...(lastmod === undefined ? {} : { lastmod }),
-          ...(route.changefreq === undefined ? {} : { changefreq: route.changefreq }),
-          ...(route.priority === undefined ? {} : { priority: route.priority }),
-          ...(alternates.length === 0 ? {} : { alternates }),
-        });
-      }
+      const lastmod = lastmodOf(route, page, options, now);
+      // No locales at all: the page is its one URL — unless it asked to be listed nowhere.
+      const nowhere = (page.locales ?? route.locales)?.length === 0;
+      const unlocalised = nowhere || disallowed(path) ? [] : [path];
+      const emitFor = locales.length === 0 ? unlocalised : localised.map((entry) => entry.path);
+      const urls = emitFor.map((emitted) => ({
+        loc: absoluteUrl(options.baseUrl, emitted),
+        ...(lastmod === undefined ? {} : { lastmod }),
+        ...(route.changefreq === undefined ? {} : { changefreq: route.changefreq }),
+        ...(route.priority === undefined ? {} : { priority: route.priority }),
+        ...(alternates.length === 0 ? {} : { alternates }),
+      }));
+      if (urls.length > 0) clusters.push({ page: path, urls });
     }
   }
-  return urls;
+  return clusters;
 }
 
-function renderUrlSet(urls: readonly SitemapUrl[]): string {
-  const needsXhtml = urls.some((url) => (url.alternates?.length ?? 0) > 0);
-  const ns = needsXhtml
-    ? ' xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"'
-    : ' xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"';
-  const body = urls
-    .map((url) => {
-      const parts = [`    <loc>${escapeXml(url.loc)}</loc>`];
-      if (url.lastmod !== undefined) parts.push(`    <lastmod>${escapeXml(url.lastmod)}</lastmod>`);
-      if (url.changefreq !== undefined) {
-        parts.push(`    <changefreq>${url.changefreq}</changefreq>`);
-      }
-      if (url.priority !== undefined) {
-        parts.push(`    <priority>${url.priority.toFixed(1)}</priority>`);
-      }
-      for (const alternate of url.alternates ?? []) {
-        parts.push(
-          `    <xhtml:link${attributes({ rel: 'alternate', hreflang: alternate.hreflang, href: alternate.href })}/>`,
-        );
-      }
-      return `  <url>\n${parts.join('\n')}\n  </url>`;
-    })
-    .join('\n');
-  return `<?xml version="1.0" encoding="UTF-8"?>\n<urlset${ns}>\n${body}\n</urlset>\n`;
+/** Every concrete URL the route table produces, with per-locale alternates. */
+export async function sitemapUrls(
+  routes: readonly RouteRecord[],
+  options: BuildSitemapOptions,
+): Promise<readonly SitemapUrl[]> {
+  return (await sitemapClusters(routes, options)).flatMap((cluster) => cluster.urls);
 }
 
-function renderIndex(files: readonly SitemapFile[], options: BuildSitemapOptions): string {
-  const body = files
-    .map((file) => {
-      const loc = escapeXml(absoluteUrl(options.baseUrl, file.path));
-      const lastmod =
-        options.lastmod === undefined
-          ? ''
-          : `\n    <lastmod>${escapeXml(options.lastmod)}</lastmod>`;
-      return `  <sitemap>\n    <loc>${loc}</loc>${lastmod}\n  </sitemap>`;
-    })
-    .join('\n');
-  return `<?xml version="1.0" encoding="UTF-8"?>\n<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n${body}\n</sitemapindex>\n`;
+/** The newest `<lastmod>` among `urls`, as written — compared as instants, not as text. */
+function newestLastmod(urls: readonly SitemapUrl[]): string | undefined {
+  let newest: string | undefined;
+  let at = Number.NEGATIVE_INFINITY;
+  for (const url of urls) {
+    const time = url.lastmod === undefined ? Number.NaN : Date.parse(url.lastmod);
+    if (Number.isFinite(time) && time > at) {
+      at = time;
+      newest = url.lastmod;
+    }
+  }
+  return newest;
 }
 
-export function chunk<T>(items: readonly T[], size: number): T[][] {
-  // The loop advances by `size`, so a non-positive one never moves the cursor: `maxUrls: 0` in a
-  // route config turned a build into an infinite loop allocating empty slices until the box ran
-  // out of memory. A fractional size is refused for a quieter reason — `slice` truncates it, so
-  // the groups silently stop being the size that was asked for.
-  assert(
-    Number.isSafeInteger(size) && size > 0,
-    `a chunk size must be a positive integer, got ${String(size)}: a non-positive step never advances and the loop cannot end`,
-    'pass a positive integer — buildSitemap(routes, { baseUrl, maxUrls: 50000 }), the sitemaps.org bound SITEMAP_MAX_URLS already carries',
-  );
-  const out: T[][] = [];
-  for (let index = 0; index < items.length; index += size) {
-    out.push(items.slice(index, index + size));
-  }
-  return out;
+const fileOf = (path: string, page: SitemapPageXml): SitemapFile => {
+  const lastmod = newestLastmod(page.urls);
+  return {
+    path,
+    xml: page.xml,
+    urlCount: page.urls.length,
+    ...(lastmod === undefined ? {} : { lastmod }),
+  };
+};
+
+const comparePages = (a: Cluster, b: Cluster): number =>
+  a.page < b.page ? -1 : a.page > b.page ? 1 : 0;
+
+/**
+ * The files of a sectioned sitemap, the default section first and the rest in declaration order.
+ * Inside a section pages are SORTED by path: the bytes then depend on which pages exist and never
+ * on the order a query happened to return them in, so a crawler re-reading a page of a section
+ * diffs it against the same neighbours. A section with nothing in it writes no file.
+ */
+function sectionFiles(
+  clusters: readonly Cluster[],
+  sections: readonly SitemapSection[],
+  maxUrls: number,
+  maxBytes: number,
+): readonly SitemapFile[] {
+  const names = [SITEMAP_DEFAULT_SECTION, ...sections.map((section) => section.name)];
+  const filed = new Map<string, Cluster[]>(names.map((name) => [name, []]));
+  for (const cluster of clusters) filed.get(sectionOf(cluster.page, sections))?.push(cluster);
+  return names.flatMap((name) => {
+    const urls = (filed.get(name) ?? []).sort(comparePages).flatMap((cluster) => cluster.urls);
+    if (urls.length === 0) return [];
+    return paginate(urls, maxUrls, maxBytes).map((page, position) =>
+      fileOf(`${SITEMAP_PARTS_DIR}/${name}-${String(position + 1)}.xml`, page),
+    );
+  });
 }
 
 export async function buildSitemap(
@@ -197,6 +274,7 @@ export async function buildSitemap(
   options: BuildSitemapOptions,
 ): Promise<SitemapResult> {
   const maxUrls = options.maxUrls ?? SITEMAP_MAX_URLS;
+  const maxBytes = options.maxBytes ?? SITEMAP_MAX_BYTES;
   // Refused here and not only in `chunk`, so the answer does not depend on how many URLs the site
   // happens to have today: `maxUrls: 2.5` is a typo whether or not this build has enough routes to
   // reach the split, exactly as a metric refuses `maxSeries: 1.5` at declaration.
@@ -205,34 +283,62 @@ export async function buildSitemap(
     `buildSitemap({ maxUrls }) must be a positive integer, got ${String(maxUrls)}`,
     'pass a positive integer — buildSitemap(routes, { baseUrl, maxUrls: 50000 }) — or omit it and take SITEMAP_MAX_URLS, the sitemaps.org bound',
   );
-  const urls = await sitemapUrls(routes, options);
+  assert(
+    Number.isSafeInteger(maxBytes) && maxBytes > 0,
+    `buildSitemap({ maxBytes }) must be a positive integer, got ${String(maxBytes)}`,
+    'pass a positive integer — buildSitemap(routes, { baseUrl, maxBytes: 52428800 }) — or omit it and take SITEMAP_MAX_BYTES, the sitemaps.org bound',
+  );
+  const sections = options.sections ?? [];
+  if (sections.length > 0) assertSections(sections, Math.min(maxUrls, SITEMAP_MAX_URLS));
+
+  const clusters = await sitemapClusters(routes, options);
+  const urlCount = clusters.reduce((count, cluster) => count + cluster.urls.length, 0);
+  // Counted before a single file is rendered: a site this far past the index cap would otherwise
+  // build tens of thousands of documents on the way to a refusal.
+  if (Math.ceil(urlCount / maxUrls) > SITEMAP_INDEX_MAX_FILES) {
+    throw sitemapTooLarge(Math.ceil(urlCount / maxUrls), SITEMAP_INDEX_MAX_FILES);
+  }
 
-  if (urls.length <= maxUrls) {
+  if (sections.length > 0 && urlCount > 0) {
+    const files = sectionFiles(clusters, sections, maxUrls, maxBytes);
+    if (files.length > SITEMAP_INDEX_MAX_FILES) {
+      throw sitemapTooLarge(files.length, SITEMAP_INDEX_MAX_FILES);
+    }
+    const entries = files.map((file) => ({
+      loc: absoluteUrl(options.baseUrl, file.path),
+      lastmod: file.lastmod,
+    }));
     return {
-      files: [{ path: '/sitemap.xml', xml: renderUrlSet(urls), urlCount: urls.length }],
-      index: undefined,
-      urlCount: urls.length,
+      files,
+      index: { path: '/sitemap.xml', xml: indexXml(entries), urlCount: files.length },
+      urlCount,
     };
   }
 
-  const groups = chunk(urls, maxUrls);
-  if (groups.length > SITEMAP_INDEX_MAX_FILES) {
-    throw sitemapTooLarge(groups.length, SITEMAP_INDEX_MAX_FILES);
+  const pages = paginate(
+    clusters.flatMap((cluster) => cluster.urls),
+    maxUrls,
+    maxBytes,
+  );
+  const [only] = pages;
+  if (pages.length === 1 && only !== undefined) {
+    return { files: [fileOf('/sitemap.xml', only)], index: undefined, urlCount };
   }
-
-  const files: SitemapFile[] = groups.map((group, position) => ({
-    path: `${SITEMAP_PARTS_DIR}/${position + 1}.xml`,
-    xml: renderUrlSet(group),
-    urlCount: group.length,
+  if (pages.length > SITEMAP_INDEX_MAX_FILES) {
+    throw sitemapTooLarge(pages.length, SITEMAP_INDEX_MAX_FILES);
+  }
+  const files = pages.map((page, position) =>
+    fileOf(`${SITEMAP_PARTS_DIR}/${String(position + 1)}.xml`, page),
+  );
+  // An unsectioned split says of every part what the caller said of the build (`options.lastmod`):
+  // the bytes it always wrote. Sections are where a part's own newest date is read.
+  const entries = files.map((file) => ({
+    loc: absoluteUrl(options.baseUrl, file.path),
+    lastmod: options.lastmod,
   }));
-
   return {
     files,
-    index: {
-      path: '/sitemap.xml',
-      xml: renderIndex(files, options),
-      urlCount: files.length,
-    },
-    urlCount: urls.length,
+    index: { path: '/sitemap.xml', xml: indexXml(entries), urlCount: files.length },
+    urlCount,
   };
 }