@se-studio/site-check 1.0.1

package/CHANGELOG.md

@@ -0,0 +1,14 @@
+# Changelog
+
+## 1.0.1
+
+### Patch Changes
+
+- Bulk version bump: patch for all packages
+
+## 1.0.0
+
+- Initial release.
+- Validate sitemap.xml, sitemap-unindexed.xml, and llms.txt.
+- Download all markdown files preserving site structure.
+- Options: `-o`/`--out`, `-H`/`--header`, `--vercel-bypass [secret]` for Vercel Deployment Protection.

package/README.md

@@ -0,0 +1,76 @@
+# @se-studio/site-check
+
+Validate SE marketing sites (sitemap.xml, sitemap-unindexed.xml, llms.txt) and download all markdown files into a local directory, preserving URL path structure.
+
+## Usage
+
+```bash
+npx @se-studio/site-check <baseUrl> [-o dir] [-H "Name: value"] [--vercel-bypass [secret]] [--check-against <devUrl>]
+```
+
+### Examples
+
+```bash
+# Local site
+npx @se-studio/site-check http://localhost:3015
+
+# Custom output directory
+npx @se-studio/site-check http://localhost:3015 -o ./out
+
+# Compare production vs development: use production sitemap, check pages exist on dev
+npx @se-studio/site-check https://example.com --check-against http://localhost:3010
+
+# Vercel Deployment Protection: use env secret
+VERCEL_AUTOMATION_BYPASS_SECRET=your-secret npx @se-studio/site-check https://preview.vercel.app --vercel-bypass
+
+# Vercel bypass with explicit secret
+npx @se-studio/site-check https://preview.vercel.app --vercel-bypass your-secret
+
+# Custom headers
+npx @se-studio/site-check https://example.com -H "Authorization: Bearer token" -H "X-Custom: value"
+```
+
+## What it does
+
+1. **Validates** the site by fetching:
+   - `sitemap.xml` (required) — must return 200 and contain `<loc>` entries; if it is a sitemap index, child sitemaps are fetched and must return valid urlset content
+   - `sitemap-unindexed.xml` (optional) — if 404, a warning is printed and the run continues
+   - `llms.txt` (required) — must return 200
+
+2. **Collects markdown URLs** from **sitemap.xml** and **sitemap-unindexed.xml** (Option B): when either URL returns a **sitemap index** (e.g. `<sitemapindex>` with child `<sitemap><loc>...</loc></sitemap>`), the tool follows those links and collects page URLs from each child sitemap’s urlset (one level only). When the response is a normal urlset, every `<loc>` is treated as a page URL. Each page URL is then converted to a `.md` URL (path + `.md`). This ensures all indexed and unindexed pages are checked.
+
+3. **Checks for unexpected rewrites**: For each page URL from the sitemaps, the tool sends a HEAD request and reads the `x-nextjs-rewritten-path` response header. If the header is present and the rewritten path is different from the requested path (after normalisation), the run **fails** (exit 1). This catches cases where the sitemap lists one URL (e.g. `/learning-hub/blog/`) but the app rewrites to another (e.g. `/articles/blog`). Rewrites that only add or remove a locale prefix are allowed when `SITE_CHECK_LOCALES` is set (see [Options](#options)).
+
+4. **Checks and downloads**: For each markdown URL, the tool fetches it. If any return non-2xx, the run **fails** (exit 1) and reports which URLs are missing or errored. Otherwise it saves each response to the output directory, preserving path structure (e.g. `blog/foo.md` → `./markdown-export/blog/foo.md`, `es-US/about.md` → `./markdown-export/es-US/about.md`).
+
+### Compare mode (`--check-against`)
+
+When `--check-against <devUrl>` is set, the tool runs in **compare mode**:
+
+- **Production** (the `<baseUrl>`) is the source of the URL list: sitemap.xml and sitemap-unindexed.xml are fetched from production. Validation (and llms.txt) applies to production.
+- **Development** (`<devUrl>`) is the site checked: each production sitemap page URL is mapped to the same path on the development origin. The tool then checks that each of those development URLs returns 2xx (HEAD) and runs the rewrite check against the development site. Markdown collection and download are **not** performed in compare mode.
+
+Use this to ensure a development or staging build has the same pages as production (e.g. before release). Page title comparison is not yet implemented; a future option may add it.
+
+## Options
+
+| Option | Description |
+|--------|-------------|
+| `baseUrl` | Base URL of the site (required). In compare mode this is the **production** URL (sitemap source). Trailing slash is stripped. |
+| `-o`, `--out <dir>` | Output directory for markdown files (default: `./markdown-export`). Ignored in compare mode. |
+| `-H`, `--header "Name: value"` | Add a request header (repeatable). Applied to both production and development requests in compare mode. |
+| `--check-against <devUrl>` | Compare mode: use production sitemap(s) but check that each page exists on development (same path on `<devUrl>`). No markdown download. |
+| `--vercel-bypass [secret]` | Set `x-vercel-protection-bypass` for [Vercel Deployment Protection](https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/protection-bypass-automation). If `secret` is omitted, uses `VERCEL_AUTOMATION_BYPASS_SECRET` from the environment. |
+
+Headers from `-H` override the Vercel bypass header if the same name is used.
+
+### Environment
+
+| Variable | Description |
+|----------|-------------|
+| `SITE_CHECK_LOCALES` | Comma-separated list of locale path segments (e.g. `en,en-gb,de`). When set, a rewrite is considered acceptable if the only difference between the requested path and the rewritten path is a leading locale segment. If unset, any rewrite to a different path fails. |
+
+## Exit codes
+
+- `0` — Validation passed, no unexpected rewrites, every sitemap page returned 200 for its `.md` URL, and files were saved (or in compare mode: all production sitemap pages exist on development and no unexpected rewrites).
+- `1` — Usage error (missing baseUrl), validation failed (sitemap.xml or llms.txt), one or more sitemap pages rewrite to a different path, one or more markdown URLs returned non-2xx (pages missing or broken), or in compare mode one or more development URLs did not return 2xx.

package/dist/check-pages-exist.d.ts

@@ -0,0 +1,19 @@
+/**
+ * Check that each page URL returns 2xx (e.g. HEAD). Used in compare mode to verify
+ * development has the same pages as production sitemap.
+ */
+interface PageExistError {
+    url: string;
+    status: number;
+    message: string;
+}
+interface CheckPagesExistResult {
+    ok: boolean;
+    errors: PageExistError[];
+}
+/**
+ * For each page URL, send HEAD request. Require 2xx. Return errors for non-2xx.
+ */
+export declare function checkPagesExist(pageUrls: string[], fetchInit: RequestInit): Promise<CheckPagesExistResult>;
+export {};
+//# sourceMappingURL=check-pages-exist.d.ts.map

package/dist/check-pages-exist.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"check-pages-exist.d.ts","sourceRoot":"","sources":["../src/check-pages-exist.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,UAAU,cAAc;IACtB,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,UAAU,qBAAqB;IAC7B,EAAE,EAAE,OAAO,CAAC;IACZ,MAAM,EAAE,cAAc,EAAE,CAAC;CAC1B;AAED;;GAEG;AACH,wBAAsB,eAAe,CACnC,QAAQ,EAAE,MAAM,EAAE,EAClB,SAAS,EAAE,WAAW,GACrB,OAAO,CAAC,qBAAqB,CAAC,CAiBhC"}

package/dist/check-pages-exist.js

@@ -0,0 +1,25 @@
+/**
+ * Check that each page URL returns 2xx (e.g. HEAD). Used in compare mode to verify
+ * development has the same pages as production sitemap.
+ */
+/**
+ * For each page URL, send HEAD request. Require 2xx. Return errors for non-2xx.
+ */
+export async function checkPagesExist(pageUrls, fetchInit) {
+    const errors = [];
+    for (const url of pageUrls) {
+        const res = await fetch(url, { ...fetchInit, method: 'HEAD' });
+        if (res.ok)
+            continue;
+        errors.push({
+            url,
+            status: res.status,
+            message: `HTTP ${res.status}`,
+        });
+    }
+    return {
+        ok: errors.length === 0,
+        errors,
+    };
+}
+//# sourceMappingURL=check-pages-exist.js.map

package/dist/check-pages-exist.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"check-pages-exist.js","sourceRoot":"","sources":["../src/check-pages-exist.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAaH;;GAEG;AACH,MAAM,CAAC,KAAK,UAAU,eAAe,CACnC,QAAkB,EAClB,SAAsB;IAEtB,MAAM,MAAM,GAAqB,EAAE,CAAC;IAEpC,KAAK,MAAM,GAAG,IAAI,QAAQ,EAAE,CAAC;QAC3B,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,GAAG,EAAE,EAAE,GAAG,SAAS,EAAE,MAAM,EAAE,MAAM,EAAE,CAAC,CAAC;QAC/D,IAAI,GAAG,CAAC,EAAE;YAAE,SAAS;QACrB,MAAM,CAAC,IAAI,CAAC;YACV,GAAG;YACH,MAAM,EAAE,GAAG,CAAC,MAAM;YAClB,OAAO,EAAE,QAAQ,GAAG,CAAC,MAAM,EAAE;SAC9B,CAAC,CAAC;IACL,CAAC;IAED,OAAO;QACL,EAAE,EAAE,MAAM,CAAC,MAAM,KAAK,CAAC;QACvB,MAAM;KACP,CAAC;AACJ,CAAC"}

package/dist/check-rewrites.d.ts

@@ -0,0 +1,21 @@
+/**
+ * Verify that sitemap page URLs do not rewrite to different paths (except localisation).
+ * Fetches each page URL and checks the x-nextjs-rewritten-path response header.
+ */
+interface RewriteCheckError {
+    url: string;
+    requestedPath: string;
+    rewrittenPath: string;
+}
+interface RewriteCheckResult {
+    ok: boolean;
+    errors: RewriteCheckError[];
+}
+/**
+ * For each page URL, fetch with HEAD and check x-nextjs-rewritten-path.
+ * If the header is present and the rewritten path is not equivalent to the requested path
+ * (after normalisation and optional locale stripping), record an error.
+ */
+export declare function checkRewrites(pageUrls: string[], fetchInit: RequestInit): Promise<RewriteCheckResult>;
+export {};
+//# sourceMappingURL=check-rewrites.d.ts.map

package/dist/check-rewrites.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"check-rewrites.d.ts","sourceRoot":"","sources":["../src/check-rewrites.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAuCH,UAAU,iBAAiB;IACzB,GAAG,EAAE,MAAM,CAAC;IACZ,aAAa,EAAE,MAAM,CAAC;IACtB,aAAa,EAAE,MAAM,CAAC;CACvB;AAED,UAAU,kBAAkB;IAC1B,EAAE,EAAE,OAAO,CAAC;IACZ,MAAM,EAAE,iBAAiB,EAAE,CAAC;CAC7B;AAED;;;;GAIG;AACH,wBAAsB,aAAa,CACjC,QAAQ,EAAE,MAAM,EAAE,EAClB,SAAS,EAAE,WAAW,GACrB,OAAO,CAAC,kBAAkB,CAAC,CA0B7B"}

package/dist/check-rewrites.js

@@ -0,0 +1,69 @@
+/**
+ * Verify that sitemap page URLs do not rewrite to different paths (except localisation).
+ * Fetches each page URL and checks the x-nextjs-rewritten-path response header.
+ */
+const REWRITTEN_PATH_HEADER = 'x-nextjs-rewritten-path';
+const LOCALES_ENV = 'SITE_CHECK_LOCALES';
+function getLocalePrefixes() {
+    const raw = process.env[LOCALES_ENV];
+    if (!raw || typeof raw !== 'string')
+        return [];
+    return raw
+        .split(',')
+        .map((s) => s.trim().toLowerCase())
+        .filter(Boolean);
+}
+/**
+ * Normalise path for comparison: strip trailing slash, lowercase.
+ * If localePrefixes are given, strip a leading segment that matches a locale.
+ */
+function normalisePath(pathname, localePrefixes) {
+    let path = pathname.replace(/\/$/, '') || '/';
+    if (path !== '/' && localePrefixes.length > 0) {
+        const segments = path.split('/').filter(Boolean);
+        const first = segments[0];
+        if (first && localePrefixes.includes(first.toLowerCase())) {
+            segments.shift();
+            path = segments.length > 0 ? `/${segments.join('/')}` : '/';
+        }
+    }
+    return path;
+}
+function pathnameFromUrl(url) {
+    try {
+        return new URL(url).pathname;
+    }
+    catch {
+        return '/';
+    }
+}
+/**
+ * For each page URL, fetch with HEAD and check x-nextjs-rewritten-path.
+ * If the header is present and the rewritten path is not equivalent to the requested path
+ * (after normalisation and optional locale stripping), record an error.
+ */
+export async function checkRewrites(pageUrls, fetchInit) {
+    const localePrefixes = getLocalePrefixes();
+    const errors = [];
+    for (const url of pageUrls) {
+        const res = await fetch(url, { ...fetchInit, method: 'HEAD' });
+        const rewritten = res.headers.get(REWRITTEN_PATH_HEADER);
+        if (!rewritten)
+            continue;
+        const requestedPath = pathnameFromUrl(url);
+        const requestedNorm = normalisePath(requestedPath, localePrefixes);
+        const rewrittenNorm = normalisePath(rewritten, localePrefixes);
+        if (requestedNorm !== rewrittenNorm) {
+            errors.push({
+                url,
+                requestedPath: requestedPath || '/',
+                rewrittenPath: rewritten,
+            });
+        }
+    }
+    return {
+        ok: errors.length === 0,
+        errors,
+    };
+}
+//# sourceMappingURL=check-rewrites.js.map

package/dist/check-rewrites.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"check-rewrites.js","sourceRoot":"","sources":["../src/check-rewrites.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,MAAM,qBAAqB,GAAG,yBAAyB,CAAC;AACxD,MAAM,WAAW,GAAG,oBAAoB,CAAC;AAEzC,SAAS,iBAAiB;IACxB,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC,WAAW,CAAC,CAAC;IACrC,IAAI,CAAC,GAAG,IAAI,OAAO,GAAG,KAAK,QAAQ;QAAE,OAAO,EAAE,CAAC;IAC/C,OAAO,GAAG;SACP,KAAK,CAAC,GAAG,CAAC;SACV,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC;SAClC,MAAM,CAAC,OAAO,CAAC,CAAC;AACrB,CAAC;AAED;;;GAGG;AACH,SAAS,aAAa,CAAC,QAAgB,EAAE,cAAwB;IAC/D,IAAI,IAAI,GAAG,QAAQ,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,IAAI,GAAG,CAAC;IAC9C,IAAI,IAAI,KAAK,GAAG,IAAI,cAAc,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC9C,MAAM,QAAQ,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;QACjD,MAAM,KAAK,GAAG,QAAQ,CAAC,CAAC,CAAC,CAAC;QAC1B,IAAI,KAAK,IAAI,cAAc,CAAC,QAAQ,CAAC,KAAK,CAAC,WAAW,EAAE,CAAC,EAAE,CAAC;YAC1D,QAAQ,CAAC,KAAK,EAAE,CAAC;YACjB,IAAI,GAAG,QAAQ,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC;QAC9D,CAAC;IACH,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC;AAED,SAAS,eAAe,CAAC,GAAW;IAClC,IAAI,CAAC;QACH,OAAO,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC;IAC/B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,GAAG,CAAC;IACb,CAAC;AACH,CAAC;AAaD;;;;GAIG;AACH,MAAM,CAAC,KAAK,UAAU,aAAa,CACjC,QAAkB,EAClB,SAAsB;IAEtB,MAAM,cAAc,GAAG,iBAAiB,EAAE,CAAC;IAC3C,MAAM,MAAM,GAAwB,EAAE,CAAC;IAEvC,KAAK,MAAM,GAAG,IAAI,QAAQ,EAAE,CAAC;QAC3B,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,GAAG,EAAE,EAAE,GAAG,SAAS,EAAE,MAAM,EAAE,MAAM,EAAE,CAAC,CAAC;QAC/D,MAAM,SAAS,GAAG,GAAG,CAAC,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,CAAC;QACzD,IAAI,CAAC,SAAS;YAAE,SAAS;QAEzB,MAAM,aAAa,GAAG,eAAe,CAAC,GAAG,CAAC,CAAC;QAC3C,MAAM,aAAa,GAAG,aAAa,CAAC,aAAa,EAAE,cAAc,CAAC,CAAC;QACnE,MAAM,aAAa,GAAG,aAAa,CAAC,SAAS,EAAE,cAAc,CAAC,CAAC;QAE/D,IAAI,aAAa,KAAK,aAAa,EAAE,CAAC;YACpC,MAAM,CAAC,IAAI,CAAC;gBACV,GAAG;gBACH,aAAa,EAAE,aAAa,IAAI,GAAG;gBACnC,aAAa,EAAE,SAAS;aACzB,CAAC,CAAC;QACL,CAAC;IACH,CAAC;IAED,OAAO;QACL,EAAE,EAAE,MAAM,CAAC,MAAM,KAAK,CAAC;QACvB,MAAM;KACP,CAAC;AACJ,CAAC"}

package/dist/cli.d.ts

@@ -0,0 +1,4 @@
+#!/usr/bin/env node
+/** biome-ignore-all lint/suspicious/noConsole: CLI output is intentional */
+export {};
+//# sourceMappingURL=cli.d.ts.map

package/dist/cli.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAEA,4EAA4E"}

package/dist/cli.js

@@ -0,0 +1,185 @@
+#!/usr/bin/env node
+/** biome-ignore-all lint/suspicious/noConsole: CLI output is intentional */
+import { checkPagesExist } from './check-pages-exist.js';
+import { checkRewrites } from './check-rewrites.js';
+import { collectMarkdownUrlsFromSitemaps, mapProdPageUrlsToDev, pageUrlsFromSitemaps, } from './collect-markdown-urls.js';
+import { downloadMarkdownFiles } from './download.js';
+import { buildFetchOptions } from './fetch-options.js';
+import { validate, validationFailed } from './validate.js';
+const DEFAULT_OUT = './markdown-export';
+function isValidAbsoluteUrl(url) {
+    try {
+        const u = new URL(url);
+        return u.protocol === 'http:' || u.protocol === 'https:';
+    }
+    catch {
+        return false;
+    }
+}
+function parseArgs(argv) {
+    const headerArgs = [];
+    let outDir = DEFAULT_OUT;
+    let vercelBypassRequested = false;
+    let vercelBypassSecret = null;
+    let baseUrl = null;
+    let checkAgainstUrl = null;
+    for (let i = 0; i < argv.length; i++) {
+        const arg = argv[i];
+        if (arg === '-o' || arg === '--out') {
+            const next = argv[i + 1];
+            if (next && !next.startsWith('-')) {
+                outDir = next;
+                i++;
+            }
+        }
+        else if (arg === '-H' || arg === '--header') {
+            const next = argv[i + 1];
+            if (next) {
+                headerArgs.push(next);
+                i++;
+            }
+        }
+        else if (arg === '--check-against') {
+            const next = argv[i + 1];
+            if (next && (next.startsWith('http://') || next.startsWith('https://'))) {
+                checkAgainstUrl = next.replace(/\/$/, '');
+                i++;
+            }
+        }
+        else if (arg === '--vercel-bypass') {
+            vercelBypassRequested = true;
+            const next = argv[i + 1];
+            if (next && !next.startsWith('-')) {
+                vercelBypassSecret = next;
+                i++;
+            }
+        }
+        else if (arg.startsWith('http://') || arg.startsWith('https://')) {
+            baseUrl = arg.replace(/\/$/, '');
+        }
+    }
+    return {
+        baseUrl,
+        checkAgainstUrl,
+        outDir,
+        headerArgs,
+        vercelBypassRequested,
+        vercelBypassSecret,
+    };
+}
+async function main() {
+    const args = parseArgs(process.argv.slice(2));
+    if (!args.baseUrl) {
+        console.error('Usage: site-check <baseUrl> [-o dir] [-H "Name: value"] [--vercel-bypass [secret]] [--check-against <devUrl>]');
+        console.error('Example: site-check http://localhost:3015 -o ./out');
+        console.error('Example (compare prod vs dev): site-check https://example.com --check-against http://localhost:3010');
+        return 1;
+    }
+    const fetchInit = buildFetchOptions({
+        headerArgs: args.headerArgs,
+        vercelBypassRequested: args.vercelBypassRequested,
+        vercelBypassSecret: args.vercelBypassSecret,
+    });
+    const isCompareMode = args.checkAgainstUrl !== null;
+    const prodUrl = args.baseUrl;
+    if (isCompareMode) {
+        console.log(`Comparing: production ${prodUrl} vs development ${args.checkAgainstUrl}`);
+    }
+    console.log('Validating…');
+    const validation = await validate(prodUrl, fetchInit);
+    const sitemapOk = validation.sitemapXml.ok ? 'OK' : `FAIL (${validation.sitemapXml.status})`;
+    console.log(`  sitemap.xml … ${sitemapOk}`);
+    if (validation.sitemapUnindexedXml.warned) {
+        console.log('  sitemap-unindexed.xml … not found (optional, continuing)');
+    }
+    else {
+        const unindexedOk = validation.sitemapUnindexedXml.ok
+            ? 'OK'
+            : `FAIL (${validation.sitemapUnindexedXml.status})`;
+        console.log(`  sitemap-unindexed.xml … ${unindexedOk}`);
+    }
+    const llmsOk = validation.llmsTxt.ok ? 'OK' : `FAIL (${validation.llmsTxt.status})`;
+    console.log(`  llms.txt … ${llmsOk}`);
+    if (validationFailed(validation)) {
+        console.error('Validation failed. Fix sitemap.xml and/or llms.txt and try again.');
+        return 1;
+    }
+    if (isCompareMode) {
+        const devBaseUrl = args.checkAgainstUrl;
+        const prodPageUrls = pageUrlsFromSitemaps(validation);
+        const devPageUrls = mapProdPageUrlsToDev(prodPageUrls, devBaseUrl);
+        const indexNote = validation.sitemapWasIndex ? ' (including from sitemap index)' : '';
+        console.log(`Using sitemap from production: ${prodPageUrls.length} page(s) → checking ${devPageUrls.length} on development${indexNote}`);
+        if (devPageUrls.length === 0) {
+            console.log('No page URLs to check on development.');
+            return 0;
+        }
+        const validDevUrls = devPageUrls.filter(isValidAbsoluteUrl);
+        if (validDevUrls.length > 0) {
+            console.log(`Checking ${validDevUrls.length} page(s) exist on development…`);
+            const existResult = await checkPagesExist(validDevUrls, fetchInit);
+            if (!existResult.ok) {
+                console.error(`\n${existResult.errors.length} page(s) did not return 2xx on development:`);
+                for (const e of existResult.errors) {
+                    console.error(`  ${e.url} … ${e.message}`);
+                }
+                return 1;
+            }
+            console.log(`Checking ${validDevUrls.length} page(s) for unexpected rewrites…`);
+            const rewriteResult = await checkRewrites(validDevUrls, fetchInit);
+            if (!rewriteResult.ok) {
+                console.error(`\n${rewriteResult.errors.length} page(s) rewrite to a different path (x-nextjs-rewritten-path):`);
+                for (const e of rewriteResult.errors) {
+                    console.error(`  ${e.url}`);
+                    console.error(`    requested: ${e.requestedPath} → rewritten: ${e.rewrittenPath}`);
+                }
+                console.error('\nEnsure sitemap URLs match the app routes, or use path rewrites only for localisation.');
+                return 1;
+            }
+        }
+        console.log('All production sitemap pages exist on development.');
+        return 0;
+    }
+    console.log('Collecting markdown URLs from sitemaps…');
+    const urls = collectMarkdownUrlsFromSitemaps(args.baseUrl, validation);
+    const indexNote = validation.sitemapWasIndex ? ' (including from sitemap index)' : '';
+    console.log(`  Found ${urls.length} URLs (sitemap.xml + sitemap-unindexed.xml)${indexNote}`);
+    if (urls.length === 0) {
+        console.log('No markdown URLs to check or download.');
+        return 0;
+    }
+    const pageUrls = pageUrlsFromSitemaps(validation);
+    const validPageUrls = pageUrls.filter(isValidAbsoluteUrl);
+    if (validPageUrls.length > 0) {
+        console.log(`Checking ${validPageUrls.length} sitemap page(s) for unexpected rewrites…`);
+        const rewriteResult = await checkRewrites(validPageUrls, fetchInit);
+        if (!rewriteResult.ok) {
+            console.error(`\n${rewriteResult.errors.length} page(s) rewrite to a different path (x-nextjs-rewritten-path):`);
+            for (const e of rewriteResult.errors) {
+                console.error(`  ${e.url}`);
+                console.error(`    requested: ${e.requestedPath} → rewritten: ${e.rewrittenPath}`);
+            }
+            console.error('\nEnsure sitemap URLs match the app routes, or use path rewrites only for localisation.');
+            return 1;
+        }
+    }
+    console.log(`Checking and downloading ${urls.length} markdown files…`);
+    const downloadResult = await downloadMarkdownFiles(urls, args.outDir, fetchInit);
+    if (downloadResult.errors.length > 0) {
+        console.error(`\n${downloadResult.errors.length} page(s) did not return 200 (markdown missing or error):`);
+        for (const e of downloadResult.errors) {
+            console.error(`  ${e.url} … ${e.message}`);
+        }
+        console.error('\nFix the markdown routes or sitemap so every sitemap URL has a working .md version.');
+        return 1;
+    }
+    console.log(`Saved ${downloadResult.saved} files to ${args.outDir}`);
+    return 0;
+}
+main()
+    .then((code) => process.exit(code))
+    .catch((err) => {
+    console.error(err);
+    process.exit(1);
+});
+//# sourceMappingURL=cli.js.map

package/dist/cli.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAEA,4EAA4E;AAE5E,OAAO,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAC;AACzD,OAAO,EAAE,aAAa,EAAE,MAAM,qBAAqB,CAAC;AACpD,OAAO,EACL,+BAA+B,EAC/B,oBAAoB,EACpB,oBAAoB,GACrB,MAAM,4BAA4B,CAAC;AACpC,OAAO,EAAE,qBAAqB,EAAE,MAAM,eAAe,CAAC;AACtD,OAAO,EAAE,iBAAiB,EAAE,MAAM,oBAAoB,CAAC;AACvD,OAAO,EAAE,QAAQ,EAAE,gBAAgB,EAAE,MAAM,eAAe,CAAC;AAE3D,MAAM,WAAW,GAAG,mBAAmB,CAAC;AAExC,SAAS,kBAAkB,CAAC,GAAW;IACrC,IAAI,CAAC;QACH,MAAM,CAAC,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC;QACvB,OAAO,CAAC,CAAC,QAAQ,KAAK,OAAO,IAAI,CAAC,CAAC,QAAQ,KAAK,QAAQ,CAAC;IAC3D,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,KAAK,CAAC;IACf,CAAC;AACH,CAAC;AAWD,SAAS,SAAS,CAAC,IAAc;IAC/B,MAAM,UAAU,GAAa,EAAE,CAAC;IAChC,IAAI,MAAM,GAAG,WAAW,CAAC;IACzB,IAAI,qBAAqB,GAAG,KAAK,CAAC;IAClC,IAAI,kBAAkB,GAAkB,IAAI,CAAC;IAC7C,IAAI,OAAO,GAAkB,IAAI,CAAC;IAClC,IAAI,eAAe,GAAkB,IAAI,CAAC;IAE1C,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,MAAM,GAAG,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;QACpB,IAAI,GAAG,KAAK,IAAI,IAAI,GAAG,KAAK,OAAO,EAAE,CAAC;YACpC,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;YACzB,IAAI,IAAI,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;gBAClC,MAAM,GAAG,IAAI,CAAC;gBACd,CAAC,EAAE,CAAC;YACN,CAAC;QACH,CAAC;aAAM,IAAI,GAAG,KAAK,IAAI,IAAI,GAAG,KAAK,UAAU,EAAE,CAAC;YAC9C,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;YACzB,IAAI,IAAI,EAAE,CAAC;gBACT,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;gBACtB,CAAC,EAAE,CAAC;YACN,CAAC;QACH,CAAC;aAAM,IAAI,GAAG,KAAK,iBAAiB,EAAE,CAAC;YACrC,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;YACzB,IAAI,IAAI,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,IAAI,IAAI,CAAC,UAAU,CAAC,UAAU,CAAC,CAAC,EAAE,CAAC;gBACxE,eAAe,GAAG,IAAI,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;gBAC1C,CAAC,EAAE,CAAC;YACN,CAAC;QACH,CAAC;aAAM,IAAI,GAAG,KAAK,iBAAiB,EAAE,CAAC;YACrC,qBAAqB,GAAG,IAAI,CAAC;YAC7B,MAAM,IAAI,GAAG,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;YACzB,IAAI,IAAI,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;gBAClC,kBAAkB,GAAG,IAAI,CAAC;gBAC1B,CAAC,EAAE,CAAC;YACN,CAAC;QACH,CAAC;aAAM,IAAI,GAAG,CAAC,UAAU,CAAC,SAAS,CAAC,IAAI,GAAG,CAAC,UAAU,CAAC,UAAU,CAAC,EAAE,CAAC;YACnE,OAAO,GAAG,GAAG,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;QACnC,CAAC;IACH,CAAC;IAED,OAAO;QACL,OAAO;QACP,eAAe;QACf,MAAM;QACN,UAAU;QACV,qBAAqB;QACrB,kBAAkB;KACnB,CAAC;AACJ,CAAC;AAED,KAAK,UAAU,IAAI;IACjB,MAAM,IAAI,GAAG,SAAS,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;IAE9C,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;QAClB,OAAO,CAAC,KAAK,CACX,+GAA+G,CAChH,CAAC;QACF,OAAO,CAAC,KAAK,CAAC,oDAAoD,CAAC,CAAC;QACpE,OAAO,CAAC,KAAK,CACX,qGAAqG,CACtG,CAAC;QACF,OAAO,CAAC,CAAC;IACX,CAAC;IAED,MAAM,SAAS,GAAG,iBAAiB,CAAC;QAClC,UAAU,EAAE,IAAI,CAAC,UAAU;QAC3B,qBAAqB,EAAE,IAAI,CAAC,qBAAqB;QACjD,kBAAkB,EAAE,IAAI,CAAC,kBAAkB;KAC5C,CAAC,CAAC;IAEH,MAAM,aAAa,GAAG,IAAI,CAAC,eAAe,KAAK,IAAI,CAAC;IACpD,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC;IAE7B,IAAI,aAAa,EAAE,CAAC;QAClB,OAAO,CAAC,GAAG,CAAC,yBAAyB,OAAO,mBAAmB,IAAI,CAAC,eAAe,EAAE,CAAC,CAAC;IACzF,CAAC;IAED,OAAO,CAAC,GAAG,CAAC,aAAa,CAAC,CAAC;IAC3B,MAAM,UAAU,GAAG,MAAM,QAAQ,CAAC,OAAO,EAAE,SAAS,CAAC,CAAC;IAEtD,MAAM,SAAS,GAAG,UAAU,CAAC,UAAU,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,UAAU,CAAC,UAAU,CAAC,MAAM,GAAG,CAAC;IAC7F,OAAO,CAAC,GAAG,CAAC,mBAAmB,SAAS,EAAE,CAAC,CAAC;IAE5C,IAAI,UAAU,CAAC,mBAAmB,CAAC,MAAM,EAAE,CAAC;QAC1C,OAAO,CAAC,GAAG,CAAC,4DAA4D,CAAC,CAAC;IAC5E,CAAC;SAAM,CAAC;QACN,MAAM,WAAW,GAAG,UAAU,CAAC,mBAAmB,CAAC,EAAE;YACnD,CAAC,CAAC,IAAI;YACN,CAAC,CAAC,SAAS,UAAU,CAAC,mBAAmB,CAAC,MAAM,GAAG,CAAC;QACtD,OAAO,CAAC,GAAG,CAAC,6BAA6B,WAAW,EAAE,CAAC,CAAC;IAC1D,CAAC;IAED,MAAM,MAAM,GAAG,UAAU,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,UAAU,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC;IACpF,OAAO,CAAC,GAAG,CAAC,gBAAgB,MAAM,EAAE,CAAC,CAAC;IAEtC,IAAI,gBAAgB,CAAC,UAAU,CAAC,EAAE,CAAC;QACjC,OAAO,CAAC,KAAK,CAAC,mEAAmE,CAAC,CAAC;QACnF,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,aAAa,EAAE,CAAC;QAClB,MAAM,UAAU,GAAG,IAAI,CAAC,eAAyB,CAAC;QAClD,MAAM,YAAY,GAAG,oBAAoB,CAAC,UAAU,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,oBAAoB,CAAC,YAAY,EAAE,UAAU,CAAC,CAAC;QACnE,MAAM,SAAS,GAAG,UAAU,CAAC,eAAe,CAAC,CAAC,CAAC,iCAAiC,CAAC,CAAC,CAAC,EAAE,CAAC;QACtF,OAAO,CAAC,GAAG,CACT,kCAAkC,YAAY,CAAC,MAAM,uBAAuB,WAAW,CAAC,MAAM,kBAAkB,SAAS,EAAE,CAC5H,CAAC;QACF,IAAI,WAAW,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YAC7B,OAAO,CAAC,GAAG,CAAC,uCAAuC,CAAC,CAAC;YACrD,OAAO,CAAC,CAAC;QACX,CAAC;QACD,MAAM,YAAY,GAAG,WAAW,CAAC,MAAM,CAAC,kBAAkB,CAAC,CAAC;QAC5D,IAAI,YAAY,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YAC5B,OAAO,CAAC,GAAG,CAAC,YAAY,YAAY,CAAC,MAAM,gCAAgC,CAAC,CAAC;YAC7E,MAAM,WAAW,GAAG,MAAM,eAAe,CAAC,YAAY,EAAE,SAAS,CAAC,CAAC;YACnE,IAAI,CAAC,WAAW,CAAC,EAAE,EAAE,CAAC;gBACpB,OAAO,CAAC,KAAK,CAAC,KAAK,WAAW,CAAC,MAAM,CAAC,MAAM,6CAA6C,CAAC,CAAC;gBAC3F,KAAK,MAAM,CAAC,IAAI,WAAW,CAAC,MAAM,EAAE,CAAC;oBACnC,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC,OAAO,EAAE,CAAC,CAAC;gBAC7C,CAAC;gBACD,OAAO,CAAC,CAAC;YACX,CAAC;YACD,OAAO,CAAC,GAAG,CAAC,YAAY,YAAY,CAAC,MAAM,mCAAmC,CAAC,CAAC;YAChF,MAAM,aAAa,GAAG,MAAM,aAAa,CAAC,YAAY,EAAE,SAAS,CAAC,CAAC;YACnE,IAAI,CAAC,aAAa,CAAC,EAAE,EAAE,CAAC;gBACtB,OAAO,CAAC,KAAK,CACX,KAAK,aAAa,CAAC,MAAM,CAAC,MAAM,iEAAiE,CAClG,CAAC;gBACF,KAAK,MAAM,CAAC,IAAI,aAAa,CAAC,MAAM,EAAE,CAAC;oBACrC,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC;oBAC5B,OAAO,CAAC,KAAK,CAAC,kBAAkB,CAAC,CAAC,aAAa,iBAAiB,CAAC,CAAC,aAAa,EAAE,CAAC,CAAC;gBACrF,CAAC;gBACD,OAAO,CAAC,KAAK,CACX,yFAAyF,CAC1F,CAAC;gBACF,OAAO,CAAC,CAAC;YACX,CAAC;QACH,CAAC;QACD,OAAO,CAAC,GAAG,CAAC,oDAAoD,CAAC,CAAC;QAClE,OAAO,CAAC,CAAC;IACX,CAAC;IAED,OAAO,CAAC,GAAG,CAAC,yCAAyC,CAAC,CAAC;IACvD,MAAM,IAAI,GAAG,+BAA+B,CAAC,IAAI,CAAC,OAAO,EAAE,UAAU,CAAC,CAAC;IACvE,MAAM,SAAS,GAAG,UAAU,CAAC,eAAe,CAAC,CAAC,CAAC,iCAAiC,CAAC,CAAC,CAAC,EAAE,CAAC;IACtF,OAAO,CAAC,GAAG,CAAC,WAAW,IAAI,CAAC,MAAM,8CAA8C,SAAS,EAAE,CAAC,CAAC;IAE7F,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACtB,OAAO,CAAC,GAAG,CAAC,wCAAwC,CAAC,CAAC;QACtD,OAAO,CAAC,CAAC;IACX,CAAC;IAED,MAAM,QAAQ,GAAG,oBAAoB,CAAC,UAAU,CAAC,CAAC;IAClD,MAAM,aAAa,GAAG,QAAQ,CAAC,MAAM,CAAC,kBAAkB,CAAC,CAAC;IAC1D,IAAI,aAAa,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC7B,OAAO,CAAC,GAAG,CAAC,YAAY,aAAa,CAAC,MAAM,2CAA2C,CAAC,CAAC;QACzF,MAAM,aAAa,GAAG,MAAM,aAAa,CAAC,aAAa,EAAE,SAAS,CAAC,CAAC;QACpE,IAAI,CAAC,aAAa,CAAC,EAAE,EAAE,CAAC;YACtB,OAAO,CAAC,KAAK,CACX,KAAK,aAAa,CAAC,MAAM,CAAC,MAAM,iEAAiE,CAClG,CAAC;YACF,KAAK,MAAM,CAAC,IAAI,aAAa,CAAC,MAAM,EAAE,CAAC;gBACrC,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC;gBAC5B,OAAO,CAAC,KAAK,CAAC,kBAAkB,CAAC,CAAC,aAAa,iBAAiB,CAAC,CAAC,aAAa,EAAE,CAAC,CAAC;YACrF,CAAC;YACD,OAAO,CAAC,KAAK,CACX,yFAAyF,CAC1F,CAAC;YACF,OAAO,CAAC,CAAC;QACX,CAAC;IACH,CAAC;IAED,OAAO,CAAC,GAAG,CAAC,4BAA4B,IAAI,CAAC,MAAM,kBAAkB,CAAC,CAAC;IACvE,MAAM,cAAc,GAAG,MAAM,qBAAqB,CAAC,IAAI,EAAE,IAAI,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;IAEjF,IAAI,cAAc,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACrC,OAAO,CAAC,KAAK,CACX,KAAK,cAAc,CAAC,MAAM,CAAC,MAAM,0DAA0D,CAC5F,CAAC;QACF,KAAK,MAAM,CAAC,IAAI,cAAc,CAAC,MAAM,EAAE,CAAC;YACtC,OAAO,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC,OAAO,EAAE,CAAC,CAAC;QAC7C,CAAC;QACD,OAAO,CAAC,KAAK,CACX,sFAAsF,CACvF,CAAC;QACF,OAAO,CAAC,CAAC;IACX,CAAC;IAED,OAAO,CAAC,GAAG,CAAC,SAAS,cAAc,CAAC,KAAK,aAAa,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC;IACrE,OAAO,CAAC,CAAC;AACX,CAAC;AAED,IAAI,EAAE;KACH,IAAI,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;KAClC,KAAK,CAAC,CAAC,GAAG,EAAE,EAAE;IACb,OAAO,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IACnB,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC,CAAC,CAAC"}

package/dist/collect-markdown-urls.d.ts

@@ -0,0 +1,25 @@
+/**
+ * Collect markdown URLs from sitemap.xml and sitemap-unindexed.xml (Option B).
+ * Every <loc> in both sitemaps is converted to a .md URL; we then check each returns 200.
+ * Download paths and static assets (e.g. PDFs) are excluded — they have no markdown version.
+ */
+import type { ValidationResult } from './validate.js';
+/**
+ * Collect all page URLs from sitemap.xml and sitemap-unindexed.xml (raw <loc> values).
+ * Uses resolved page URL lists when present (e.g. after following sitemap index links).
+ * Used for rewrite checks so every sitemap URL is verified to not rewrite to a different path.
+ */
+export declare function pageUrlsFromSitemaps(validationResult: ValidationResult): string[];
+/**
+ * Collect all markdown URLs from sitemap.xml and sitemap-unindexed.xml (Option B).
+ * Uses resolved page URL lists when present (e.g. after following sitemap index links).
+ * Ensures every sitemap page has a corresponding markdown URL to validate.
+ */
+export declare function collectMarkdownUrlsFromSitemaps(baseUrl: string, validationResult: ValidationResult): string[];
+/**
+ * Map production page URLs to development page URLs (same path, dev origin).
+ * Excludes non-page assets (download paths, static file extensions).
+ * Used in compare mode: sitemap from prod, check existence on dev.
+ */
+export declare function mapProdPageUrlsToDev(prodPageUrls: string[], devBaseUrl: string): string[];
+//# sourceMappingURL=collect-markdown-urls.d.ts.map

package/dist/collect-markdown-urls.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"collect-markdown-urls.d.ts","sourceRoot":"","sources":["../src/collect-markdown-urls.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,eAAe,CAAC;AA+CtD;;;;GAIG;AACH,wBAAgB,oBAAoB,CAAC,gBAAgB,EAAE,gBAAgB,GAAG,MAAM,EAAE,CAajF;AAiCD;;;;GAIG;AACH,wBAAgB,+BAA+B,CAC7C,OAAO,EAAE,MAAM,EACf,gBAAgB,EAAE,gBAAgB,GACjC,MAAM,EAAE,CAGV;AAiBD;;;;GAIG;AACH,wBAAgB,oBAAoB,CAAC,YAAY,EAAE,MAAM,EAAE,EAAE,UAAU,EAAE,MAAM,GAAG,MAAM,EAAE,CAQzF"}

package/dist/collect-markdown-urls.js

@@ -0,0 +1,144 @@
+/**
+ * Collect markdown URLs from sitemap.xml and sitemap-unindexed.xml (Option B).
+ * Every <loc> in both sitemaps is converted to a .md URL; we then check each returns 200.
+ * Download paths and static assets (e.g. PDFs) are excluded — they have no markdown version.
+ */
+/** Path segment extensions for static assets; these URLs are excluded from markdown checks. */
+const STATIC_EXTENSIONS = new Set([
+    '.pdf',
+    '.zip',
+    '.doc',
+    '.docx',
+    '.xls',
+    '.xlsx',
+    '.ppt',
+    '.pptx',
+    '.csv',
+    '.txt',
+    '.rtf',
+    '.odt',
+    '.ods',
+    '.odp',
+    '.epub',
+    '.mobi',
+]);
+/**
+ * True if this sitemap URL should not be requested as markdown (download path or static file).
+ */
+function isNonPageAssetUrl(url) {
+    try {
+        const u = new URL(url);
+        const path = u.pathname.replace(/\/$/, '') || '/';
+        if (path.startsWith('/download/'))
+            return true;
+        const lastDot = path.lastIndexOf('.');
+        if (lastDot === -1)
+            return false;
+        return STATIC_EXTENSIONS.has(path.slice(lastDot).toLowerCase());
+    }
+    catch {
+        return false;
+    }
+}
+function extractLocUrls(xmlBody) {
+    const urls = [];
+    const re = /<loc>([^<]+)<\/loc>/g;
+    for (let m = re.exec(xmlBody); m !== null; m = re.exec(xmlBody)) {
+        urls.push(m[1]?.trim() ?? '');
+    }
+    return urls.filter(Boolean);
+}
+/**
+ * Collect all page URLs from sitemap.xml and sitemap-unindexed.xml (raw <loc> values).
+ * Uses resolved page URL lists when present (e.g. after following sitemap index links).
+ * Used for rewrite checks so every sitemap URL is verified to not rewrite to a different path.
+ */
+export function pageUrlsFromSitemaps(validationResult) {
+    const pageUrls = new Set();
+    const sitemapUrls = validationResult.sitemapPageUrls ??
+        (validationResult.sitemapXml.body ? extractLocUrls(validationResult.sitemapXml.body) : []);
+    const unindexedUrls = validationResult.sitemapUnindexedPageUrls ??
+        (validationResult.sitemapUnindexedXml.body
+            ? extractLocUrls(validationResult.sitemapUnindexedXml.body)
+            : []);
+    for (const url of sitemapUrls)
+        pageUrls.add(url);
+    for (const url of unindexedUrls)
+        pageUrls.add(url);
+    return [...pageUrls];
+}
+/**
+ * Convert a page URL (e.g. https://example.com/about/) to markdown URL (https://example.com/about.md).
+ */
+function pageUrlToMarkdownUrl(pageUrl, baseUrl) {
+    const base = baseUrl.replace(/\/$/, '');
+    try {
+        const u = new URL(pageUrl);
+        if (u.origin !== new URL(base).origin)
+            return '';
+        const path = u.pathname.replace(/\/$/, '') || '/';
+        if (path === '/')
+            return `${base}/index.md`;
+        return `${base}${path}.md`;
+    }
+    catch {
+        return '';
+    }
+}
+/**
+ * Build markdown URL list from a list of page URLs (e.g. resolved from sitemaps).
+ * Excludes non-page assets and converts each page URL to its .md equivalent.
+ */
+function markdownUrlsFromPageUrls(baseUrl, pageUrls) {
+    const base = baseUrl.replace(/\/$/, '');
+    const markdownUrls = [];
+    for (const pageUrl of pageUrls) {
+        if (isNonPageAssetUrl(pageUrl))
+            continue;
+        const mdUrl = pageUrlToMarkdownUrl(pageUrl, base);
+        if (mdUrl)
+            markdownUrls.push(mdUrl);
+    }
+    return markdownUrls;
+}
+/**
+ * Collect all markdown URLs from sitemap.xml and sitemap-unindexed.xml (Option B).
+ * Uses resolved page URL lists when present (e.g. after following sitemap index links).
+ * Ensures every sitemap page has a corresponding markdown URL to validate.
+ */
+export function collectMarkdownUrlsFromSitemaps(baseUrl, validationResult) {
+    const pageUrls = pageUrlsFromSitemaps(validationResult);
+    return markdownUrlsFromPageUrls(baseUrl, pageUrls);
+}
+/**
+ * Convert a production page URL to the same path on a development base URL.
+ * Keeps pathname (and search/hash); only the origin is replaced.
+ */
+function toDevPageUrl(prodPageUrl, devBaseUrl) {
+    const devBase = devBaseUrl.replace(/\/$/, '');
+    try {
+        const u = new URL(prodPageUrl);
+        const devOrigin = new URL(devBase).origin;
+        return `${devOrigin}${u.pathname}${u.search}${u.hash}`;
+    }
+    catch {
+        return '';
+    }
+}
+/**
+ * Map production page URLs to development page URLs (same path, dev origin).
+ * Excludes non-page assets (download paths, static file extensions).
+ * Used in compare mode: sitemap from prod, check existence on dev.
+ */
+export function mapProdPageUrlsToDev(prodPageUrls, devBaseUrl) {
+    const devUrls = [];
+    for (const prodUrl of prodPageUrls) {
+        if (isNonPageAssetUrl(prodUrl))
+            continue;
+        const devUrl = toDevPageUrl(prodUrl, devBaseUrl);
+        if (devUrl)
+            devUrls.push(devUrl);
+    }
+    return devUrls;
+}
+//# sourceMappingURL=collect-markdown-urls.js.map

package/dist/collect-markdown-urls.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"collect-markdown-urls.js","sourceRoot":"","sources":["../src/collect-markdown-urls.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAIH,+FAA+F;AAC/F,MAAM,iBAAiB,GAAG,IAAI,GAAG,CAAC;IAChC,MAAM;IACN,MAAM;IACN,MAAM;IACN,OAAO;IACP,MAAM;IACN,OAAO;IACP,MAAM;IACN,OAAO;IACP,MAAM;IACN,MAAM;IACN,MAAM;IACN,MAAM;IACN,MAAM;IACN,MAAM;IACN,OAAO;IACP,OAAO;CACR,CAAC,CAAC;AAEH;;GAEG;AACH,SAAS,iBAAiB,CAAC,GAAW;IACpC,IAAI,CAAC;QACH,MAAM,CAAC,GAAG,IAAI,GAAG,CAAC,GAAG,CAAC,CAAC;QACvB,MAAM,IAAI,GAAG,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,IAAI,GAAG,CAAC;QAClD,IAAI,IAAI,CAAC,UAAU,CAAC,YAAY,CAAC;YAAE,OAAO,IAAI,CAAC;QAC/C,MAAM,OAAO,GAAG,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,CAAC;QACtC,IAAI,OAAO,KAAK,CAAC,CAAC;YAAE,OAAO,KAAK,CAAC;QACjC,OAAO,iBAAiB,CAAC,GAAG,CAAC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC;IAClE,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,KAAK,CAAC;IACf,CAAC;AACH,CAAC;AAED,SAAS,cAAc,CAAC,OAAe;IACrC,MAAM,IAAI,GAAa,EAAE,CAAC;IAC1B,MAAM,EAAE,GAAG,sBAAsB,CAAC;IAClC,KAAK,IAAI,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,KAAK,IAAI,EAAE,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC;QAChE,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,CAAC;IAChC,CAAC;IACD,OAAO,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC;AAC9B,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,oBAAoB,CAAC,gBAAkC;IACrE,MAAM,QAAQ,GAAG,IAAI,GAAG,EAAU,CAAC;IACnC,MAAM,WAAW,GACf,gBAAgB,CAAC,eAAe;QAChC,CAAC,gBAAgB,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,cAAc,CAAC,gBAAgB,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;IAC7F,MAAM,aAAa,GACjB,gBAAgB,CAAC,wBAAwB;QACzC,CAAC,gBAAgB,CAAC,mBAAmB,CAAC,IAAI;YACxC,CAAC,CAAC,cAAc,CAAC,gBAAgB,CAAC,mBAAmB,CAAC,IAAI,CAAC;YAC3D,CAAC,CAAC,EAAE,CAAC,CAAC;IACV,KAAK,MAAM,GAAG,IAAI,WAAW;QAAE,QAAQ,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;IACjD,KAAK,MAAM,GAAG,IAAI,aAAa;QAAE,QAAQ,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;IACnD,OAAO,CAAC,GAAG,QAAQ,CAAC,CAAC;AACvB,CAAC;AAED;;GAEG;AACH,SAAS,oBAAoB,CAAC,OAAe,EAAE,OAAe;IAC5D,MAAM,IAAI,GAAG,OAAO,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;IACxC,IAAI,CAAC;QACH,MAAM,CAAC,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC;QAC3B,IAAI,CAAC,CAAC,MAAM,KAAK,IAAI,GAAG,CAAC,IAAI,CAAC,CAAC,MAAM;YAAE,OAAO,EAAE,CAAC;QACjD,MAAM,IAAI,GAAG,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,IAAI,GAAG,CAAC;QAClD,IAAI,IAAI,KAAK,GAAG;YAAE,OAAO,GAAG,IAAI,WAAW,CAAC;QAC5C,OAAO,GAAG,IAAI,GAAG,IAAI,KAAK,CAAC;IAC7B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,EAAE,CAAC;IACZ,CAAC;AACH,CAAC;AAED;;;GAGG;AACH,SAAS,wBAAwB,CAAC,OAAe,EAAE,QAAkB;IACnE,MAAM,IAAI,GAAG,OAAO,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;IACxC,MAAM,YAAY,GAAa,EAAE,CAAC;IAClC,KAAK,MAAM,OAAO,IAAI,QAAQ,EAAE,CAAC;QAC/B,IAAI,iBAAiB,CAAC,OAAO,CAAC;YAAE,SAAS;QACzC,MAAM,KAAK,GAAG,oBAAoB,CAAC,OAAO,EAAE,IAAI,CAAC,CAAC;QAClD,IAAI,KAAK;YAAE,YAAY,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;IACtC,CAAC;IACD,OAAO,YAAY,CAAC;AACtB,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,+BAA+B,CAC7C,OAAe,EACf,gBAAkC;IAElC,MAAM,QAAQ,GAAG,oBAAoB,CAAC,gBAAgB,CAAC,CAAC;IACxD,OAAO,wBAAwB,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC;AACrD,CAAC;AAED;;;GAGG;AACH,SAAS,YAAY,CAAC,WAAmB,EAAE,UAAkB;IAC3D,MAAM,OAAO,GAAG,UAAU,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;IAC9C,IAAI,CAAC;QACH,MAAM,CAAC,GAAG,IAAI,GAAG,CAAC,WAAW,CAAC,CAAC;QAC/B,MAAM,SAAS,GAAG,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC;QAC1C,OAAO,GAAG,SAAS,GAAG,CAAC,CAAC,QAAQ,GAAG,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC;IACzD,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,EAAE,CAAC;IACZ,CAAC;AACH,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,oBAAoB,CAAC,YAAsB,EAAE,UAAkB;IAC7E,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,KAAK,MAAM,OAAO,IAAI,YAAY,EAAE,CAAC;QACnC,IAAI,iBAAiB,CAAC,OAAO,CAAC;YAAE,SAAS;QACzC,MAAM,MAAM,GAAG,YAAY,CAAC,OAAO,EAAE,UAAU,CAAC,CAAC;QACjD,IAAI,MAAM;YAAE,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IACnC,CAAC;IACD,OAAO,OAAO,CAAC;AACjB,CAAC"}

package/dist/download.d.ts

@@ -0,0 +1,19 @@
+/**
+ * Download markdown files from URLs and save to output directory preserving path structure.
+ */
+interface DownloadResult {
+    saved: number;
+    skipped: number;
+    errors: Array<{
+        url: string;
+        status?: number;
+        message: string;
+    }>;
+}
+/**
+ * Fetch each markdown URL and write body to outDir/{relativePath}. Create parent dirs as needed.
+ * Non-2xx: log and skip (do not write). Return counts and errors.
+ */
+export declare function downloadMarkdownFiles(markdownUrls: string[], outDir: string, fetchInit: RequestInit): Promise<DownloadResult>;
+export {};
+//# sourceMappingURL=download.d.ts.map

package/dist/download.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"download.d.ts","sourceRoot":"","sources":["../src/download.ts"],"names":[],"mappings":"AAAA;;GAEG;AAmBH,UAAU,cAAc;IACtB,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,EAAE,MAAM,CAAC;IAChB,MAAM,EAAE,KAAK,CAAC;QAAE,GAAG,EAAE,MAAM,CAAC;QAAC,MAAM,CAAC,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;CAClE;AAED;;;GAGG;AACH,wBAAsB,qBAAqB,CACzC,YAAY,EAAE,MAAM,EAAE,EACtB,MAAM,EAAE,MAAM,EACd,SAAS,EAAE,WAAW,GACrB,OAAO,CAAC,cAAc,CAAC,CAwBzB"}

package/dist/download.js

@@ -0,0 +1,48 @@
+/**
+ * Download markdown files from URLs and save to output directory preserving path structure.
+ */
+import * as fs from 'node:fs';
+import * as path from 'node:path';
+/**
+ * Derive relative file path from markdown URL (e.g. .../blog/foo.md -> blog/foo.md, .../index.md -> index.md).
+ */
+function urlToRelativePath(markdownUrl) {
+    try {
+        const u = new URL(markdownUrl);
+        let p = u.pathname.replace(/^\//, '') || 'index.md';
+        if (p.endsWith('/'))
+            p = `${p}index.md`;
+        return p || 'index.md';
+    }
+    catch {
+        return 'index.md';
+    }
+}
+/**
+ * Fetch each markdown URL and write body to outDir/{relativePath}. Create parent dirs as needed.
+ * Non-2xx: log and skip (do not write). Return counts and errors.
+ */
+export async function downloadMarkdownFiles(markdownUrls, outDir, fetchInit) {
+    const result = { saved: 0, skipped: 0, errors: [] };
+    for (const url of markdownUrls) {
+        const res = await fetch(url, fetchInit);
+        if (!res.ok) {
+            result.skipped++;
+            result.errors.push({
+                url,
+                status: res.status,
+                message: `HTTP ${res.status}`,
+            });
+            continue;
+        }
+        const body = await res.text();
+        const relativePath = urlToRelativePath(url);
+        const filePath = path.join(outDir, relativePath);
+        const dir = path.dirname(filePath);
+        fs.mkdirSync(dir, { recursive: true });
+        fs.writeFileSync(filePath, body, 'utf8');
+        result.saved++;
+    }
+    return result;
+}
+//# sourceMappingURL=download.js.map

package/dist/download.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"download.js","sourceRoot":"","sources":["../src/download.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,KAAK,EAAE,MAAM,SAAS,CAAC;AAC9B,OAAO,KAAK,IAAI,MAAM,WAAW,CAAC;AAElC;;GAEG;AACH,SAAS,iBAAiB,CAAC,WAAmB;IAC5C,IAAI,CAAC;QACH,MAAM,CAAC,GAAG,IAAI,GAAG,CAAC,WAAW,CAAC,CAAC;QAC/B,IAAI,CAAC,GAAG,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,IAAI,UAAU,CAAC;QACpD,IAAI,CAAC,CAAC,QAAQ,CAAC,GAAG,CAAC;YAAE,CAAC,GAAG,GAAG,CAAC,UAAU,CAAC;QACxC,OAAO,CAAC,IAAI,UAAU,CAAC;IACzB,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,UAAU,CAAC;IACpB,CAAC;AACH,CAAC;AAQD;;;GAGG;AACH,MAAM,CAAC,KAAK,UAAU,qBAAqB,CACzC,YAAsB,EACtB,MAAc,EACd,SAAsB;IAEtB,MAAM,MAAM,GAAmB,EAAE,KAAK,EAAE,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,MAAM,EAAE,EAAE,EAAE,CAAC;IAEpE,KAAK,MAAM,GAAG,IAAI,YAAY,EAAE,CAAC;QAC/B,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,GAAG,EAAE,SAAS,CAAC,CAAC;QACxC,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;YACZ,MAAM,CAAC,OAAO,EAAE,CAAC;YACjB,MAAM,CAAC,MAAM,CAAC,IAAI,CAAC;gBACjB,GAAG;gBACH,MAAM,EAAE,GAAG,CAAC,MAAM;gBAClB,OAAO,EAAE,QAAQ,GAAG,CAAC,MAAM,EAAE;aAC9B,CAAC,CAAC;YACH,SAAS;QACX,CAAC;QACD,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC;QAC9B,MAAM,YAAY,GAAG,iBAAiB,CAAC,GAAG,CAAC,CAAC;QAC5C,MAAM,QAAQ,GAAG,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,YAAY,CAAC,CAAC;QACjD,MAAM,GAAG,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;QACnC,EAAE,CAAC,SAAS,CAAC,GAAG,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;QACvC,EAAE,CAAC,aAAa,CAAC,QAAQ,EAAE,IAAI,EAAE,MAAM,CAAC,CAAC;QACzC,MAAM,CAAC,KAAK,EAAE,CAAC;IACjB,CAAC;IAED,OAAO,MAAM,CAAC;AAChB,CAAC"}

package/dist/fetch-options.d.ts

@@ -0,0 +1,19 @@
+/**
+ * Build shared RequestInit (headers) for all fetch calls.
+ * Supports generic -H / --header and --vercel-bypass for Vercel Deployment Protection.
+ */
+interface FetchOptionsInput {
+    /** Headers from -H / --header (e.g. ["Name: value"]) */
+    headerArgs?: string[];
+    /** Secret for --vercel-bypass; if flag set but empty, use env */
+    vercelBypassSecret?: string | null;
+    /** Whether --vercel-bypass was passed (so we use env if secret not provided) */
+    vercelBypassRequested?: boolean;
+}
+/**
+ * Build RequestInit with headers from CLI args.
+ * Vercel bypass is applied first; user -H headers override.
+ */
+export declare function buildFetchOptions(input: FetchOptionsInput): RequestInit;
+export {};
+//# sourceMappingURL=fetch-options.d.ts.map

package/dist/fetch-options.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"fetch-options.d.ts","sourceRoot":"","sources":["../src/fetch-options.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAKH,UAAU,iBAAiB;IACzB,wDAAwD;IACxD,UAAU,CAAC,EAAE,MAAM,EAAE,CAAC;IACtB,iEAAiE;IACjE,kBAAkB,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACnC,gFAAgF;IAChF,qBAAqB,CAAC,EAAE,OAAO,CAAC;CACjC;AAqBD;;;GAGG;AACH,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,iBAAiB,GAAG,WAAW,CAuBvE"}

package/dist/fetch-options.js

@@ -0,0 +1,49 @@
+/**
+ * Build shared RequestInit (headers) for all fetch calls.
+ * Supports generic -H / --header and --vercel-bypass for Vercel Deployment Protection.
+ */
+const VERCEL_BYPASS_HEADER = 'x-vercel-protection-bypass';
+const VERCEL_BYPASS_ENV = 'VERCEL_AUTOMATION_BYPASS_SECRET';
+/**
+ * Parse a "Name: value" header string. Handles optional quotes around value.
+ */
+function parseHeaderArg(arg) {
+    const trimmed = arg.trim();
+    const colonIdx = trimmed.indexOf(':');
+    if (colonIdx <= 0)
+        return null;
+    const name = trimmed.slice(0, colonIdx).trim();
+    let value = trimmed.slice(colonIdx + 1).trim();
+    if ((value.startsWith('"') && value.endsWith('"')) ||
+        (value.startsWith("'") && value.endsWith("'"))) {
+        value = value.slice(1, -1);
+    }
+    if (!name)
+        return null;
+    return { name, value };
+}
+/**
+ * Build RequestInit with headers from CLI args.
+ * Vercel bypass is applied first; user -H headers override.
+ */
+export function buildFetchOptions(input) {
+    const headers = new Headers();
+    // 1. Vercel bypass (if requested)
+    const bypassSecret = input.vercelBypassSecret !== undefined && input.vercelBypassSecret !== null
+        ? input.vercelBypassSecret
+        : input.vercelBypassRequested
+            ? (process.env[VERCEL_BYPASS_ENV] ?? '')
+            : '';
+    if (bypassSecret) {
+        headers.set(VERCEL_BYPASS_HEADER, bypassSecret);
+    }
+    // 2. User -H headers (override bypass if same name)
+    for (const arg of input.headerArgs ?? []) {
+        const parsed = parseHeaderArg(arg);
+        if (parsed) {
+            headers.set(parsed.name, parsed.value);
+        }
+    }
+    return { headers };
+}
+//# sourceMappingURL=fetch-options.js.map

package/dist/fetch-options.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"fetch-options.js","sourceRoot":"","sources":["../src/fetch-options.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,MAAM,oBAAoB,GAAG,4BAA4B,CAAC;AAC1D,MAAM,iBAAiB,GAAG,iCAAiC,CAAC;AAW5D;;GAEG;AACH,SAAS,cAAc,CAAC,GAAW;IACjC,MAAM,OAAO,GAAG,GAAG,CAAC,IAAI,EAAE,CAAC;IAC3B,MAAM,QAAQ,GAAG,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;IACtC,IAAI,QAAQ,IAAI,CAAC;QAAE,OAAO,IAAI,CAAC;IAC/B,MAAM,IAAI,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,EAAE,QAAQ,CAAC,CAAC,IAAI,EAAE,CAAC;IAC/C,IAAI,KAAK,GAAG,OAAO,CAAC,KAAK,CAAC,QAAQ,GAAG,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;IAC/C,IACE,CAAC,KAAK,CAAC,UAAU,CAAC,GAAG,CAAC,IAAI,KAAK,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC;QAC9C,CAAC,KAAK,CAAC,UAAU,CAAC,GAAG,CAAC,IAAI,KAAK,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,EAC9C,CAAC;QACD,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;IAC7B,CAAC;IACD,IAAI,CAAC,IAAI;QAAE,OAAO,IAAI,CAAC;IACvB,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC;AACzB,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,iBAAiB,CAAC,KAAwB;IACxD,MAAM,OAAO,GAAG,IAAI,OAAO,EAAE,CAAC;IAE9B,kCAAkC;IAClC,MAAM,YAAY,GAChB,KAAK,CAAC,kBAAkB,KAAK,SAAS,IAAI,KAAK,CAAC,kBAAkB,KAAK,IAAI;QACzE,CAAC,CAAC,KAAK,CAAC,kBAAkB;QAC1B,CAAC,CAAC,KAAK,CAAC,qBAAqB;YAC3B,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,iBAAiB,CAAC,IAAI,EAAE,CAAC;YACxC,CAAC,CAAC,EAAE,CAAC;IACX,IAAI,YAAY,EAAE,CAAC;QACjB,OAAO,CAAC,GAAG,CAAC,oBAAoB,EAAE,YAAY,CAAC,CAAC;IAClD,CAAC;IAED,oDAAoD;IACpD,KAAK,MAAM,GAAG,IAAI,KAAK,CAAC,UAAU,IAAI,EAAE,EAAE,CAAC;QACzC,MAAM,MAAM,GAAG,cAAc,CAAC,GAAG,CAAC,CAAC;QACnC,IAAI,MAAM,EAAE,CAAC;YACX,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,IAAI,EAAE,MAAM,CAAC,KAAK,CAAC,CAAC;QACzC,CAAC;IACH,CAAC;IAED,OAAO,EAAE,OAAO,EAAE,CAAC;AACrB,CAAC"}

package/dist/validate.d.ts

@@ -0,0 +1,32 @@
+/**
+ * Fetch and validate sitemap.xml, sitemap-unindexed.xml, and llms.txt.
+ * When sitemap.xml or sitemap-unindexed.xml is a sitemap index, follows child
+ * sitemaps and collects page URLs from their urlset content.
+ */
+export interface ValidationResult {
+    sitemapXml: {
+        ok: boolean;
+        status: number;
+        body?: string;
+    };
+    sitemapUnindexedXml: {
+        ok: boolean;
+        status: number;
+        body?: string;
+        warned?: boolean;
+    };
+    llmsTxt: {
+        ok: boolean;
+        status: number;
+        body?: string;
+    };
+    /** Page URLs from sitemap.xml (resolved from child sitemaps when root is an index). */
+    sitemapPageUrls?: string[];
+    /** Page URLs from sitemap-unindexed.xml (resolved from child sitemaps when root is an index). */
+    sitemapUnindexedPageUrls?: string[];
+    /** True when sitemap.xml was a sitemap index (child sitemaps were followed). */
+    sitemapWasIndex?: boolean;
+}
+export declare function validate(baseUrl: string, fetchInit: RequestInit): Promise<ValidationResult>;
+export declare function validationFailed(result: ValidationResult): boolean;
+//# sourceMappingURL=validate.d.ts.map

package/dist/validate.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"validate.d.ts","sourceRoot":"","sources":["../src/validate.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,MAAM,WAAW,gBAAgB;IAC/B,UAAU,EAAE;QAAE,EAAE,EAAE,OAAO,CAAC;QAAC,MAAM,EAAE,MAAM,CAAC;QAAC,IAAI,CAAC,EAAE,MAAM,CAAA;KAAE,CAAC;IAC3D,mBAAmB,EAAE;QAAE,EAAE,EAAE,OAAO,CAAC;QAAC,MAAM,EAAE,MAAM,CAAC;QAAC,IAAI,CAAC,EAAE,MAAM,CAAC;QAAC,MAAM,CAAC,EAAE,OAAO,CAAA;KAAE,CAAC;IACtF,OAAO,EAAE;QAAE,EAAE,EAAE,OAAO,CAAC;QAAC,MAAM,EAAE,MAAM,CAAC;QAAC,IAAI,CAAC,EAAE,MAAM,CAAA;KAAE,CAAC;IACxD,uFAAuF;IACvF,eAAe,CAAC,EAAE,MAAM,EAAE,CAAC;IAC3B,iGAAiG;IACjG,wBAAwB,CAAC,EAAE,MAAM,EAAE,CAAC;IACpC,gFAAgF;IAChF,eAAe,CAAC,EAAE,OAAO,CAAC;CAC3B;AAwCD,wBAAsB,QAAQ,CAAC,OAAO,EAAE,MAAM,EAAE,SAAS,EAAE,WAAW,GAAG,OAAO,CAAC,gBAAgB,CAAC,CAuCjG;AAED,wBAAgB,gBAAgB,CAAC,MAAM,EAAE,gBAAgB,GAAG,OAAO,CAElE"}

package/dist/validate.js

@@ -0,0 +1,79 @@
+/**
+ * Fetch and validate sitemap.xml, sitemap-unindexed.xml, and llms.txt.
+ * When sitemap.xml or sitemap-unindexed.xml is a sitemap index, follows child
+ * sitemaps and collects page URLs from their urlset content.
+ */
+function hasLocTag(body) {
+    return /<loc>[^<]+<\/loc>/.test(body);
+}
+function extractLocUrls(xmlBody) {
+    const urls = [];
+    const re = /<loc>([^<]+)<\/loc>/g;
+    for (let m = re.exec(xmlBody); m !== null; m = re.exec(xmlBody)) {
+        const u = m[1]?.trim();
+        if (u)
+            urls.push(u);
+    }
+    return urls;
+}
+function isSitemapIndex(body) {
+    return /<sitemapindex[\s>]/.test(body) || body.includes('<sitemap>');
+}
+async function resolveSitemapToPageUrls(body, fetchInit) {
+    if (!isSitemapIndex(body)) {
+        const pageUrls = extractLocUrls(body);
+        return { ok: pageUrls.length > 0 || hasLocTag(body), pageUrls };
+    }
+    const childUrls = extractLocUrls(body);
+    const allPageUrls = [];
+    for (const childUrl of childUrls) {
+        const res = await fetch(childUrl, fetchInit);
+        if (!res.ok)
+            return { ok: false, pageUrls: [] };
+        const childBody = await res.text();
+        // Empty child sitemaps (0 <loc> entries) are valid; they contribute no URLs.
+        allPageUrls.push(...extractLocUrls(childBody));
+    }
+    return { ok: allPageUrls.length > 0, pageUrls: allPageUrls };
+}
+export async function validate(baseUrl, fetchInit) {
+    const base = baseUrl.replace(/\/$/, '');
+    const result = {
+        sitemapXml: { ok: false, status: 0 },
+        sitemapUnindexedXml: { ok: false, status: 0 },
+        llmsTxt: { ok: false, status: 0 },
+    };
+    const sitemapRes = await fetch(`${base}/sitemap.xml`, fetchInit);
+    result.sitemapXml.status = sitemapRes.status;
+    if (sitemapRes.ok) {
+        const body = await sitemapRes.text();
+        result.sitemapXml.body = body;
+        const resolved = await resolveSitemapToPageUrls(body, fetchInit);
+        result.sitemapXml.ok = resolved.ok;
+        result.sitemapPageUrls = resolved.pageUrls;
+        result.sitemapWasIndex = isSitemapIndex(body);
+    }
+    const unindexedRes = await fetch(`${base}/sitemap-unindexed.xml`, fetchInit);
+    result.sitemapUnindexedXml.status = unindexedRes.status;
+    if (unindexedRes.status === 404) {
+        result.sitemapUnindexedXml.warned = true;
+    }
+    else if (unindexedRes.ok) {
+        const body = await unindexedRes.text();
+        result.sitemapUnindexedXml.body = body;
+        const resolved = await resolveSitemapToPageUrls(body, fetchInit);
+        result.sitemapUnindexedXml.ok = resolved.ok;
+        result.sitemapUnindexedPageUrls = resolved.pageUrls;
+    }
+    const llmsRes = await fetch(`${base}/llms.txt`, fetchInit);
+    result.llmsTxt.status = llmsRes.status;
+    if (llmsRes.ok) {
+        result.llmsTxt.body = await llmsRes.text();
+        result.llmsTxt.ok = true;
+    }
+    return result;
+}
+export function validationFailed(result) {
+    return !result.sitemapXml.ok || !result.llmsTxt.ok;
+}
+//# sourceMappingURL=validate.js.map

package/dist/validate.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"validate.js","sourceRoot":"","sources":["../src/validate.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAcH,SAAS,SAAS,CAAC,IAAY;IAC7B,OAAO,mBAAmB,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AACxC,CAAC;AAED,SAAS,cAAc,CAAC,OAAe;IACrC,MAAM,IAAI,GAAa,EAAE,CAAC;IAC1B,MAAM,EAAE,GAAG,sBAAsB,CAAC;IAClC,KAAK,IAAI,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC,KAAK,IAAI,EAAE,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC;QAChE,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,IAAI,EAAE,CAAC;QACvB,IAAI,CAAC;YAAE,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACtB,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC;AAED,SAAS,cAAc,CAAC,IAAY;IAClC,OAAO,oBAAoB,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAC;AACvE,CAAC;AAED,KAAK,UAAU,wBAAwB,CACrC,IAAY,EACZ,SAAsB;IAEtB,IAAI,CAAC,cAAc,CAAC,IAAI,CAAC,EAAE,CAAC;QAC1B,MAAM,QAAQ,GAAG,cAAc,CAAC,IAAI,CAAC,CAAC;QACtC,OAAO,EAAE,EAAE,EAAE,QAAQ,CAAC,MAAM,GAAG,CAAC,IAAI,SAAS,CAAC,IAAI,CAAC,EAAE,QAAQ,EAAE,CAAC;IAClE,CAAC;IACD,MAAM,SAAS,GAAG,cAAc,CAAC,IAAI,CAAC,CAAC;IACvC,MAAM,WAAW,GAAa,EAAE,CAAC;IACjC,KAAK,MAAM,QAAQ,IAAI,SAAS,EAAE,CAAC;QACjC,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;QAC7C,IAAI,CAAC,GAAG,CAAC,EAAE;YAAE,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,QAAQ,EAAE,EAAE,EAAE,CAAC;QAChD,MAAM,SAAS,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC;QACnC,6EAA6E;QAC7E,WAAW,CAAC,IAAI,CAAC,GAAG,cAAc,CAAC,SAAS,CAAC,CAAC,CAAC;IACjD,CAAC;IACD,OAAO,EAAE,EAAE,EAAE,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,QAAQ,EAAE,WAAW,EAAE,CAAC;AAC/D,CAAC;AAED,MAAM,CAAC,KAAK,UAAU,QAAQ,CAAC,OAAe,EAAE,SAAsB;IACpE,MAAM,IAAI,GAAG,OAAO,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;IACxC,MAAM,MAAM,GAAqB;QAC/B,UAAU,EAAE,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,CAAC,EAAE;QACpC,mBAAmB,EAAE,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,CAAC,EAAE;QAC7C,OAAO,EAAE,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,CAAC,EAAE;KAClC,CAAC;IAEF,MAAM,UAAU,GAAG,MAAM,KAAK,CAAC,GAAG,IAAI,cAAc,EAAE,SAAS,CAAC,CAAC;IACjE,MAAM,CAAC,UAAU,CAAC,MAAM,GAAG,UAAU,CAAC,MAAM,CAAC;IAC7C,IAAI,UAAU,CAAC,EAAE,EAAE,CAAC;QAClB,MAAM,IAAI,GAAG,MAAM,UAAU,CAAC,IAAI,EAAE,CAAC;QACrC,MAAM,CAAC,UAAU,CAAC,IAAI,GAAG,IAAI,CAAC;QAC9B,MAAM,QAAQ,GAAG,MAAM,wBAAwB,CAAC,IAAI,EAAE,SAAS,CAAC,CAAC;QACjE,MAAM,CAAC,UAAU,CAAC,EAAE,GAAG,QAAQ,CAAC,EAAE,CAAC;QACnC,MAAM,CAAC,eAAe,GAAG,QAAQ,CAAC,QAAQ,CAAC;QAC3C,MAAM,CAAC,eAAe,GAAG,cAAc,CAAC,IAAI,CAAC,CAAC;IAChD,CAAC;IAED,MAAM,YAAY,GAAG,MAAM,KAAK,CAAC,GAAG,IAAI,wBAAwB,EAAE,SAAS,CAAC,CAAC;IAC7E,MAAM,CAAC,mBAAmB,CAAC,MAAM,GAAG,YAAY,CAAC,MAAM,CAAC;IACxD,IAAI,YAAY,CAAC,MAAM,KAAK,GAAG,EAAE,CAAC;QAChC,MAAM,CAAC,mBAAmB,CAAC,MAAM,GAAG,IAAI,CAAC;IAC3C,CAAC;SAAM,IAAI,YAAY,CAAC,EAAE,EAAE,CAAC;QAC3B,MAAM,IAAI,GAAG,MAAM,YAAY,CAAC,IAAI,EAAE,CAAC;QACvC,MAAM,CAAC,mBAAmB,CAAC,IAAI,GAAG,IAAI,CAAC;QACvC,MAAM,QAAQ,GAAG,MAAM,wBAAwB,CAAC,IAAI,EAAE,SAAS,CAAC,CAAC;QACjE,MAAM,CAAC,mBAAmB,CAAC,EAAE,GAAG,QAAQ,CAAC,EAAE,CAAC;QAC5C,MAAM,CAAC,wBAAwB,GAAG,QAAQ,CAAC,QAAQ,CAAC;IACtD,CAAC;IAED,MAAM,OAAO,GAAG,MAAM,KAAK,CAAC,GAAG,IAAI,WAAW,EAAE,SAAS,CAAC,CAAC;IAC3D,MAAM,CAAC,OAAO,CAAC,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC;IACvC,IAAI,OAAO,CAAC,EAAE,EAAE,CAAC;QACf,MAAM,CAAC,OAAO,CAAC,IAAI,GAAG,MAAM,OAAO,CAAC,IAAI,EAAE,CAAC;QAC3C,MAAM,CAAC,OAAO,CAAC,EAAE,GAAG,IAAI,CAAC;IAC3B,CAAC;IAED,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,MAAM,UAAU,gBAAgB,CAAC,MAAwB;IACvD,OAAO,CAAC,MAAM,CAAC,UAAU,CAAC,EAAE,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;AACrD,CAAC"}

package/package.json

@@ -0,0 +1,45 @@
+{
+  "name": "@se-studio/site-check",
+  "version": "1.0.1",
+  "description": "Validate SE marketing sites (sitemap, llms.txt) and download markdown files preserving structure",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/Something-Else-Studio/se-core-product",
+    "directory": "packages/site-check"
+  },
+  "license": "MIT",
+  "type": "module",
+  "main": "./dist/cli.js",
+  "bin": {
+    "site-check": "./dist/cli.js"
+  },
+  "files": [
+    "dist",
+    "*.md"
+  ],
+  "keywords": [
+    "sitemap",
+    "llms.txt",
+    "markdown",
+    "validation",
+    "cli",
+    "vercel",
+    "deployment-protection"
+  ],
+  "homepage": "https://github.com/Something-Else-Studio/se-core-product#readme",
+  "bugs": {
+    "url": "https://github.com/Something-Else-Studio/se-core-product/issues"
+  },
+  "devDependencies": {
+    "@biomejs/biome": "^2.4.6",
+    "@types/node": "^22.19.15",
+    "typescript": "^5.9.3"
+  },
+  "scripts": {
+    "build": "tsc --project tsconfig.build.json",
+    "dev": "tsc --project tsconfig.build.json --watch",
+    "type-check": "tsc --noEmit",
+    "lint": "biome lint .",
+    "clean": "rm -rf dist .turbo *.tsbuildinfo"
+  }
+}