@d-zero/page-cluster 0.5.6 → 0.5.7

package/README.md

@@ -79,6 +79,7 @@ jq -c '.[]' crawl-output.json | page-cluster > clusters.jsonl
 
 - `--content-block-attribute <name>` — CMS が自由編集コンテンツブロックに付与している属性名（例: `data-bgb`）が分かっている場合に指定する。指定すると比較前にその属性を持つ要素配下を無視するので、同じテンプレートで本文構成だけ違うページを混同しなくなる。唯一の site-specific なオプションで、未指定でも `<main>` / `role="main"` を起点にした自動深さキャップが常時働く（詳細は `resolve-page-cluster-keys.ts` の JSDoc を参照）
 - `--cluster-reasons-file <path>` — 上記の「クラスタ選定理由」を `<path>` に JSON として書き出す。ページ数の上限はない。20,000 ページ以下のコーパスでは、指定すると進捗表示（後述）は出なくなる（進捗を出さない非ストリーミング経路に常に振り分けられるため。20,000 ページ超のストリーミング経路では進捗表示・クラスタ理由の両方が動く）
+- `--validation-file <path>` — 分類結果の事後検証レポート（下記「分類結果の事後検証」参照）を `<path>` に JSON として書き出す。指定すると、レポートの検出結果のうち安全に統合できるもの（構造トークン集合が完全一致、またはミラー軸による裏付けあり）が **stdout の `clusterKey` にも自動的に反映される**
 - `--help` / `-h` — ヘルプを表示する
 - `--version` / `-v` — バージョンを表示する
 
@@ -119,6 +120,13 @@ silence したい場合は `2>/dev/null`。ログに残したい場合は `2> pr
 | `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類                                                                                                                                                                                                                   |
 | `@d-zero/page-cluster/is-chrome-landmark-instance`   | `isChromeLandmarkInstance` — 1 つの landmark インスタンスのトークン集合と `ClusterReason.landmarks[type].shellTokens` のようなシェルトークン集合を突き合わせて chrome/content を判定するステートレス関数                                                                                            |
 | `@d-zero/page-cluster/jaccard-similarity`            | `jaccardSimilarity` — 2 つのトークン集合の Jaccard 類似度。`ClusterReason` 同士（`structuralCoreTokens` や `shellTokens`）を比較して兄弟クラスタとの差分を調べる用途などに使う                                                                                                                      |
+| `@d-zero/page-cluster/validate-cluster-partition`    | `ClusterPartitionReport` 型、`validateClusterPartition` — 分類済みの `clusterKey` を、分類そのものではなく分類結果同士を突き合わせて事後検証する（下記「分類結果の事後検証」参照）。通常は `resolvePageClusterKeys` の `onPartitionReport` 経由で使うので直接呼ぶ必要はない                         |
+| `@d-zero/page-cluster/find-cross-cluster-duplicates` | `ClusteredPage` 型、`CrossClusterDuplicate` 型、`findCrossClusterDuplicates` — 別クラスタに分かれているが同一テンプレートらしいページ対を検出する                                                                                                                                                   |
+| `@d-zero/page-cluster/compute-cluster-cohesion`      | `ClusterCohesion` 型、`computeClusterCohesion` — クラスタ内メンバーが互いにどれだけ似ているかを分布として報告する（無関係なテンプレートが混ざったクラスタの検出に使う）                                                                                                                             |
+| `@d-zero/page-cluster/detect-mirror-axis`            | `MirrorAxis` 型、`detectMirrorAxis` — 言語ディレクトリなど、URL のあるセグメントだけを変えてサイトの一部をミラーしている「軸」を、事前知識なしに URL パス集合から発見する                                                                                                                           |
+| `@d-zero/page-cluster/normalize-path-by-mirror-axis` | `normalizePathByMirrorAxis` — 検出した軸に沿ってページの URL パスを正規化し、ミラー元同士を同じ形状に揃える                                                                                                                                                                                         |
+| `@d-zero/page-cluster/normalize-href-by-mirror-axis` | `normalizeHrefByMirrorAxis` — 検出した軸に沿って URL（主に stylesheet href）を正規化する                                                                                                                                                                                                            |
+| `@d-zero/page-cluster/merge-validated-clusters`      | `mergeValidatedClusters` — 確認済みのクラスタ対の統合を `clusterKey` 配列に適用する純関数                                                                                                                                                                                                           |
 
 ```ts
 import { resolvePageClusterKeysFromArray } from '@d-zero/page-cluster/resolve-page-cluster-keys';
@@ -192,6 +200,33 @@ flowchart TD
 
 マージが起きるとユニットのメンバー構成が変わり、文書頻度も quorum core も変わる。そのため毎ラウンド、統合後のプールから全指標を**再計算**してマージを再試行する。fine stage・L2 stage の両方でマージが 1 件も出なくなった時点で不動点に到達したとみなして収束する（安全弁として最大 10 ラウンド。実データでは 7 ラウンド以内に収束）。L2 stage は fine stage が空振りしたラウンドでしか実行されない最後の粗い経路で、誤マージ防止のために shell（ランドマーク由来トークン）の相互裏付けを要求する。
 
+fine stage・L2 stage いずれの経路で提案されたマージも、適用前に**凝集度ガード**を通る: 統合後の quorum core が統合前の core に対して一定比率を下回るなら、そのマージは破棄される。個々の経路が「統合前のペア類似度」だけを見て提案する一方、複数ラウンドにわたる連鎖的な統合は「統合後に実際どれだけまとまっているか」を悪化させ得る（無関係なテンプレート同士が少しずつ吸収し合う catch-all 化）ため、経路をまたいだ単一のチェックポイントとして機能する。あわせて、L2 stage 自体もラウンド開始前に判別力を検査し、参加ユニット全体が同一の署名形状に潰れている（`main` 直下の浅い階層しか手がかりが残っていない等）場合は、そのラウンドの L2 比較を丸ごとスキップする。詳細は `merge-cross-block-clusters.ts` の `filterMergesByCohesion` / `hasDiscriminatingL2Signatures` の JSDoc を参照。
+
 ### Self-tuning
 
 閾値の多くは **max-gap auto-cut**（度数分布の隣接ギャップ最大の中点を境界とする）でデータから自己発見される。① Stage A のカット高、② Stage B の shell 判定、③ chrome discovery のグローバル/ローカル判定、④ Pass 0 の URL パス深さ選択、の 4 箇所で同一プリミティブを再利用しているので、サイトごとにハイパーパラメータをチューニングする必要はない。詳細は `autoCutThreshold` の JSDoc を参照。例外的に Stage B fine stage の complete-linkage だけは固定閾値 0.8 を使う（理由は `merge-cross-block-clusters.ts` の JSDoc を参照）。
+
+## 分類結果の事後検証
+
+Stage A/B は「クラスタリング中に」正しい判断をしようとするが、判断材料はその時点でのペア類似度に限られる。分類が終わった**あとで**、確定したパーティション同士を突き合わせて検証する方が、同じ種類の誤りをかえって見つけやすい場合がある: マージ中は局所的なペア情報とマージ順序しか持たないが、事後なら分割全体を一度に見られるからだ。この事後検証は Stage A/B のアルゴリズムに一切依存しないので、`resolvePageClusterKeys` 以外で作られた分類結果（保存済みの `clusterKey` をアーカイブから読み戻した場合など）にも使える。
+
+3 つのチェックを行う。
+
+- **クラスタ間重複検出**（`findCrossClusterDuplicates`） — 別クラスタに分かれているページ対のうち、構造トークン集合が完全一致するもの（軸の裏付けなしで確定）と、ミラー軸で裏付けられる近似一致のものを検出する
+- **クラスタ内凝集度**（`computeClusterCohesion`） — クラスタ内のメンバー同士がどれだけ似ているかを中央値・10 パーセンタイル・最小値の分布として報告する。無関係なテンプレートが混ざったクラスタは、共通のシェル由来トークンだけが一致する形で `structuralCoreTokens` 自体は非空のまま残ることがあるため、core の有無だけでは過剰マージを検出できない
+- **ミラー軸の自動発見**（`detectMirrorAxis`） — 言語ディレクトリのように、URL のあるセグメントだけを変えてサイトの一部をミラーしている構造を、言語コード等の事前知識なしに発見する。同じ値集合が何種類の異なるパス骨格にわたって反復するかを数え、`autoCutThreshold` で「たまたま値が 2 つ以上あるだけの兄弟ページ」と「サイト全体を貫く軸」を切り分ける
+
+`validateClusterPartition` はこの 3 つをまとめて呼び出すエントリポイント。`ClusterReason` と同じく、判断結果ではなく構造化データだけを返す — 検出された重複をどう扱うか（統合するかどうか、`suspicious` フラグをどう解釈するか）は呼び出し側の責務。実際に統合を適用する場合は `mergeValidatedClusters` に確認済みのクラスタ対を渡す。
+
+```ts
+import { mergeValidatedClusters } from '@d-zero/page-cluster/merge-validated-clusters';
+import { validateClusterPartition } from '@d-zero/page-cluster/validate-cluster-partition';
+
+const report = validateClusterPartition(pages); // pages: { clusterKey, tokens, paths, stylesheetHrefs }[]
+const safeToMerge = report.crossClusterDuplicates.filter(
+	(d) => d.similarity === 1 || d.corroboratedByMirrorAxis,
+);
+const mergedKeys = mergeValidatedClusters(clusterKeys, safeToMerge);
+```
+
+`resolvePageClusterKeys` に `onPartitionReport` コールバックを渡すと、この検証が Stage B 完了直後に自動的に走り、上記と同じ安全な統合ポリシー（完全一致、またはミラー軸による裏付けあり）が返り値の `clusterKey` にもそのまま反映される。`onClusterReason` と同じくオプトインで、渡さない限り既存の挙動・出力は一切変わらない。詳細はその JSDoc を参照。

package/dist/build-mirrored-template-fixture.d.ts

@@ -0,0 +1,61 @@
+import type { PageClusterSignals } from './resolve-page-cluster-keys.js';
+/**
+ * Options for {@link buildMirroredTemplateFixture}.
+ */
+export type MirroredTemplateFixtureOptions = {
+    /**
+     * URL segment values a whole section is mirrored under (e.g. language
+     * codes). Each value gets its own copy of every template, at its own
+     * per-value stylesheet href — modeling a site that duplicates a section's
+     * CSS per mirror instead of sharing one file.
+     */
+    readonly axisValues?: readonly string[];
+    /** Pages generated per (template, axis value) pair. */
+    readonly pagesPerTemplate?: number;
+    /**
+     * Tag wrapping each page's content. `'main'` reproduces the L2-stage
+     * signature collapse this fixture exists to catch (see
+     * `merge-cross-block-clusters.ts`'s `l2Signature` JSDoc): every template's
+     * content sits under a `main > article > div > section > div` chain, so
+     * `l2Signature`'s class-stripped, 2-level-past-`main` truncation reduces
+     * every template to the same key set. `'div'` (or any other tag) opts out
+     * of `l2Signature`'s `main`-anchoring, so the fine stage alone must
+     * separate the templates correctly.
+     */
+    readonly wrapperTag?: 'main' | 'div';
+};
+/**
+ * One generated page's ground truth, alongside the `PageClusterSignals` a
+ * clustering call actually sees. Tests assert against `template`/`axisValue`
+ * rather than parsing them back out of a `clusterKey` string.
+ */
+export type MirroredTemplateFixturePage = {
+    readonly signals: PageClusterSignals;
+    readonly template: string;
+    readonly axisValue: string;
+    readonly page: number;
+};
+/**
+ * Builds a synthetic multi-template, multi-mirror site: every template
+ * shares a site-wide shell (header/gnav/footer) and shares
+ * breadcrumb/page-head/related content blocks, and each template's own
+ * content is a distinct ordered subset of a shared CMS module pool (see
+ * {@link MODULE_POOL}) so templates partially overlap in tokens rather than
+ * being trivially disjoint. Every template is duplicated once per
+ * `axisValues` entry, each with its own per-mirror stylesheet href — this
+ * shape collapses every template into one catch-all cluster when `main` is
+ * the wrapper tag (a `main`-anchored wrapper chain degenerates
+ * `l2Signature`), while the same site with content under a plain `<div>`
+ * clusters correctly (see `wrapperTag`'s own JSDoc).
+ * @param options
+ * @example
+ * ```ts
+ * const { pages } = buildMirroredTemplateFixture({ wrapperTag: 'main' });
+ * const keys = await resolvePageClusterKeysFromArray(pages.map((p) => p.signals));
+ * // keys collapse to a single value — the bug this fixture reproduces
+ * ```
+ */
+export declare function buildMirroredTemplateFixture(options?: MirroredTemplateFixtureOptions): {
+    readonly pages: readonly MirroredTemplateFixturePage[];
+    readonly templates: readonly string[];
+};

package/dist/build-mirrored-template-fixture.js

@@ -0,0 +1,127 @@
+/**
+ * @param count
+ * @param render
+ */
+function repeatMarkup(count, render) {
+    let out = '';
+    for (let i = 0; i < count; i++)
+        out += render(i);
+    return out;
+}
+/**
+ * Pool of CMS-style content blocks that {@link TEMPLATE_MODULES} composes
+ * templates from. Modeled after real CMS "block" systems: templates share
+ * modules with each other (a FAQ page and a contact page both use
+ * `notice`/`contact`), which is what makes this fixture exercise Stage B's
+ * cross-block containment/shape logic instead of trivially disjoint token
+ * sets.
+ */
+const MODULE_POOL = {
+    lead: () => `<div class="mod-lead"><p class="mod-lead__text">x</p><p class="mod-lead__note">n</p></div>`,
+    heading: () => `<div class="mod-heading"><h2 class="mod-heading__ttl">h</h2><p class="mod-heading__sub">s</p></div>`,
+    photo: () => `<figure class="mod-photo"><img class="mod-photo__img" src="p.jpg"><figcaption class="mod-photo__cap">c</figcaption></figure>`,
+    table: (n) => `<div class="mod-table"><table class="mod-table__table"><thead><tr><th class="mod-table__th">h</th></tr></thead><tbody>${repeatMarkup(n, (i) => `<tr class="mod-table__tr"><td class="mod-table__td">v${i}</td></tr>`)}</tbody></table></div>`,
+    accordion: (n) => `<div class="mod-accordion">${repeatMarkup(n, (i) => `<dl class="mod-accordion__item"><dt class="mod-accordion__q"><button class="mod-accordion__toggle">q${i}</button></dt><dd class="mod-accordion__a"><p class="mod-accordion__text">a${i}</p></dd></dl>`)}</div>`,
+    cards: (n) => `<ul class="mod-cards">${repeatMarkup(n, (i) => `<li class="mod-cards__item"><a class="mod-cards__link" href="#${i}"><span class="mod-cards__fig"><img class="mod-cards__img" src="c.jpg"></span><span class="mod-cards__ttl">c${i}</span></a></li>`)}</ul>`,
+    steps: (n) => `<ol class="mod-steps">${repeatMarkup(n, (i) => `<li class="mod-steps__item"><span class="mod-steps__num">${i}</span><h3 class="mod-steps__ttl">s${i}</h3><p class="mod-steps__text">d${i}</p></li>`)}</ol>`,
+    map: () => `<div class="mod-map"><iframe class="mod-map__frame" src="map"></iframe><p class="mod-map__note">m</p></div>`,
+    deflist: (n) => `<dl class="mod-deflist">${repeatMarkup(n, (i) => `<dt class="mod-deflist__dt">d${i}</dt><dd class="mod-deflist__dd">v${i}</dd>`)}</dl>`,
+    gallery: (n) => `<ul class="mod-gallery">${repeatMarkup(n, (i) => `<li class="mod-gallery__item"><figure class="mod-gallery__fig"><img class="mod-gallery__img" src="g.jpg"><figcaption class="mod-gallery__cap">g${i}</figcaption></figure></li>`)}</ul>`,
+    form: (n) => `<form class="mod-form" action="#">${repeatMarkup(n, (i) => `<div class="mod-form__row"><label class="mod-form__label">l${i}</label><input class="mod-form__input"></div>`)}<button class="mod-form__submit">go</button></form>`,
+    notice: () => `<div class="mod-notice"><p class="mod-notice__text">n</p><a class="mod-notice__link" href="#">more</a></div>`,
+    timetable: (n) => `<div class="mod-timetable"><table class="mod-timetable__table"><tbody>${repeatMarkup(n, (i) => `<tr class="mod-timetable__tr"><td class="mod-timetable__time">t${i}</td><td class="mod-timetable__dest">d${i}</td></tr>`)}</tbody></table></div>`,
+    price: (n) => `<div class="mod-price"><ul class="mod-price__list">${repeatMarkup(n, (i) => `<li class="mod-price__item"><span class="mod-price__name">n${i}</span><span class="mod-price__yen">y${i}</span></li>`)}</ul></div>`,
+    banner: () => `<div class="mod-banner"><a class="mod-banner__link" href="#"><img class="mod-banner__img" src="b.jpg"></a></div>`,
+    video: () => `<div class="mod-video"><video class="mod-video__el" src="v.mp4"></video></div>`,
+    tabs: (n) => `<div class="mod-tabs"><ul class="mod-tabs__nav">${repeatMarkup(n, (i) => `<li class="mod-tabs__navitem"><button class="mod-tabs__btn">t${i}</button></li>`)}</ul><div class="mod-tabs__panel"><p class="mod-tabs__text">p</p></div></div>`,
+    download: (n) => `<ul class="mod-download">${repeatMarkup(n, (i) => `<li class="mod-download__item"><a class="mod-download__link" href="f${i}.pdf"><span class="mod-download__icon"></span><span class="mod-download__name">f${i}</span></a></li>`)}</ul>`,
+    contact: () => `<div class="mod-contact"><p class="mod-contact__tel">tel</p><p class="mod-contact__hours">h</p><a class="mod-contact__mail" href="#">m</a></div>`,
+    anchorlinks: (n) => `<nav class="mod-anchor"><ul class="mod-anchor__list">${repeatMarkup(n, (i) => `<li class="mod-anchor__item"><a class="mod-anchor__link" href="#a${i}">a${i}</a></li>`)}</ul></nav>`,
+};
+/**
+ * Templates as ordered subsets of {@link MODULE_POOL}, modeling how real CMS
+ * templates share some blocks and differ in others.
+ */
+const TEMPLATE_MODULES = {
+    faq: ['heading', 'anchorlinks', 'accordion', 'notice', 'contact'],
+    course: ['heading', 'photo', 'steps', 'map', 'deflist', 'notice'],
+    index: ['heading', 'lead', 'cards', 'banner', 'notice'],
+    fare: ['heading', 'lead', 'price', 'table', 'notice', 'contact'],
+    article: ['heading', 'lead', 'photo', 'deflist', 'download'],
+    gallery: ['heading', 'gallery', 'video', 'banner'],
+    search: ['heading', 'form', 'tabs', 'notice'],
+    access: ['heading', 'map', 'timetable', 'deflist', 'contact'],
+};
+/**
+ * @param n
+ * @param render
+ */
+function times(n, render) {
+    let out = '';
+    for (let i = 0; i < n; i++)
+        out += render(i);
+    return out;
+}
+const HEADER = `<header class="site-header"><div class="site-header__inner"><p class="site-logo"><a class="site-logo__link" href="/"><img class="site-logo__img" src="logo.svg"></a></p><nav class="gnav"><ul class="gnav__list">${times(8, (i) => `<li class="gnav__item"><a class="gnav__link" href="/s${i}/">n${i}</a></li>`)}</ul></nav><div class="site-header__util"><a class="site-header__lang" href="#">lang</a></div></div></header>`;
+const FOOTER = `<footer class="site-footer"><div class="site-footer__inner"><nav class="fnav"><ul class="fnav__list">${times(10, (i) => `<li class="fnav__item"><a class="fnav__link" href="/f${i}/">f${i}</a></li>`)}</ul></nav><p class="site-copyright">c</p></div></footer>`;
+const BREADCRUMB = `<nav class="breadcrumb"><ol class="breadcrumb__list">${times(3, (i) => `<li class="breadcrumb__item"><a class="breadcrumb__link" href="/b${i}/">b${i}</a></li>`)}</ol></nav>`;
+const PAGE_HEAD = `<div class="page-head"><div class="page-head__inner"><h1 class="page-head__ttl">t</h1><p class="page-head__lead">l</p></div></div>`;
+const RELATED = `<aside class="related"><h2 class="related__ttl">r</h2><ul class="related__list">${times(4, (i) => `<li class="related__item"><a class="related__link" href="/r${i}/">r${i}</a></li>`)}</ul></aside>`;
+/**
+ * Builds a synthetic multi-template, multi-mirror site: every template
+ * shares a site-wide shell (header/gnav/footer) and shares
+ * breadcrumb/page-head/related content blocks, and each template's own
+ * content is a distinct ordered subset of a shared CMS module pool (see
+ * {@link MODULE_POOL}) so templates partially overlap in tokens rather than
+ * being trivially disjoint. Every template is duplicated once per
+ * `axisValues` entry, each with its own per-mirror stylesheet href — this
+ * shape collapses every template into one catch-all cluster when `main` is
+ * the wrapper tag (a `main`-anchored wrapper chain degenerates
+ * `l2Signature`), while the same site with content under a plain `<div>`
+ * clusters correctly (see `wrapperTag`'s own JSDoc).
+ * @param options
+ * @example
+ * ```ts
+ * const { pages } = buildMirroredTemplateFixture({ wrapperTag: 'main' });
+ * const keys = await resolvePageClusterKeysFromArray(pages.map((p) => p.signals));
+ * // keys collapse to a single value — the bug this fixture reproduces
+ * ```
+ */
+export function buildMirroredTemplateFixture(options) {
+    const axisValues = options?.axisValues ?? ['en', 'zh', 'ko', 'th'];
+    const pagesPerTemplate = options?.pagesPerTemplate ?? 6;
+    const wrapperTag = options?.wrapperTag ?? 'main';
+    const pages = [];
+    for (const [template, moduleNames] of Object.entries(TEMPLATE_MODULES)) {
+        for (const axisValue of axisValues) {
+            for (let page = 0; page < pagesPerTemplate; page++) {
+                // Every third page drops its last module — models conditional
+                // rendering (an optional section) without changing the template.
+                const mods = page % 3 === 2 ? moduleNames.slice(0, -1) : moduleNames;
+                const body = mods
+                    .map((m) => `<div class="bg-base"><section class="sec sec--${m}"><div class="container">${MODULE_POOL[m](2 + (page % 3))}</div></section></div>`)
+                    .join('');
+                const inner = `<article>${PAGE_HEAD}${BREADCRUMB}${body}${RELATED}</article>`;
+                const wrapped = wrapperTag === 'main'
+                    ? `<main>${inner}</main>`
+                    : `<div class="main-area">${inner}</div>`;
+                pages.push({
+                    template,
+                    axisValue,
+                    page,
+                    signals: {
+                        paths: [axisValue, template, `p${page}.html`],
+                        stylesheetHrefs: [
+                            'https://example.test/css/reset.css',
+                            'https://example.test/css/style.css',
+                            `https://example.test/${axisValue}/${template}/page.css`,
+                        ],
+                        host: 'example.test',
+                        html: `<html><body>${HEADER}${wrapped}${FOOTER}</body></html>`,
+                    },
+                });
+            }
+        }
+    }
+    return { pages, templates: Object.keys(TEMPLATE_MODULES) };
+}

package/dist/cli.d.ts

@@ -6,6 +6,7 @@
 type CliArgs = {
     readonly contentBlockAttribute?: string;
     readonly clusterReasonsFile?: string;
+    readonly validationFile?: string;
     readonly help?: boolean;
     readonly version?: boolean;
     readonly unknownFlag?: string;

package/dist/cli.js

@@ -9,7 +9,7 @@ import process from 'node:process';
 import { Lanes } from '@d-zero/dealer';
 import { resolvePageClusterKeys } from './resolve-page-cluster-keys.js';
 const HELP_TEXT = `Usage:
-  page-cluster [--content-block-attribute <name>] [--cluster-reasons-file <path>] < pages.jsonl > clusters.jsonl
+  page-cluster [--content-block-attribute <name>] [--cluster-reasons-file <path>] [--validation-file <path>] < pages.jsonl > clusters.jsonl
 
 Input (JSONL, one page per line):
   {
@@ -47,6 +47,29 @@ Output (JSONL, one line per input page, in input order):
     }
   }
 
+  With --validation-file <path>, a separate JSON file is written once
+  processing completes, checking the finished partition against itself:
+  whether pages that ended up in different clusters are nonetheless
+  structurally identical or near-identical (a likely over-split), and
+  whether any single cluster's own members actually agree with each other
+  (a likely over-merge). Registering this flag also applies a built-in
+  auto-merge to the clusterKeys in the stdout output above: any pair of
+  clusters found byte-identical, or near-identical and corroborated by a
+  detected URL mirror axis (e.g. a language directory), is merged before
+  output — see ClusterPartitionReport's own JSDoc for the exact policy.
+  {
+    "mirrorAxis": { "position": 0, "values": ["en", "zh"] },
+    "cohesion": [
+      { "clusterKey": "...", "memberCount": 42, "sampledMemberCount": 40,
+        "medianPairSimilarity": 0.91, "p10PairSimilarity": 0.78,
+        "minPairSimilarity": 0.6, "suspicious": false }
+    ],
+    "crossClusterDuplicates": [
+      { "clusterKeyA": "...", "clusterKeyB": "...", "similarity": 1,
+        "corroboratedByMirrorAxis": true }
+    ]
+  }
+
 Options:
   --content-block-attribute <name>   CMS-provided attribute marking freeform
                                      content blocks that should be stripped
@@ -55,6 +78,10 @@ Options:
                                      object described above to <path> after
                                      processing completes. No page-count
                                      limit.
+  --validation-file <path>           Write the partition-validation report
+                                     described above to <path>, and apply
+                                     its built-in auto-merge to the output
+                                     clusterKeys.
   --help                             Print this help and exit.
   --version                          Print the package version and exit.
 
@@ -121,6 +148,16 @@ export function parseArgs(argv) {
                 i++;
                 break;
             }
+            case '--validation-file': {
+                const next = argv[i + 1];
+                if (next === undefined) {
+                    out.unknownFlag = `${arg} requires a value`;
+                    return out;
+                }
+                out.validationFile = next;
+                i++;
+                break;
+            }
             default: {
                 out.unknownFlag = arg;
                 return out;
@@ -299,6 +336,24 @@ function formatProgressLine(event, elapsedSec) {
         }
     }
 }
+/**
+ * `JSON.stringify` serializes a `Set` as `{}` — `ClusterPartitionReport`'s
+ * only non-JSON-safe field is `mirrorAxis.values`, so this is the one spot
+ * that needs converting before the report can be written to a file.
+ * @param report
+ */
+function toJsonSafePartitionReport(report) {
+    return {
+        mirrorAxis: report.mirrorAxis
+            ? {
+                position: report.mirrorAxis.position,
+                values: [...report.mirrorAxis.values].toSorted(),
+            }
+            : null,
+        cohesion: report.cohesion,
+        crossClusterDuplicates: report.crossClusterDuplicates,
+    };
+}
 /**
  * Test-friendly entry point: takes the run's stdin/stdout/stderr streams
  * and the parsed CLI flags rather than reading them out of the process
@@ -370,6 +425,9 @@ export async function runCli(options) {
         const reasonsByClusterKey = args.clusterReasonsFile
             ? new Map()
             : undefined;
+        // Same opt-in gate as `reasonsByClusterKey` — `onPartitionReport`
+        // itself gates the underlying validation pass (see its own JSDoc).
+        let partitionReport;
         const resolveOptions = {
             contentBlockAttribute: args.contentBlockAttribute,
             onProgress: (event) => {
@@ -378,6 +436,9 @@ export async function runCli(options) {
             onClusterReason: reasonsByClusterKey
                 ? (key, reason) => reasonsByClusterKey.set(key, reason)
                 : undefined,
+            onPartitionReport: args.validationFile
+                ? (report) => (partitionReport = report)
+                : undefined,
         };
         let clusterKeys;
         try {
@@ -402,6 +463,15 @@ export async function runCli(options) {
                 return 1;
             }
         }
+        if (args.validationFile && partitionReport) {
+            try {
+                await writeFile(args.validationFile, JSON.stringify(toJsonSafePartitionReport(partitionReport), null, 2));
+            }
+            catch (error) {
+                renderProgress(lanes, useTty, errorLine(error.message));
+                return 1;
+            }
+        }
         return 0;
     }
     finally {

package/dist/compute-cluster-cohesion.d.ts

@@ -0,0 +1,59 @@
+/**
+ * Options for {@link computeClusterCohesion}.
+ */
+export type ClusterCohesionOptions = {
+    /** @see DEFAULT_MAX_SAMPLE_SIZE */
+    readonly maxSampleSize?: number;
+    /** @see DEFAULT_SUSPICIOUS_MEDIAN_BELOW */
+    readonly suspiciousMedianBelow?: number;
+};
+/**
+ * One cluster's internal-agreement summary: how similar its members'
+ * structural tokens actually are to each other, as opposed to
+ * {@link ./merge-cross-block-clusters.js | computeQuorumCore}'s frequency
+ * core, which only reports *which* tokens are shared, not *how much* of each
+ * member's tokens that core actually covers.
+ */
+export type ClusterCohesion = {
+    readonly clusterKey: string;
+    readonly memberCount: number;
+    /** How many of `memberCount` members the pairwise comparison sampled. */
+    readonly sampledMemberCount: number;
+    /** `null` when fewer than 2 members were sampled (nothing to compare). */
+    readonly medianPairSimilarity: number | null;
+    /** 10th percentile (nearest-rank) of sampled pairwise similarities. */
+    readonly p10PairSimilarity: number | null;
+    readonly minPairSimilarity: number | null;
+    /** `medianPairSimilarity !== null && medianPairSimilarity < suspiciousMedianBelow`. */
+    readonly suspicious: boolean;
+};
+/**
+ * Reports, per cluster, how similar its members' structural token sets
+ * actually are to each other — not just which tokens they share (that's
+ * {@link ./merge-cross-block-clusters.js | computeQuorumCore}'s job), but
+ * whether the members hang together at all. A cluster built by merging
+ * unrelated templates has member pairs that mostly disagree even though a
+ * small frequency core still exists among them; this surfaces that
+ * disagreement directly, as a distribution rather than a single score, since
+ * a single "average similarity" would be pulled toward the middle by exactly
+ * the kind of partial-overlap noise this is meant to catch.
+ *
+ * Sampling uses {@link ./reservoir-sample.js | reservoirSample} seeded by
+ * each cluster's own key, so repeated calls on the same partition sample the
+ * same members and produce the same report.
+ *
+ * Deliberately returns only structured numbers, no verdict text — same
+ * design as {@link ./build-cluster-reason.js | ClusterReason} — so a caller
+ * decides what "suspicious" should mean for their own use (an interactive
+ * review queue vs. an automated gate might want different behavior for the
+ * same numbers).
+ * @param membersByKey Every cluster's member token sets, keyed by cluster key.
+ * @param options
+ * @example
+ * ```ts
+ * const report = computeClusterCohesion(membersByKey);
+ * const worstFirst = [...report].filter((r) => r.suspicious)
+ *   .toSorted((a, b) => (a.medianPairSimilarity ?? 0) - (b.medianPairSimilarity ?? 0));
+ * ```
+ */
+export declare function computeClusterCohesion(membersByKey: ReadonlyMap<string, readonly ReadonlySet<string>[]>, options?: ClusterCohesionOptions): readonly ClusterCohesion[];

package/dist/compute-cluster-cohesion.js

@@ -0,0 +1,107 @@
+import { jaccardSimilarity } from './jaccard-similarity.js';
+import { reservoirSample } from './reservoir-sample.js';
+/**
+ * Maximum members sampled per cluster before computing pairwise similarity.
+ * Bounds the cost to `O(maxSampleSize²)` per cluster regardless of how large
+ * the cluster actually is — 40² / 2 = 780 comparisons, cheap even across
+ * many clusters.
+ */
+const DEFAULT_MAX_SAMPLE_SIZE = 40;
+/**
+ * Below this median pairwise-similarity, a cluster is flagged `suspicious`.
+ *
+ * Chosen against the bundled `buildMirroredTemplateFixture` fixture (see
+ * `compute-cluster-cohesion.spec.ts`): every single-template cluster there
+ * has a median of `1.0` (structural tokens are identical across pages that
+ * differ only in text content and per-mirror stylesheet href), while every
+ * cluster built by mixing two *different* templates' members has a median
+ * at or below `0.70`. `0.75` sits with margin on both sides of that gap.
+ * Real crawl data can have single-template clusters with genuinely lower
+ * cohesion than this synthetic fixture models (legitimate content-driven
+ * structural variation the fixture doesn't produce) — `suspiciousMedianBelow`
+ * exists so a caller who has measured their own corpus can override it.
+ */
+const DEFAULT_SUSPICIOUS_MEDIAN_BELOW = 0.75;
+/**
+ * @param sortedAscending
+ */
+function median(sortedAscending) {
+    const mid = Math.floor(sortedAscending.length / 2);
+    if (sortedAscending.length % 2 === 1)
+        return sortedAscending[mid];
+    return (sortedAscending[mid - 1] + sortedAscending[mid]) / 2;
+}
+/**
+ * @param sortedAscending
+ */
+function p10(sortedAscending) {
+    const index = Math.floor(0.1 * (sortedAscending.length - 1));
+    return sortedAscending[index];
+}
+/**
+ * Reports, per cluster, how similar its members' structural token sets
+ * actually are to each other — not just which tokens they share (that's
+ * {@link ./merge-cross-block-clusters.js | computeQuorumCore}'s job), but
+ * whether the members hang together at all. A cluster built by merging
+ * unrelated templates has member pairs that mostly disagree even though a
+ * small frequency core still exists among them; this surfaces that
+ * disagreement directly, as a distribution rather than a single score, since
+ * a single "average similarity" would be pulled toward the middle by exactly
+ * the kind of partial-overlap noise this is meant to catch.
+ *
+ * Sampling uses {@link ./reservoir-sample.js | reservoirSample} seeded by
+ * each cluster's own key, so repeated calls on the same partition sample the
+ * same members and produce the same report.
+ *
+ * Deliberately returns only structured numbers, no verdict text — same
+ * design as {@link ./build-cluster-reason.js | ClusterReason} — so a caller
+ * decides what "suspicious" should mean for their own use (an interactive
+ * review queue vs. an automated gate might want different behavior for the
+ * same numbers).
+ * @param membersByKey Every cluster's member token sets, keyed by cluster key.
+ * @param options
+ * @example
+ * ```ts
+ * const report = computeClusterCohesion(membersByKey);
+ * const worstFirst = [...report].filter((r) => r.suspicious)
+ *   .toSorted((a, b) => (a.medianPairSimilarity ?? 0) - (b.medianPairSimilarity ?? 0));
+ * ```
+ */
+export function computeClusterCohesion(membersByKey, options) {
+    const maxSampleSize = options?.maxSampleSize ?? DEFAULT_MAX_SAMPLE_SIZE;
+    const suspiciousMedianBelow = options?.suspiciousMedianBelow ?? DEFAULT_SUSPICIOUS_MEDIAN_BELOW;
+    const results = [];
+    for (const [clusterKey, members] of membersByKey) {
+        const sample = reservoirSample(members, maxSampleSize, clusterKey);
+        if (sample.length < 2) {
+            results.push({
+                clusterKey,
+                memberCount: members.length,
+                sampledMemberCount: sample.length,
+                medianPairSimilarity: null,
+                p10PairSimilarity: null,
+                minPairSimilarity: null,
+                suspicious: false,
+            });
+            continue;
+        }
+        const similarities = [];
+        for (let a = 0; a < sample.length; a++) {
+            for (let b = a + 1; b < sample.length; b++) {
+                similarities.push(jaccardSimilarity(sample[a], sample[b]));
+            }
+        }
+        similarities.sort((x, y) => x - y);
+        const medianSimilarity = median(similarities);
+        results.push({
+            clusterKey,
+            memberCount: members.length,
+            sampledMemberCount: sample.length,
+            medianPairSimilarity: medianSimilarity,
+            p10PairSimilarity: p10(similarities),
+            minPairSimilarity: similarities[0],
+            suspicious: medianSimilarity < suspiciousMedianBelow,
+        });
+    }
+    return results;
+}

package/dist/detect-mirror-axis.d.ts

@@ -0,0 +1,74 @@
+/**
+ * A URL path segment position at which a whole site section is mirrored
+ * under a fixed set of alternative values — most commonly a language
+ * directory (`/en/`, `/zh/`, `/ko/`, `/th/`), but the same shape also arises
+ * from staging/production mirrors, device-variant subsites (`/sp/`), or
+ * versioned documentation trees. Detected by {@link detectMirrorAxis} from
+ * `paths` alone, with no built-in list of language codes or other
+ * site-specific vocabulary.
+ */
+export type MirrorAxis = {
+    /** Index into a page's `paths` array where the mirrored value sits. */
+    readonly position: number;
+    /** The alternative values observed at `position` across the mirror. */
+    readonly values: ReadonlySet<string>;
+};
+/**
+ * Options for {@link detectMirrorAxis}.
+ */
+export type DetectMirrorAxisOptions = {
+    /**
+     * How many leading path segments to scan for a mirror axis. Bounds the
+     * cost to `O(maxPosition × pageCount)`; a mirror axis deep enough to need
+     * a larger value is unusual (language/environment/version directories
+     * are conventionally near the root).
+     */
+    readonly maxPosition?: number;
+    /**
+     * Minimum number of distinct path skeletons that must share the exact
+     * same value set at a position before it is even considered a candidate
+     * axis. A single skeleton with multiple values at some position (e.g. a
+     * `/faq/{01,02,03}` set of sibling pages) is not a mirror — nothing about
+     * it repeats — so admitting `skeletonCount === 1` candidates would
+     * misidentify ordinary sibling pages as a site-wide axis. 3 is the
+     * smallest count for which "this exact value set recurs" stops being
+     * describable as coincidence.
+     */
+    readonly minSkeletonCount?: number;
+};
+/**
+ * Finds a mirror axis in a page corpus's URL paths, with no built-in
+ * knowledge of language codes or any other site-specific vocabulary — the
+ * axis is inferred purely from how often the exact same set of alternative
+ * values recurs across otherwise-identical path skeletons.
+ *
+ * For each scanned segment position, every page's path is reduced to a
+ * "skeleton" (that position blanked to `*`); skeletons that recur with
+ * `≥ 2` distinct values at that position are candidates, keyed by their
+ * exact value set. Candidates are ranked by how many distinct skeletons
+ * share that value set — the more independent path shapes that recur under
+ * the same alternative values, the more likely that position is a genuine
+ * site-wide mirror rather than a coincidence — and the top-ranked position's
+ * value set is accepted unless {@link autoCutThreshold}'s max-gap cut (using
+ * the top candidate's own count as `upperBound`, so the cut never selects a
+ * value the top candidate doesn't already clear) finds the next-best
+ * candidate too close behind to call decisively.
+ *
+ * Returns `null` when no position has any candidate clearing
+ * `minSkeletonCount` — including a single-language/single-mirror corpus
+ * (nothing recurs under alternative values at all) and a corpus whose only
+ * repeated-value-set position is a single skeleton's sibling pages (e.g.
+ * `/faq/{01,02,03}` — excluded by `minSkeletonCount`, see its own JSDoc).
+ * @param pagePaths Every page's URL path segments (e.g. `ExURL.paths`).
+ * @param options
+ * @example
+ * ```ts
+ * const axis = detectMirrorAxis([
+ *   ['en', 'faq', 'index.html'], ['zh', 'faq', 'index.html'],
+ *   ['en', 'access', 'index.html'], ['zh', 'access', 'index.html'],
+ *   ['en', 'gallery', 'index.html'], ['zh', 'gallery', 'index.html'],
+ * ]);
+ * // { position: 0, values: Set(['en', 'zh']) }
+ * ```
+ */
+export declare function detectMirrorAxis(pagePaths: readonly (readonly string[])[], options?: DetectMirrorAxisOptions): MirrorAxis | null;