@d-zero/page-cluster 0.5.6 → 0.6.0

package/README.md

@@ -79,6 +79,7 @@ jq -c '.[]' crawl-output.json | page-cluster > clusters.jsonl
 
 - `--content-block-attribute <name>` — CMS が自由編集コンテンツブロックに付与している属性名（例: `data-bgb`）が分かっている場合に指定する。指定すると比較前にその属性を持つ要素配下を無視するので、同じテンプレートで本文構成だけ違うページを混同しなくなる。唯一の site-specific なオプションで、未指定でも `<main>` / `role="main"` を起点にした自動深さキャップが常時働く（詳細は `resolve-page-cluster-keys.ts` の JSDoc を参照）
 - `--cluster-reasons-file <path>` — 上記の「クラスタ選定理由」を `<path>` に JSON として書き出す。ページ数の上限はない。20,000 ページ以下のコーパスでは、指定すると進捗表示（後述）は出なくなる（進捗を出さない非ストリーミング経路に常に振り分けられるため。20,000 ページ超のストリーミング経路では進捗表示・クラスタ理由の両方が動く）
+- `--validation-file <path>` — 分類結果の事後検証レポート（下記「分類結果の事後検証」参照）を `<path>` に JSON として書き出す。指定すると、レポートの検出結果のうち安全に統合できるもの（構造トークン集合が完全一致、またはミラー軸による裏付けあり）が **stdout の `clusterKey` にも自動的に反映される**
 - `--help` / `-h` — ヘルプを表示する
 - `--version` / `-v` — バージョンを表示する
 
@@ -119,6 +120,13 @@ silence したい場合は `2>/dev/null`。ログに残したい場合は `2> pr
 | `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類                                                                                                                                                                                                                   |
 | `@d-zero/page-cluster/is-chrome-landmark-instance`   | `isChromeLandmarkInstance` — 1 つの landmark インスタンスのトークン集合と `ClusterReason.landmarks[type].shellTokens` のようなシェルトークン集合を突き合わせて chrome/content を判定するステートレス関数                                                                                            |
 | `@d-zero/page-cluster/jaccard-similarity`            | `jaccardSimilarity` — 2 つのトークン集合の Jaccard 類似度。`ClusterReason` 同士（`structuralCoreTokens` や `shellTokens`）を比較して兄弟クラスタとの差分を調べる用途などに使う                                                                                                                      |
+| `@d-zero/page-cluster/validate-cluster-partition`    | `ClusterPartitionReport` 型、`validateClusterPartition` — 分類済みの `clusterKey` を、分類そのものではなく分類結果同士を突き合わせて事後検証する（下記「分類結果の事後検証」参照）。通常は `resolvePageClusterKeys` の `onPartitionReport` 経由で使うので直接呼ぶ必要はない                         |
+| `@d-zero/page-cluster/find-cross-cluster-duplicates` | `ClusteredPage` 型、`CrossClusterDuplicate` 型、`findCrossClusterDuplicates` — 別クラスタに分かれているが同一テンプレートらしいページ対を検出する                                                                                                                                                   |
+| `@d-zero/page-cluster/compute-cluster-cohesion`      | `ClusterCohesion` 型、`computeClusterCohesion` — クラスタ内メンバーが互いにどれだけ似ているかを分布として報告する（無関係なテンプレートが混ざったクラスタの検出に使う）                                                                                                                             |
+| `@d-zero/page-cluster/detect-mirror-axis`            | `MirrorAxis` 型、`detectMirrorAxis` — 言語ディレクトリなど、URL のあるセグメントだけを変えてサイトの一部をミラーしている「軸」を、事前知識なしに URL パス集合から発見する                                                                                                                           |
+| `@d-zero/page-cluster/normalize-path-by-mirror-axis` | `normalizePathByMirrorAxis` — 検出した軸に沿ってページの URL パスを正規化し、ミラー元同士を同じ形状に揃える                                                                                                                                                                                         |
+| `@d-zero/page-cluster/normalize-href-by-mirror-axis` | `normalizeHrefByMirrorAxis` — 検出した軸に沿って URL（主に stylesheet href）を正規化する                                                                                                                                                                                                            |
+| `@d-zero/page-cluster/merge-validated-clusters`      | `mergeValidatedClusters` — 確認済みのクラスタ対の統合を `clusterKey` 配列に適用する純関数                                                                                                                                                                                                           |
 
 ```ts
 import { resolvePageClusterKeysFromArray } from '@d-zero/page-cluster/resolve-page-cluster-keys';
@@ -192,6 +200,33 @@ flowchart TD
 
 マージが起きるとユニットのメンバー構成が変わり、文書頻度も quorum core も変わる。そのため毎ラウンド、統合後のプールから全指標を**再計算**してマージを再試行する。fine stage・L2 stage の両方でマージが 1 件も出なくなった時点で不動点に到達したとみなして収束する（安全弁として最大 10 ラウンド。実データでは 7 ラウンド以内に収束）。L2 stage は fine stage が空振りしたラウンドでしか実行されない最後の粗い経路で、誤マージ防止のために shell（ランドマーク由来トークン）の相互裏付けを要求する。
 
+fine stage・L2 stage いずれの経路で提案されたマージも、適用前に**凝集度ガード**を通る: 統合後の quorum core が統合前の core に対して一定比率を下回るなら、そのマージは破棄される。個々の経路が「統合前のペア類似度」だけを見て提案する一方、複数ラウンドにわたる連鎖的な統合は「統合後に実際どれだけまとまっているか」を悪化させ得る（無関係なテンプレート同士が少しずつ吸収し合う catch-all 化）ため、経路をまたいだ単一のチェックポイントとして機能する。あわせて、L2 stage 自体もラウンド開始前に判別力を検査し、参加ユニット全体が同一の署名形状に潰れている（`main` 直下の浅い階層しか手がかりが残っていない等）場合は、そのラウンドの L2 比較を丸ごとスキップする。詳細は `merge-cross-block-clusters.ts` の `filterMergesByCohesion` / `hasDiscriminatingL2Signatures` の JSDoc を参照。
+
 ### Self-tuning
 
 閾値の多くは **max-gap auto-cut**（度数分布の隣接ギャップ最大の中点を境界とする）でデータから自己発見される。① Stage A のカット高、② Stage B の shell 判定、③ chrome discovery のグローバル/ローカル判定、④ Pass 0 の URL パス深さ選択、の 4 箇所で同一プリミティブを再利用しているので、サイトごとにハイパーパラメータをチューニングする必要はない。詳細は `autoCutThreshold` の JSDoc を参照。例外的に Stage B fine stage の complete-linkage だけは固定閾値 0.8 を使う（理由は `merge-cross-block-clusters.ts` の JSDoc を参照）。
+
+## 分類結果の事後検証
+
+Stage A/B は「クラスタリング中に」正しい判断をしようとするが、判断材料はその時点でのペア類似度に限られる。分類が終わった**あとで**、確定したパーティション同士を突き合わせて検証する方が、同じ種類の誤りをかえって見つけやすい場合がある: マージ中は局所的なペア情報とマージ順序しか持たないが、事後なら分割全体を一度に見られるからだ。この事後検証は Stage A/B のアルゴリズムに一切依存しないので、`resolvePageClusterKeys` 以外で作られた分類結果（保存済みの `clusterKey` をアーカイブから読み戻した場合など）にも使える。
+
+3 つのチェックを行う。
+
+- **クラスタ間重複検出**（`findCrossClusterDuplicates`） — 別クラスタに分かれているページ対のうち、構造トークン集合が完全一致するもの（軸の裏付けなしで確定）と、ミラー軸で裏付けられる近似一致のものを検出する
+- **クラスタ内凝集度**（`computeClusterCohesion`） — クラスタ内のメンバー同士がどれだけ似ているかを中央値・10 パーセンタイル・最小値の分布として報告する。無関係なテンプレートが混ざったクラスタは、共通のシェル由来トークンだけが一致する形で `structuralCoreTokens` 自体は非空のまま残ることがあるため、core の有無だけでは過剰マージを検出できない
+- **ミラー軸の自動発見**（`detectMirrorAxis`） — 言語ディレクトリのように、URL のあるセグメントだけを変えてサイトの一部をミラーしている構造を、言語コード等の事前知識なしに発見する。同じ値集合が何種類の異なるパス骨格にわたって反復するかを数え、`autoCutThreshold` で「たまたま値が 2 つ以上あるだけの兄弟ページ」と「サイト全体を貫く軸」を切り分ける
+
+`validateClusterPartition` はこの 3 つをまとめて呼び出すエントリポイント。`ClusterReason` と同じく、判断結果ではなく構造化データだけを返す — 検出された重複をどう扱うか（統合するかどうか、`suspicious` フラグをどう解釈するか）は呼び出し側の責務。実際に統合を適用する場合は `mergeValidatedClusters` に確認済みのクラスタ対を渡す。
+
+```ts
+import { mergeValidatedClusters } from '@d-zero/page-cluster/merge-validated-clusters';
+import { validateClusterPartition } from '@d-zero/page-cluster/validate-cluster-partition';
+
+const report = validateClusterPartition(pages); // pages: { clusterKey, tokens, paths, stylesheetHrefs }[]
+const safeToMerge = report.crossClusterDuplicates.filter(
+	(d) => d.similarity === 1 || d.corroboratedByMirrorAxis,
+);
+const mergedKeys = mergeValidatedClusters(clusterKeys, safeToMerge);
+```
+
+`resolvePageClusterKeys` に `onPartitionReport` コールバックを渡すと、この検証が Stage B 完了直後に自動的に走り、上記と同じ安全な統合ポリシー（完全一致、またはミラー軸による裏付けあり）が返り値の `clusterKey` にもそのまま反映される。`onClusterReason` と同じくオプトインで、渡さない限り既存の挙動・出力は一切変わらない。詳細はその JSDoc を参照。

package/dist/build-mirrored-template-fixture.d.ts

@@ -0,0 +1,61 @@
+import type { PageClusterSignals } from './resolve-page-cluster-keys.js';
+/**
+ * Options for {@link buildMirroredTemplateFixture}.
+ */
+export type MirroredTemplateFixtureOptions = {
+    /**
+     * URL segment values a whole section is mirrored under (e.g. language
+     * codes). Each value gets its own copy of every template, at its own
+     * per-value stylesheet href — modeling a site that duplicates a section's
+     * CSS per mirror instead of sharing one file.
+     */
+    readonly axisValues?: readonly string[];
+    /** Pages generated per (template, axis value) pair. */
+    readonly pagesPerTemplate?: number;
+    /**
+     * Tag wrapping each page's content. `'main'` reproduces the L2-stage
+     * signature collapse this fixture exists to catch (see
+     * `merge-cross-block-clusters.ts`'s `l2Signature` JSDoc): every template's
+     * content sits under a `main > article > div > section > div` chain, so
+     * `l2Signature`'s class-stripped, 2-level-past-`main` truncation reduces
+     * every template to the same key set. `'div'` (or any other tag) opts out
+     * of `l2Signature`'s `main`-anchoring, so the fine stage alone must
+     * separate the templates correctly.
+     */
+    readonly wrapperTag?: 'main' | 'div';
+};
+/**
+ * One generated page's ground truth, alongside the `PageClusterSignals` a
+ * clustering call actually sees. Tests assert against `template`/`axisValue`
+ * rather than parsing them back out of a `clusterKey` string.
+ */
+export type MirroredTemplateFixturePage = {
+    readonly signals: PageClusterSignals;
+    readonly template: string;
+    readonly axisValue: string;
+    readonly page: number;
+};
+/**
+ * Builds a synthetic multi-template, multi-mirror site: every template
+ * shares a site-wide shell (header/gnav/footer) and shares
+ * breadcrumb/page-head/related content blocks, and each template's own
+ * content is a distinct ordered subset of a shared CMS module pool (see
+ * {@link MODULE_POOL}) so templates partially overlap in tokens rather than
+ * being trivially disjoint. Every template is duplicated once per
+ * `axisValues` entry, each with its own per-mirror stylesheet href — this
+ * shape collapses every template into one catch-all cluster when `main` is
+ * the wrapper tag (a `main`-anchored wrapper chain degenerates
+ * `l2Signature`), while the same site with content under a plain `<div>`
+ * clusters correctly (see `wrapperTag`'s own JSDoc).
+ * @param options
+ * @example
+ * ```ts
+ * const { pages } = buildMirroredTemplateFixture({ wrapperTag: 'main' });
+ * const keys = await resolvePageClusterKeysFromArray(pages.map((p) => p.signals));
+ * // keys collapse to a single value — the bug this fixture reproduces
+ * ```
+ */
+export declare function buildMirroredTemplateFixture(options?: MirroredTemplateFixtureOptions): {
+    readonly pages: readonly MirroredTemplateFixturePage[];
+    readonly templates: readonly string[];
+};

package/dist/build-mirrored-template-fixture.js

@@ -0,0 +1,127 @@
+/**
+ * @param count
+ * @param render
+ */
+function repeatMarkup(count, render) {
+    let out = '';
+    for (let i = 0; i < count; i++)
+        out += render(i);
+    return out;
+}
+/**
+ * Pool of CMS-style content blocks that {@link TEMPLATE_MODULES} composes
+ * templates from. Modeled after real CMS "block" systems: templates share
+ * modules with each other (a FAQ page and a contact page both use
+ * `notice`/`contact`), which is what makes this fixture exercise Stage B's
+ * cross-block containment/shape logic instead of trivially disjoint token
+ * sets.
+ */
+const MODULE_POOL = {
+    lead: () => `<div class="mod-lead"><p class="mod-lead__text">x</p><p class="mod-lead__note">n</p></div>`,
+    heading: () => `<div class="mod-heading"><h2 class="mod-heading__ttl">h</h2><p class="mod-heading__sub">s</p></div>`,
+    photo: () => `<figure class="mod-photo"><img class="mod-photo__img" src="p.jpg"><figcaption class="mod-photo__cap">c</figcaption></figure>`,
+    table: (n) => `<div class="mod-table"><table class="mod-table__table"><thead><tr><th class="mod-table__th">h</th></tr></thead><tbody>${repeatMarkup(n, (i) => `<tr class="mod-table__tr"><td class="mod-table__td">v${i}</td></tr>`)}</tbody></table></div>`,
+    accordion: (n) => `<div class="mod-accordion">${repeatMarkup(n, (i) => `<dl class="mod-accordion__item"><dt class="mod-accordion__q"><button class="mod-accordion__toggle">q${i}</button></dt><dd class="mod-accordion__a"><p class="mod-accordion__text">a${i}</p></dd></dl>`)}</div>`,
+    cards: (n) => `<ul class="mod-cards">${repeatMarkup(n, (i) => `<li class="mod-cards__item"><a class="mod-cards__link" href="#${i}"><span class="mod-cards__fig"><img class="mod-cards__img" src="c.jpg"></span><span class="mod-cards__ttl">c${i}</span></a></li>`)}</ul>`,
+    steps: (n) => `<ol class="mod-steps">${repeatMarkup(n, (i) => `<li class="mod-steps__item"><span class="mod-steps__num">${i}</span><h3 class="mod-steps__ttl">s${i}</h3><p class="mod-steps__text">d${i}</p></li>`)}</ol>`,
+    map: () => `<div class="mod-map"><iframe class="mod-map__frame" src="map"></iframe><p class="mod-map__note">m</p></div>`,
+    deflist: (n) => `<dl class="mod-deflist">${repeatMarkup(n, (i) => `<dt class="mod-deflist__dt">d${i}</dt><dd class="mod-deflist__dd">v${i}</dd>`)}</dl>`,
+    gallery: (n) => `<ul class="mod-gallery">${repeatMarkup(n, (i) => `<li class="mod-gallery__item"><figure class="mod-gallery__fig"><img class="mod-gallery__img" src="g.jpg"><figcaption class="mod-gallery__cap">g${i}</figcaption></figure></li>`)}</ul>`,
+    form: (n) => `<form class="mod-form" action="#">${repeatMarkup(n, (i) => `<div class="mod-form__row"><label class="mod-form__label">l${i}</label><input class="mod-form__input"></div>`)}<button class="mod-form__submit">go</button></form>`,
+    notice: () => `<div class="mod-notice"><p class="mod-notice__text">n</p><a class="mod-notice__link" href="#">more</a></div>`,
+    timetable: (n) => `<div class="mod-timetable"><table class="mod-timetable__table"><tbody>${repeatMarkup(n, (i) => `<tr class="mod-timetable__tr"><td class="mod-timetable__time">t${i}</td><td class="mod-timetable__dest">d${i}</td></tr>`)}</tbody></table></div>`,
+    price: (n) => `<div class="mod-price"><ul class="mod-price__list">${repeatMarkup(n, (i) => `<li class="mod-price__item"><span class="mod-price__name">n${i}</span><span class="mod-price__yen">y${i}</span></li>`)}</ul></div>`,
+    banner: () => `<div class="mod-banner"><a class="mod-banner__link" href="#"><img class="mod-banner__img" src="b.jpg"></a></div>`,
+    video: () => `<div class="mod-video"><video class="mod-video__el" src="v.mp4"></video></div>`,
+    tabs: (n) => `<div class="mod-tabs"><ul class="mod-tabs__nav">${repeatMarkup(n, (i) => `<li class="mod-tabs__navitem"><button class="mod-tabs__btn">t${i}</button></li>`)}</ul><div class="mod-tabs__panel"><p class="mod-tabs__text">p</p></div></div>`,
+    download: (n) => `<ul class="mod-download">${repeatMarkup(n, (i) => `<li class="mod-download__item"><a class="mod-download__link" href="f${i}.pdf"><span class="mod-download__icon"></span><span class="mod-download__name">f${i}</span></a></li>`)}</ul>`,
+    contact: () => `<div class="mod-contact"><p class="mod-contact__tel">tel</p><p class="mod-contact__hours">h</p><a class="mod-contact__mail" href="#">m</a></div>`,
+    anchorlinks: (n) => `<nav class="mod-anchor"><ul class="mod-anchor__list">${repeatMarkup(n, (i) => `<li class="mod-anchor__item"><a class="mod-anchor__link" href="#a${i}">a${i}</a></li>`)}</ul></nav>`,
+};
+/**
+ * Templates as ordered subsets of {@link MODULE_POOL}, modeling how real CMS
+ * templates share some blocks and differ in others.
+ */
+const TEMPLATE_MODULES = {
+    faq: ['heading', 'anchorlinks', 'accordion', 'notice', 'contact'],
+    course: ['heading', 'photo', 'steps', 'map', 'deflist', 'notice'],
+    index: ['heading', 'lead', 'cards', 'banner', 'notice'],
+    fare: ['heading', 'lead', 'price', 'table', 'notice', 'contact'],
+    article: ['heading', 'lead', 'photo', 'deflist', 'download'],
+    gallery: ['heading', 'gallery', 'video', 'banner'],
+    search: ['heading', 'form', 'tabs', 'notice'],
+    access: ['heading', 'map', 'timetable', 'deflist', 'contact'],
+};
+/**
+ * @param n
+ * @param render
+ */
+function times(n, render) {
+    let out = '';
+    for (let i = 0; i < n; i++)
+        out += render(i);
+    return out;
+}
+const HEADER = `<header class="site-header"><div class="site-header__inner"><p class="site-logo"><a class="site-logo__link" href="/"><img class="site-logo__img" src="logo.svg"></a></p><nav class="gnav"><ul class="gnav__list">${times(8, (i) => `<li class="gnav__item"><a class="gnav__link" href="/s${i}/">n${i}</a></li>`)}</ul></nav><div class="site-header__util"><a class="site-header__lang" href="#">lang</a></div></div></header>`;
+const FOOTER = `<footer class="site-footer"><div class="site-footer__inner"><nav class="fnav"><ul class="fnav__list">${times(10, (i) => `<li class="fnav__item"><a class="fnav__link" href="/f${i}/">f${i}</a></li>`)}</ul></nav><p class="site-copyright">c</p></div></footer>`;
+const BREADCRUMB = `<nav class="breadcrumb"><ol class="breadcrumb__list">${times(3, (i) => `<li class="breadcrumb__item"><a class="breadcrumb__link" href="/b${i}/">b${i}</a></li>`)}</ol></nav>`;
+const PAGE_HEAD = `<div class="page-head"><div class="page-head__inner"><h1 class="page-head__ttl">t</h1><p class="page-head__lead">l</p></div></div>`;
+const RELATED = `<aside class="related"><h2 class="related__ttl">r</h2><ul class="related__list">${times(4, (i) => `<li class="related__item"><a class="related__link" href="/r${i}/">r${i}</a></li>`)}</ul></aside>`;
+/**
+ * Builds a synthetic multi-template, multi-mirror site: every template
+ * shares a site-wide shell (header/gnav/footer) and shares
+ * breadcrumb/page-head/related content blocks, and each template's own
+ * content is a distinct ordered subset of a shared CMS module pool (see
+ * {@link MODULE_POOL}) so templates partially overlap in tokens rather than
+ * being trivially disjoint. Every template is duplicated once per
+ * `axisValues` entry, each with its own per-mirror stylesheet href — this
+ * shape collapses every template into one catch-all cluster when `main` is
+ * the wrapper tag (a `main`-anchored wrapper chain degenerates
+ * `l2Signature`), while the same site with content under a plain `<div>`
+ * clusters correctly (see `wrapperTag`'s own JSDoc).
+ * @param options
+ * @example
+ * ```ts
+ * const { pages } = buildMirroredTemplateFixture({ wrapperTag: 'main' });
+ * const keys = await resolvePageClusterKeysFromArray(pages.map((p) => p.signals));
+ * // keys collapse to a single value — the bug this fixture reproduces
+ * ```
+ */
+export function buildMirroredTemplateFixture(options) {
+    const axisValues = options?.axisValues ?? ['en', 'zh', 'ko', 'th'];
+    const pagesPerTemplate = options?.pagesPerTemplate ?? 6;
+    const wrapperTag = options?.wrapperTag ?? 'main';
+    const pages = [];
+    for (const [template, moduleNames] of Object.entries(TEMPLATE_MODULES)) {
+        for (const axisValue of axisValues) {
+            for (let page = 0; page < pagesPerTemplate; page++) {
+                // Every third page drops its last module — models conditional
+                // rendering (an optional section) without changing the template.
+                const mods = page % 3 === 2 ? moduleNames.slice(0, -1) : moduleNames;
+                const body = mods
+                    .map((m) => `<div class="bg-base"><section class="sec sec--${m}"><div class="container">${MODULE_POOL[m](2 + (page % 3))}</div></section></div>`)
+                    .join('');
+                const inner = `<article>${PAGE_HEAD}${BREADCRUMB}${body}${RELATED}</article>`;
+                const wrapped = wrapperTag === 'main'
+                    ? `<main>${inner}</main>`
+                    : `<div class="main-area">${inner}</div>`;
+                pages.push({
+                    template,
+                    axisValue,
+                    page,
+                    signals: {
+                        paths: [axisValue, template, `p${page}.html`],
+                        stylesheetHrefs: [
+                            'https://example.test/css/reset.css',
+                            'https://example.test/css/style.css',
+                            `https://example.test/${axisValue}/${template}/page.css`,
+                        ],
+                        host: 'example.test',
+                        html: `<html><body>${HEADER}${wrapped}${FOOTER}</body></html>`,
+                    },
+                });
+            }
+        }
+    }
+    return { pages, templates: Object.keys(TEMPLATE_MODULES) };
+}

package/dist/cli.d.ts

@@ -6,6 +6,7 @@
 type CliArgs = {
     readonly contentBlockAttribute?: string;
     readonly clusterReasonsFile?: string;
+    readonly validationFile?: string;
     readonly help?: boolean;
     readonly version?: boolean;
     readonly unknownFlag?: string;

package/dist/cli.js

@@ -6,10 +6,11 @@
 // animated header on a TTY, appended `[page-cluster] …` lines otherwise.
 import { writeFile } from 'node:fs/promises';
 import process from 'node:process';
+import { unwrapSuppressedError } from '@d-zero/cli-core';
 import { Lanes } from '@d-zero/dealer';
 import { resolvePageClusterKeys } from './resolve-page-cluster-keys.js';
 const HELP_TEXT = `Usage:
-  page-cluster [--content-block-attribute <name>] [--cluster-reasons-file <path>] < pages.jsonl > clusters.jsonl
+  page-cluster [--content-block-attribute <name>] [--cluster-reasons-file <path>] [--validation-file <path>] < pages.jsonl > clusters.jsonl
 
 Input (JSONL, one page per line):
   {
@@ -47,6 +48,29 @@ Output (JSONL, one line per input page, in input order):
     }
   }
 
+  With --validation-file <path>, a separate JSON file is written once
+  processing completes, checking the finished partition against itself:
+  whether pages that ended up in different clusters are nonetheless
+  structurally identical or near-identical (a likely over-split), and
+  whether any single cluster's own members actually agree with each other
+  (a likely over-merge). Registering this flag also applies a built-in
+  auto-merge to the clusterKeys in the stdout output above: any pair of
+  clusters found byte-identical, or near-identical and corroborated by a
+  detected URL mirror axis (e.g. a language directory), is merged before
+  output — see ClusterPartitionReport's own JSDoc for the exact policy.
+  {
+    "mirrorAxis": { "position": 0, "values": ["en", "zh"] },
+    "cohesion": [
+      { "clusterKey": "...", "memberCount": 42, "sampledMemberCount": 40,
+        "medianPairSimilarity": 0.91, "p10PairSimilarity": 0.78,
+        "minPairSimilarity": 0.6, "suspicious": false }
+    ],
+    "crossClusterDuplicates": [
+      { "clusterKeyA": "...", "clusterKeyB": "...", "similarity": 1,
+        "corroboratedByMirrorAxis": true }
+    ]
+  }
+
 Options:
   --content-block-attribute <name>   CMS-provided attribute marking freeform
                                      content blocks that should be stripped
@@ -55,6 +79,10 @@ Options:
                                      object described above to <path> after
                                      processing completes. No page-count
                                      limit.
+  --validation-file <path>           Write the partition-validation report
+                                     described above to <path>, and apply
+                                     its built-in auto-merge to the output
+                                     clusterKeys.
   --help                             Print this help and exit.
   --version                          Print the package version and exit.
 
@@ -121,6 +149,16 @@ export function parseArgs(argv) {
                 i++;
                 break;
             }
+            case '--validation-file': {
+                const next = argv[i + 1];
+                if (next === undefined) {
+                    out.unknownFlag = `${arg} requires a value`;
+                    return out;
+                }
+                out.validationFile = next;
+                i++;
+                break;
+            }
             default: {
                 out.unknownFlag = arg;
                 return out;
@@ -261,6 +299,20 @@ function errorLine(message) {
         verbose: `error: ${message}`,
     };
 }
+/**
+ * Formats a caught error for `errorLine()`. `SuppressedError` (thrown when a
+ * `using`-scoped body error and a disposal error occur together) hides the
+ * real cause behind a generic message, so its underlying causes are
+ * unwrapped and joined into one line — `errorLine()`/`renderProgress()` must
+ * still be called exactly once per catch site, since Lanes' TTY repaint only
+ * keeps the latest frame (see {@link errorLine}'s JSDoc).
+ * @param error
+ */
+function formatErrorMessage(error) {
+    return unwrapSuppressedError(error)
+        .map((cause) => (cause instanceof Error ? cause.message : String(cause)))
+        .join(' / ');
+}
 /**
  * Maps a library `ProgressEvent` to a human-facing `ProgressLine`. The
  * verbose arm keeps the historical `pass0:` / `pass1:` / `pass1b:` /
@@ -299,6 +351,24 @@ function formatProgressLine(event, elapsedSec) {
         }
     }
 }
+/**
+ * `JSON.stringify` serializes a `Set` as `{}` — `ClusterPartitionReport`'s
+ * only non-JSON-safe field is `mirrorAxis.values`, so this is the one spot
+ * that needs converting before the report can be written to a file.
+ * @param report
+ */
+function toJsonSafePartitionReport(report) {
+    return {
+        mirrorAxis: report.mirrorAxis
+            ? {
+                position: report.mirrorAxis.position,
+                values: [...report.mirrorAxis.values].toSorted(),
+            }
+            : null,
+        cohesion: report.cohesion,
+        crossClusterDuplicates: report.crossClusterDuplicates,
+    };
+}
 /**
  * Test-friendly entry point: takes the run's stdin/stdout/stderr streams
  * and the parsed CLI flags rather than reading them out of the process
@@ -331,7 +401,10 @@ export async function runCli(options) {
     // pass in) does — reading it defensively lets both real usage and
     // unit-test doubles work without a separate `--no-progress` flag.
     const useTty = options.stderr.isTTY === true;
-    const lanes = new Lanes({ stream: options.stderr, verbose: !useTty });
+    // `using` により、この関数を抜けるすべての経路（下記の各 `return` は
+    // もちろん、想定外の例外を含む）で確実に lanes.close() が呼ばれ、
+    // Display の setTimeout タイマーが解放される。
+    using lanes = new Lanes({ stream: options.stderr, verbose: !useTty });
     // Verbose Lanes prepends `#header` to every `update()` line. Without
     // this seed call the header would be undefined and each progress line
     // would begin with the literal string `undefined ` — bug caught by
@@ -342,71 +415,77 @@ export async function runCli(options) {
     }
     const startTime = Date.now();
     const elapsed = () => Math.max(0, Math.round((Date.now() - startTime) / 1000));
-    // Every early return past this point must run through the finally block
-    // so `lanes.close()` releases the display's setTimeout timer — without
-    // it a `return 1` on a stdin parse error would leave the process
-    // hanging on the timer's next tick.
+    renderProgress(lanes, useTty, READING_INPUT);
+    // Load every JSONL line into memory once so the ids array stays
+    // parallel to the pages array — the streaming driver reads its
+    // factory twice, and stdin is a one-shot pipe.
+    const ids = [];
+    const pages = [];
     try {
-        renderProgress(lanes, useTty, READING_INPUT);
-        // Load every JSONL line into memory once so the ids array stays
-        // parallel to the pages array — the streaming driver reads its
-        // factory twice, and stdin is a one-shot pipe.
-        const ids = [];
-        const pages = [];
+        for await (const { id, page } of readJsonlPages(options.stdin)) {
+            ids.push(id);
+            pages.push(page);
+        }
+    }
+    catch (error) {
+        renderProgress(lanes, useTty, errorLine(formatErrorMessage(error)));
+        return 1;
+    }
+    renderProgress(lanes, useTty, readingDoneLine(pages.length));
+    // Only worth collecting when the caller asked for the file — a
+    // ClusterReason Map costs bookkeeping proportional to cluster count,
+    // not page count, but there's no reason to pay even that when unused.
+    const reasonsByClusterKey = args.clusterReasonsFile
+        ? new Map()
+        : undefined;
+    // Same opt-in gate as `reasonsByClusterKey` — `onPartitionReport`
+    // itself gates the underlying validation pass (see its own JSDoc).
+    let partitionReport;
+    const resolveOptions = {
+        contentBlockAttribute: args.contentBlockAttribute,
+        onProgress: (event) => {
+            renderProgress(lanes, useTty, formatProgressLine(event, elapsed()));
+        },
+        onClusterReason: reasonsByClusterKey
+            ? (key, reason) => reasonsByClusterKey.set(key, reason)
+            : undefined,
+        onPartitionReport: args.validationFile
+            ? (report) => (partitionReport = report)
+            : undefined,
+    };
+    let clusterKeys;
+    try {
+        clusterKeys = await resolvePageClusterKeys(() => pages, resolveOptions);
+    }
+    catch (error) {
+        renderProgress(lanes, useTty, errorLine(formatErrorMessage(error)));
+        return 1;
+    }
+    const clusterCount = new Set(clusterKeys).size;
+    renderProgress(lanes, useTty, doneLine(pages.length, clusterCount, elapsed()));
+    for (const [index, key] of clusterKeys.entries()) {
+        const row = { id: ids[index] ?? index, clusterKey: key };
+        options.stdout.write(`${JSON.stringify(row)}\n`);
+    }
+    if (args.clusterReasonsFile && reasonsByClusterKey) {
         try {
-            for await (const { id, page } of readJsonlPages(options.stdin)) {
-                ids.push(id);
-                pages.push(page);
-            }
+            await writeFile(args.clusterReasonsFile, JSON.stringify(Object.fromEntries(reasonsByClusterKey), null, 2));
         }
         catch (error) {
-            renderProgress(lanes, useTty, errorLine(error.message));
+            renderProgress(lanes, useTty, errorLine(formatErrorMessage(error)));
             return 1;
         }
-        renderProgress(lanes, useTty, readingDoneLine(pages.length));
-        // Only worth collecting when the caller asked for the file — a
-        // ClusterReason Map costs bookkeeping proportional to cluster count,
-        // not page count, but there's no reason to pay even that when unused.
-        const reasonsByClusterKey = args.clusterReasonsFile
-            ? new Map()
-            : undefined;
-        const resolveOptions = {
-            contentBlockAttribute: args.contentBlockAttribute,
-            onProgress: (event) => {
-                renderProgress(lanes, useTty, formatProgressLine(event, elapsed()));
-            },
-            onClusterReason: reasonsByClusterKey
-                ? (key, reason) => reasonsByClusterKey.set(key, reason)
-                : undefined,
-        };
-        let clusterKeys;
+    }
+    if (args.validationFile && partitionReport) {
         try {
-            clusterKeys = await resolvePageClusterKeys(() => pages, resolveOptions);
+            await writeFile(args.validationFile, JSON.stringify(toJsonSafePartitionReport(partitionReport), null, 2));
         }
         catch (error) {
-            renderProgress(lanes, useTty, errorLine(error.message));
+            renderProgress(lanes, useTty, errorLine(formatErrorMessage(error)));
             return 1;
         }
-        const clusterCount = new Set(clusterKeys).size;
-        renderProgress(lanes, useTty, doneLine(pages.length, clusterCount, elapsed()));
-        for (const [index, key] of clusterKeys.entries()) {
-            const row = { id: ids[index] ?? index, clusterKey: key };
-            options.stdout.write(`${JSON.stringify(row)}\n`);
-        }
-        if (args.clusterReasonsFile && reasonsByClusterKey) {
-            try {
-                await writeFile(args.clusterReasonsFile, JSON.stringify(Object.fromEntries(reasonsByClusterKey), null, 2));
-            }
-            catch (error) {
-                renderProgress(lanes, useTty, errorLine(error.message));
-                return 1;
-            }
-        }
-        return 0;
-    }
-    finally {
-        lanes.close();
     }
+    return 0;
 }
 /**
  * Reads the package version out of `package.json` at runtime. Kept as a

package/dist/compute-cluster-cohesion.d.ts

@@ -0,0 +1,59 @@
+/**
+ * Options for {@link computeClusterCohesion}.
+ */
+export type ClusterCohesionOptions = {
+    /** @see DEFAULT_MAX_SAMPLE_SIZE */
+    readonly maxSampleSize?: number;
+    /** @see DEFAULT_SUSPICIOUS_MEDIAN_BELOW */
+    readonly suspiciousMedianBelow?: number;
+};
+/**
+ * One cluster's internal-agreement summary: how similar its members'
+ * structural tokens actually are to each other, as opposed to
+ * {@link ./merge-cross-block-clusters.js | computeQuorumCore}'s frequency
+ * core, which only reports *which* tokens are shared, not *how much* of each
+ * member's tokens that core actually covers.
+ */
+export type ClusterCohesion = {
+    readonly clusterKey: string;
+    readonly memberCount: number;
+    /** How many of `memberCount` members the pairwise comparison sampled. */
+    readonly sampledMemberCount: number;
+    /** `null` when fewer than 2 members were sampled (nothing to compare). */
+    readonly medianPairSimilarity: number | null;
+    /** 10th percentile (nearest-rank) of sampled pairwise similarities. */
+    readonly p10PairSimilarity: number | null;
+    readonly minPairSimilarity: number | null;
+    /** `medianPairSimilarity !== null && medianPairSimilarity < suspiciousMedianBelow`. */
+    readonly suspicious: boolean;
+};
+/**
+ * Reports, per cluster, how similar its members' structural token sets
+ * actually are to each other — not just which tokens they share (that's
+ * {@link ./merge-cross-block-clusters.js | computeQuorumCore}'s job), but
+ * whether the members hang together at all. A cluster built by merging
+ * unrelated templates has member pairs that mostly disagree even though a
+ * small frequency core still exists among them; this surfaces that
+ * disagreement directly, as a distribution rather than a single score, since
+ * a single "average similarity" would be pulled toward the middle by exactly
+ * the kind of partial-overlap noise this is meant to catch.
+ *
+ * Sampling uses {@link ./reservoir-sample.js | reservoirSample} seeded by
+ * each cluster's own key, so repeated calls on the same partition sample the
+ * same members and produce the same report.
+ *
+ * Deliberately returns only structured numbers, no verdict text — same
+ * design as {@link ./build-cluster-reason.js | ClusterReason} — so a caller
+ * decides what "suspicious" should mean for their own use (an interactive
+ * review queue vs. an automated gate might want different behavior for the
+ * same numbers).
+ * @param membersByKey Every cluster's member token sets, keyed by cluster key.
+ * @param options
+ * @example
+ * ```ts
+ * const report = computeClusterCohesion(membersByKey);
+ * const worstFirst = [...report].filter((r) => r.suspicious)
+ *   .toSorted((a, b) => (a.medianPairSimilarity ?? 0) - (b.medianPairSimilarity ?? 0));
+ * ```
+ */
+export declare function computeClusterCohesion(membersByKey: ReadonlyMap<string, readonly ReadonlySet<string>[]>, options?: ClusterCohesionOptions): readonly ClusterCohesion[];