@d-zero/page-cluster 0.4.0 → 0.5.0

package/README.md

@@ -15,7 +15,7 @@ yarn add @d-zero/page-cluster
 ### CLI
 
 ```sh
-page-cluster [--content-block-attribute <name>] [--include-landmark-positions] < pages.jsonl > clusters.jsonl
+page-cluster [--content-block-attribute <name>] [--cluster-reasons-file <path>] < pages.jsonl > clusters.jsonl
 ```
 
 **入力**: JSONL 1 行 1 ページ。フィールドは以下。`html` 以外はすべて任意（`paths` / `stylesheetHrefs` がないと粗い分類になる）。
@@ -36,44 +36,38 @@ page-cluster [--content-block-attribute <name>] [--include-landmark-positions] <
 { "id": "任意の識別子", "clusterKey": "..." }
 ```
 
-`--include-landmark-positions` を指定すると、各行に `landmarks` フィールドが追加される。header / footer / nav / aside / form / search / main のインスタンスごとに、HTML 内の位置（1-based の line/column と文字列オフセットの両方）を返す。header 〜 search の 6 種は追加で、そのページが属する最終クラスタ内での頻度分析（`shellQuorum`）に基づく `isChrome`（サイト/セクション共通の chrome か、ページ固有のコンテンツか）を持つ。`main` は常にコンテンツなので `isChrome` を持たない。
+`--cluster-reasons-file <path>` を指定すると、処理完了後に別ファイルとして「クラスタ選定理由」を書き出す。ページ単位ではなく**クラスタ単位**（`clusterKey` をキーにしたオブジェクト、1 クラスタにつき 1 エントリ）なので、ファイルサイズはページ数ではなくクラスタ数に比例する — ページ単位のレポートと違い、コーパスサイズの上限はない。各エントリは「なぜこのページ達が同じクラスタになったか」の根拠を構造化データとして返す: ブロッキング理由（共有 stylesheet 集合 or URL パスプレフィックス）、クラスタ内で共有されている DOM 構造トークンのコア、landmark タイプ（header/footer/nav/aside/form/search）ごとのクラスタ内共通性、そして同一ブロッキンググループ内で分岐した兄弟クラスタのキー一覧。
 
 ```json
 {
-	"id": "任意の識別子",
-	"clusterKey": "...",
-	"landmarks": {
-		"header": [
-			{
-				"startLine": 1,
-				"startColumn": 7,
-				"endLine": 1,
-				"endColumn": 30,
-				"startOffset": 6,
-				"endOffset": 29,
-				"isChrome": true
-			}
+	"[\"path:news\",\"cluster:0\"]": {
+		"memberCount": 42,
+		"blocking": [
+			{ "blockKey": "path:news", "reason": { "kind": "path", "pathKey": "news" } }
 		],
-		"footer": [],
-		"nav": [],
-		"aside": [],
-		"form": [],
-		"search": [],
-		"main": [
-			{
-				"startLine": 2,
-				"startColumn": 1,
-				"endLine": 10,
-				"endColumn": 8,
-				"startOffset": 40,
-				"endOffset": 120
+		"structuralCoreTokens": ["body>main>article", "..."],
+		"landmarks": {
+			"header": {
+				"presenceRate": 1,
+				"chromeRate": 1,
+				"shellTokens": ["..."],
+				"memberCountWithInstance": 42
+			},
+			"aside": {
+				"presenceRate": 0.3,
+				"chromeRate": 0,
+				"shellTokens": [],
+				"memberCountWithInstance": 13
 			}
-		]
+		},
+		"siblingClusterKeys": ["[\"path:news\",\"cluster:1\"]"]
 	}
 }
 ```
 
-20,000 ページを超えるコーパス（ストリーミング経路）では `--include-landmark-positions` は使えない（エラーで終了する）。ストリーミング経路はリザーバサンプリングと近似割当を使うため、ページ単位の chrome 判定に必要な「そのページが属する最終クラスタの shell トークン」という概念を持たないため。
+上の例は、header はクラスタ全体で共通の chrome（`chromeRate: 1`）である一方、aside は 42 ページ中 13 ページにしか無く chrome とは判定されていない（`chromeRate: 0`）ことを示す。つまり「ヘッダーは共通だが、サイドナビの有無で分かれているページがある」という状況を数値で表している。`siblingClusterKeys` は同じブロッキンググループ内で Stage A/B が結局別クラスタのままにした相手のキーで、それぞれの `ClusterReason` を突き合わせれば「何が違って分かれたか」を呼び出し側で解釈できる。理由は構造化データのみで、文言化（「ヘッダーが共通です」等）は呼び出し側の責務。
+
+landmark の位置情報そのもの（HTML 内のどこにあるか）が必要な場合は、ライブラリの `extractLandmarks`（ステートレス・公開 API）をページの HTML に対して自分で呼び、その結果と `ClusterReason.landmarks[type].shellTokens` を突き合わせて `isChromeLandmarkInstance`（同じく公開 API）で chrome 判定すればよい。詳細は [Library](#library) 節を参照。
 
 クローラ出力が JSON 配列の場合は `jq` で line-delimited に変換して食わせる:
 
@@ -84,7 +78,7 @@ jq -c '.[]' crawl-output.json | page-cluster > clusters.jsonl
 #### オプション
 
 - `--content-block-attribute <name>` — CMS が自由編集コンテンツブロックに付与している属性名（例: `data-bgb`）が分かっている場合に指定する。指定すると比較前にその属性を持つ要素配下を無視するので、同じテンプレートで本文構成だけ違うページを混同しなくなる。唯一の site-specific なオプションで、未指定でも `<main>` / `role="main"` を起点にした自動深さキャップが常時働く（詳細は `resolve-page-cluster-keys.ts` の JSDoc を参照）
-- `--include-landmark-positions` — 出力の各行に上記の `landmarks` フィールドを追加する。20,000 ページ超のコーパスでは使えない。指定すると進捗表示（後述）は出なくなる（進捗を出さない非ストリーミング経路に常に振り分けられるため）
+- `--cluster-reasons-file <path>` — 上記の「クラスタ選定理由」を `<path>` に JSON として書き出す。ページ数の上限はない。20,000 ページ以下のコーパスでは、指定すると進捗表示（後述）は出なくなる（進捗を出さない非ストリーミング経路に常に振り分けられるため。20,000 ページ超のストリーミング経路では進捗表示・クラスタ理由の両方が動く）
 - `--help` / `-h` — ヘルプを表示する
 - `--version` / `-v` — バージョンを表示する
 
@@ -116,12 +110,15 @@ silence したい場合は `2>/dev/null`。ログに残したい場合は `2> pr
 
 サブパスエクスポート構成。import パスと提供関数の対応は以下。
 
-| import パス                                          | 提供関数                                                                                                                                                                                                                                                                                                                          |
-| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `@d-zero/page-cluster`                               | `tokenize` — `<body>` 配下を構造トークン列に変換する低レベルプリミティブ                                                                                                                                                                                                                                                          |
-| `@d-zero/page-cluster/resolve-page-cluster-keys`     | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array 入力）。いずれも `includeLandmarkPositions: true` を渡すと `clusterKey` に加えて位置情報つきの `landmarks`（`PageLandmarkReport`）を返す |
-| `@d-zero/page-cluster/extract-landmarks`             | `extractLandmarks` — header / footer / nav / aside / form / search / main の 7 種を抽出し、インスタンスごとの生 HTML と HTML 内の位置（line/column・文字列オフセット）を返す                                                                                                                                                      |
-| `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類                                                                                                                                                                                                                                                 |
+| import パス                                          | 提供関数                                                                                                                                                                                                                                                                                            |
+| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `@d-zero/page-cluster`                               | `tokenize` — `<body>` 配下を構造トークン列に変換する低レベルプリミティブ                                                                                                                                                                                                                            |
+| `@d-zero/page-cluster/resolve-page-cluster-keys`     | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array 入力）。`onClusterReason` コールバックを渡すと、確定したクラスタごとに 1 回だけ `ClusterReason` を通知する |
+| `@d-zero/page-cluster/build-cluster-reason`          | `ClusterReason` / `LandmarkClusterProfile` 型、`buildClusterReason` — クラスタ選定理由の型定義と組み立て関数（通常は `resolvePageClusterKeys` の `onClusterReason` 経由で使うので直接呼ぶ必要はない）                                                                                               |
+| `@d-zero/page-cluster/extract-landmarks`             | `extractLandmarks` — header / footer / nav / aside / form / search / main の 7 種を抽出し、インスタンスごとの生 HTML と HTML 内の位置（line/column・文字列オフセット）を返す                                                                                                                        |
+| `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類                                                                                                                                                                                                                   |
+| `@d-zero/page-cluster/is-chrome-landmark-instance`   | `isChromeLandmarkInstance` — 1 つの landmark インスタンスのトークン集合と `ClusterReason.landmarks[type].shellTokens` のようなシェルトークン集合を突き合わせて chrome/content を判定するステートレス関数                                                                                            |
+| `@d-zero/page-cluster/jaccard-similarity`            | `jaccardSimilarity` — 2 つのトークン集合の Jaccard 類似度。`ClusterReason` 同士（`structuralCoreTokens` や `shellTokens`）を比較して兄弟クラスタとの差分を調べる用途などに使う                                                                                                                      |
 
 ```ts
 import { resolvePageClusterKeysFromArray } from '@d-zero/page-cluster/resolve-page-cluster-keys';
@@ -175,7 +172,7 @@ flowchart TD
 - **chrome discovery** — 全ページのランドマーク署名の度数分布に auto-cut を当て、閾値以上を「グローバル chrome」（サイト共通のヘッダー等）として比較から除外し、閾値未満かつ 2 ページ以上に出現するものを「ローカル chrome」（セクション固有のナビ等）としてトークン再注入する
 - **Stage A（ブロック内クラスタリング）** — ブロックごとに直線的な処理。`<main>` の深さキャップ（候補深度を全走査して knee を探す自動選択）→ tokenize → complete-linkage 階層クラスタリング → max-gap auto-cut でカット高を決定 → 最後に包含関係にあるクラスタを吸収する包含割当（割当チェーンを辿り、循環はメンバー最大のクラスタをルートに選んで解決）
 - **Pass 1b（ストリーミング時のみ）** — 20,000 ページ超では各ブロックをリザーバサンプリング（最大 100 ページ、ブロックキーをシードにした決定的乱数）で代表させ、サンプル外のページは Stage A 完了後に max-Jaccard で最寄りクラスタへ一括割当する。メモリ使用量はコーパス全体ではなくサンプルサイズに比例する
-- **Stage B（ブロック越えマージ）** — ブロック分割はあくまで比較コスト削減のためなので、最後に同一テンプレートがブロックを跨いで分かれていないか再統合する。これが唯一の反復処理（次節）
+- **Stage B（ブロック越えマージ）** — ブロック分割はあくまで比較コスト削減のためなので、最後に同一テンプレートがブロックを跨いで分かれていないか再統合する。これが唯一の反復処理（次節）。収束後、`onClusterReason` が指定されていれば、確定した最終クラスタごとに `ClusterReason` を 1 回ずつ組み立てて通知する — 追加の全コーパススキャンではなく、Stage A/B が既に計算済みの中間データ（quorum core、landmark インスタンス、ブロッキング根拠）を再利用するだけなので、クラスタ数にしか比例しない
 
 ### Stage B: ブロック越え統合の不動点ループ
 

package/dist/build-cluster-reason.d.ts

@@ -0,0 +1,119 @@
+import type { BlockingReason } from './derive-blocking-reason.js';
+import type { LandmarkType } from './extract-landmarks.js';
+import type { PerPageLandmarkInstance } from './per-page-landmark-signatures.js';
+/**
+ * One final cluster's common-vs-varying profile for a single landmark type
+ * (header/footer/nav/aside/form/search), derived by running
+ * {@link ./shell-quorum.js | shellQuorum} on just that type's instances across
+ * every member page. Absent from {@link ClusterReason}'s `landmarks` entirely
+ * when no member page carries an instance of the type.
+ */
+export type LandmarkClusterProfile = {
+    /** Fraction (0–1) of the cluster's member pages carrying at least one instance of this type. */
+    readonly presenceRate: number;
+    /**
+     * Fraction (0–1) of this type's instances, across all member pages, that
+     * {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance}
+     * classified as chrome against this cluster's own shell for the type —
+     * the "header/footer is shared chrome" half of the user-facing story.
+     *
+     * The denominator is per-page-deduplicated instances (same rule
+     * {@link ./per-page-landmark-signatures.js | computePerPageLandmarkInstances}
+     * always applies: two instances on the same page that tokenize to the
+     * same signature — a CMS-duplicated footer, or a `<header
+     * role="navigation">` matching both `header` and `nav` at the identical
+     * span — count once). A caller reconstructing a single instance's chrome
+     * verdict via `extractLandmarks` + `tokenize` +
+     * `isChromeLandmarkInstance` classifies that one instance correctly
+     * regardless of this rule; only a caller trying to reproduce this exact
+     * ratio by counting raw, un-deduplicated instances across a page would
+     * see a different number.
+     */
+    readonly chromeRate: number;
+    /**
+     * The shell token set `shellQuorum` discovered for this type within this
+     * cluster. Exposed as raw evidence — comparing two sibling clusters'
+     * `shellTokens` for the same type (e.g. via
+     * {@link ./jaccard-similarity.js | jaccardSimilarity}) is how a caller
+     * answers "do these two clusters share the same footer but differ in
+     * whether they have a sidebar nav".
+     */
+    readonly shellTokens: readonly string[];
+    /** How many member pages contributed at least one instance of this type. */
+    readonly memberCountWithInstance: number;
+};
+/**
+ * Structured, uninterpreted explanation of why a final cluster's member
+ * pages ended up together, and which sibling clusters (same Pass-0 block,
+ * different final cluster) they were nonetheless split from. Deliberately
+ * carries no human-readable text or verdicts ("these are the same template
+ * except for X") — that judgment belongs to the caller, which has the
+ * product context (and the localization requirements) `@d-zero/page-cluster`
+ * itself does not.
+ */
+export type ClusterReason = {
+    /** Number of pages this final cluster contains. */
+    readonly memberCount: number;
+    /**
+     * The distinct Pass-0 blocking keys that fed into this final cluster, and
+     * the evidence behind each. Usually one entry; more than one means Stage B
+     * merged pages that started in different blocks — itself a notable part
+     * of the explanation (e.g. two differently-styled URL sections turned out
+     * to share the same DOM template).
+     */
+    readonly blocking: readonly {
+        readonly blockKey: string;
+        readonly reason: BlockingReason;
+    }[];
+    /**
+     * The frequency-quorum core of this cluster's DOM structural tokens (see
+     * {@link ./merge-cross-block-clusters.js | computeQuorumCore}) — the
+     * "this DOM structure is common" evidence, independent of CSS or
+     * landmarks.
+     */
+    readonly structuralCoreTokens: readonly string[];
+    /** Per-landmark-type commonality. Types absent from every member page are omitted. */
+    readonly landmarks: {
+        readonly [K in LandmarkType]?: LandmarkClusterProfile;
+    };
+    /**
+     * Final cluster keys that share at least one Pass-0 block with this
+     * cluster but were not merged into it by Stage A/B — candidates for "why
+     * did these split" comparison. Does not include clusters that started in
+     * a different block from this one.
+     */
+    readonly siblingClusterKeys: readonly string[];
+};
+/**
+ * Builds one final cluster's {@link ClusterReason} from data Stage A/B
+ * already computed for clustering itself — no re-tokenization, no re-running
+ * `shellQuorum`'s corpus-wide discovery pass, just re-deriving per-type shells
+ * and quorum cores from the same pooled member state
+ * {@link ./merge-cross-block-clusters.js | mergeCrossBlockClusters} already
+ * built and would otherwise have discarded.
+ * @param input
+ * @param input.tokenSets
+ * @param input.landmarkInstances
+ * @param input.blocking
+ * @param input.siblingClusterKeys
+ * @param input.chromeThreshold
+ * @example
+ * ```ts
+ * const reason = buildClusterReason({
+ *   tokenSets: finalGroup.tokenSets,
+ *   landmarkInstances: finalGroup.landmarkInstances,
+ *   blocking: [{ blockKey: 'css:abc123', reason: { kind: 'css', distinctiveStylesheetHrefs: ['/a.css'] } }],
+ *   siblingClusterKeys: ['["css:abc123","cluster:1"]'],
+ * });
+ * ```
+ */
+export declare function buildClusterReason(input: {
+    readonly tokenSets: readonly ReadonlySet<string>[];
+    readonly landmarkInstances: readonly (readonly PerPageLandmarkInstance[])[];
+    readonly blocking: readonly {
+        readonly blockKey: string;
+        readonly reason: BlockingReason;
+    }[];
+    readonly siblingClusterKeys: readonly string[];
+    readonly chromeThreshold?: number;
+}): ClusterReason;

package/dist/build-cluster-reason.js

@@ -0,0 +1,63 @@
+import { DEFAULT_CHROME_OVERLAP_THRESHOLD, isChromeLandmarkInstance, } from './is-chrome-landmark-instance.js';
+import { computeQuorumCore } from './merge-cross-block-clusters.js';
+import { ALL_LANDMARK_TYPES } from './per-page-landmark-signatures.js';
+import { shellQuorum } from './shell-quorum.js';
+/**
+ * Builds one final cluster's {@link ClusterReason} from data Stage A/B
+ * already computed for clustering itself — no re-tokenization, no re-running
+ * `shellQuorum`'s corpus-wide discovery pass, just re-deriving per-type shells
+ * and quorum cores from the same pooled member state
+ * {@link ./merge-cross-block-clusters.js | mergeCrossBlockClusters} already
+ * built and would otherwise have discarded.
+ * @param input
+ * @param input.tokenSets
+ * @param input.landmarkInstances
+ * @param input.blocking
+ * @param input.siblingClusterKeys
+ * @param input.chromeThreshold
+ * @example
+ * ```ts
+ * const reason = buildClusterReason({
+ *   tokenSets: finalGroup.tokenSets,
+ *   landmarkInstances: finalGroup.landmarkInstances,
+ *   blocking: [{ blockKey: 'css:abc123', reason: { kind: 'css', distinctiveStylesheetHrefs: ['/a.css'] } }],
+ *   siblingClusterKeys: ['["css:abc123","cluster:1"]'],
+ * });
+ * ```
+ */
+export function buildClusterReason(input) {
+    const memberCount = input.tokenSets.length;
+    const chromeThreshold = input.chromeThreshold ?? DEFAULT_CHROME_OVERLAP_THRESHOLD;
+    const structuralCoreTokens = [...computeQuorumCore(input.tokenSets)].toSorted();
+    const landmarks = {};
+    for (const type of ALL_LANDMARK_TYPES) {
+        const perMemberInstancesOfType = input.landmarkInstances.map((instances) => instances.filter((instance) => instance.type === type));
+        const membersWithInstance = perMemberInstancesOfType.filter((instances) => instances.length > 0);
+        if (membersWithInstance.length === 0)
+            continue;
+        const shellTokens = shellQuorum(perMemberInstancesOfType);
+        let chromeInstanceCount = 0;
+        let totalInstanceCount = 0;
+        for (const instances of membersWithInstance) {
+            for (const instance of instances) {
+                totalInstanceCount++;
+                if (isChromeLandmarkInstance(instance.tokens, shellTokens, chromeThreshold)) {
+                    chromeInstanceCount++;
+                }
+            }
+        }
+        landmarks[type] = {
+            presenceRate: membersWithInstance.length / memberCount,
+            chromeRate: totalInstanceCount === 0 ? 0 : chromeInstanceCount / totalInstanceCount,
+            shellTokens: [...shellTokens].toSorted(),
+            memberCountWithInstance: membersWithInstance.length,
+        };
+    }
+    return {
+        memberCount,
+        blocking: input.blocking,
+        structuralCoreTokens,
+        landmarks,
+        siblingClusterKeys: input.siblingClusterKeys,
+    };
+}

package/dist/cli.d.ts

@@ -5,7 +5,7 @@
  */
 type CliArgs = {
     readonly contentBlockAttribute?: string;
-    readonly includeLandmarkPositions?: boolean;
+    readonly clusterReasonsFile?: string;
     readonly help?: boolean;
     readonly version?: boolean;
     readonly unknownFlag?: string;

package/dist/cli.js

@@ -4,11 +4,12 @@
 // writes JSONL to stdout (one cluster assignment per line, in input order),
 // and streams progress to stderr via `@d-zero/dealer`'s `Lanes` — in-place
 // animated header on a TTY, appended `[page-cluster] …` lines otherwise.
+import { writeFile } from 'node:fs/promises';
 import process from 'node:process';
 import { Lanes } from '@d-zero/dealer';
 import { resolvePageClusterKeys } from './resolve-page-cluster-keys.js';
 const HELP_TEXT = `Usage:
-  page-cluster [--content-block-attribute <name>] [--include-landmark-positions] < pages.jsonl > clusters.jsonl
+  page-cluster [--content-block-attribute <name>] [--cluster-reasons-file <path>] < pages.jsonl > clusters.jsonl
 
 Input (JSONL, one page per line):
   {
@@ -22,20 +23,27 @@ Input (JSONL, one page per line):
 Output (JSONL, one line per input page, in input order):
   { "id": "...", "clusterKey": "..." }
 
-  With --include-landmark-positions, each line additionally carries a
-  \`landmarks\` field: every header/footer/nav/aside/form/search/main
-  instance's position (1-based line/column plus string offsets), with the
-  six excisable types (all but main) also carrying an \`isChrome\` verdict
-  against that page's final cluster:
+  With --cluster-reasons-file <path>, a separate JSON file is written once
+  processing completes: an object keyed by clusterKey, one entry per final
+  cluster (not per page — a ClusterReason is sized by cluster count, so
+  this file has no page-count limit, unlike a per-page report would).
+  Each ClusterReason reports the blocking evidence that grouped the
+  cluster (shared stylesheet set or URL path prefix), the shared DOM-
+  structural token core, per-landmark-type (header/footer/nav/aside/form/
+  search) commonality within the cluster, and the sibling cluster keys it
+  was split from within the same blocking group — e.g. "header/footer are
+  common chrome across both clusters, but they differ in whether a
+  sidebar nav is present":
   {
-    "id": "...", "clusterKey": "...",
-    "landmarks": {
-      "header": [{ "startLine": 1, "startColumn": 7, "endLine": 1,
-                    "endColumn": 30, "startOffset": 6, "endOffset": 29,
-                    "isChrome": true }],
-      "footer": [...], "nav": [...], "aside": [...], "form": [...], "search": [...],
-      "main": [{ "startLine": 2, "startColumn": 1, "endLine": 10,
-                  "endColumn": 8, "startOffset": 40, "endOffset": 120 }]
+    "[\\"path:news\\",\\"cluster:0\\"]": {
+      "memberCount": 42,
+      "blocking": [{ "blockKey": "path:news", "reason": { "kind": "path", "pathKey": "news" } }],
+      "structuralCoreTokens": ["body>main>article", "..."],
+      "landmarks": {
+        "header": { "presenceRate": 1, "chromeRate": 1, "shellTokens": ["..."], "memberCountWithInstance": 42 },
+        "aside": { "presenceRate": 0.3, "chromeRate": 0, "shellTokens": [], "memberCountWithInstance": 13 }
+      },
+      "siblingClusterKeys": ["[\\"path:news\\",\\"cluster:1\\"]"]
     }
   }
 
@@ -43,14 +51,10 @@ Options:
   --content-block-attribute <name>   CMS-provided attribute marking freeform
                                      content blocks that should be stripped
                                      before comparison (e.g. \`data-bgb\`).
-  --include-landmark-positions       Add the \`landmarks\` field described
-                                     above to every output line. Not
-                                     supported for corpora over 20,000 pages
-                                     (throws instead of streaming). Disables
-                                     progress output on stderr — this option
-                                     always routes through the same
-                                     non-progress-emitting code path as a
-                                     run without progress.
+  --cluster-reasons-file <path>      Write the per-cluster ClusterReason
+                                     object described above to <path> after
+                                     processing completes. No page-count
+                                     limit.
   --help                             Print this help and exit.
   --version                          Print the package version and exit.
 
@@ -107,8 +111,14 @@ export function parseArgs(argv) {
                 i++;
                 break;
             }
-            case '--include-landmark-positions': {
-                out.includeLandmarkPositions = true;
+            case '--cluster-reasons-file': {
+                const next = argv[i + 1];
+                if (next === undefined) {
+                    out.unknownFlag = `${arg} requires a value`;
+                    return out;
+                }
+                out.clusterReasonsFile = next;
+                i++;
                 break;
             }
             default: {
@@ -354,30 +364,24 @@ export async function runCli(options) {
             return 1;
         }
         renderProgress(lanes, useTty, readingDoneLine(pages.length));
+        // Only worth collecting when the caller asked for the file — a
+        // ClusterReason Map costs bookkeeping proportional to cluster count,
+        // not page count, but there's no reason to pay even that when unused.
+        const reasonsByClusterKey = args.clusterReasonsFile
+            ? new Map()
+            : undefined;
         const resolveOptions = {
             contentBlockAttribute: args.contentBlockAttribute,
             onProgress: (event) => {
                 renderProgress(lanes, useTty, formatProgressLine(event, elapsed()));
             },
+            onClusterReason: reasonsByClusterKey
+                ? (key, reason) => reasonsByClusterKey.set(key, reason)
+                : undefined,
         };
-        // `includeLandmarkPositions` always routes resolvePageClusterKeys
-        // through its non-progress-emitting sync path (see that option's own
-        // JSDoc), so the onProgress callback above is set but never invoked
-        // in this branch — no separate "quiet" resolveOptions variant needed.
         let clusterKeys;
-        let landmarksByIndex;
         try {
-            if (args.includeLandmarkPositions) {
-                const results = await resolvePageClusterKeys(() => pages, {
-                    ...resolveOptions,
-                    includeLandmarkPositions: true,
-                });
-                clusterKeys = results.map((r) => r.clusterKey);
-                landmarksByIndex = results.map((r) => r.landmarks);
-            }
-            else {
-                clusterKeys = await resolvePageClusterKeys(() => pages, resolveOptions);
-            }
+            clusterKeys = await resolvePageClusterKeys(() => pages, resolveOptions);
         }
         catch (error) {
             renderProgress(lanes, useTty, errorLine(error.message));
@@ -386,14 +390,18 @@ export async function runCli(options) {
         const clusterCount = new Set(clusterKeys).size;
         renderProgress(lanes, useTty, doneLine(pages.length, clusterCount, elapsed()));
         for (const [index, key] of clusterKeys.entries()) {
-            const row = {
-                id: ids[index] ?? index,
-                clusterKey: key,
-            };
-            if (landmarksByIndex)
-                row.landmarks = landmarksByIndex[index];
+            const row = { id: ids[index] ?? index, clusterKey: key };
             options.stdout.write(`${JSON.stringify(row)}\n`);
         }
+        if (args.clusterReasonsFile && reasonsByClusterKey) {
+            try {
+                await writeFile(args.clusterReasonsFile, JSON.stringify(Object.fromEntries(reasonsByClusterKey), null, 2));
+            }
+            catch (error) {
+                renderProgress(lanes, useTty, errorLine(error.message));
+                return 1;
+            }
+        }
         return 0;
     }
     finally {

package/dist/derive-blocking-reason.d.ts

@@ -0,0 +1,32 @@
+/**
+ * The evidence behind one Pass-0 blocking key — which signal
+ * {@link ./resolve-blocking-group-keys.js | resolveBlockingGroupKeys} (and,
+ * for the `orphanMerge` variant, {@link ./reassign-orphan-block-keys.js |
+ * reassignOrphanBlockKeys}) actually used to decide it, carried verbatim with
+ * no added interpretation. Every page sharing a `css:<hash>` blocking key
+ * shares the exact same `distinctiveStylesheetHrefs` set by construction (the
+ * hash is derived from that set), so one `BlockingReason` per distinct
+ * blocking key is enough — it does not need to vary per page.
+ */
+export type BlockingReason = {
+    readonly kind: 'css';
+    /**
+     * The sorted, deduplicated, first-party stylesheet hrefs left after
+     * corpus-wide chrome removal — the exact set
+     * {@link ./derive-stylesheet-group-key.js | deriveStylesheetGroupKey}
+     * hashed into this blocking key.
+     */
+    readonly distinctiveStylesheetHrefs: readonly string[];
+} | {
+    readonly kind: 'path';
+    /** The `derivePathGroupKey` result this blocking key was derived from. */
+    readonly pathKey: string;
+} | {
+    readonly kind: 'orphanMerge';
+    /**
+     * The confined `path:` key this stylesheet-less page was folded into a
+     * same-section `css:` block under — see
+     * {@link ./reassign-orphan-block-keys.js | reassignOrphanBlockKeys}.
+     */
+    readonly pathKey: string;
+};

package/dist/derive-blocking-reason.js

@@ -0,0 +1 @@
+export {};

package/dist/merge-cross-block-clusters.d.ts

@@ -18,11 +18,47 @@ export type CrossBlockUnit = {
     readonly memberTokenSets: readonly ReadonlySet<string>[];
     readonly memberLandmarkInstances: readonly (readonly PerPageLandmarkInstance[])[];
 };
+/**
+ * Frequency-based token core of a group of member pages: a token must be
+ * present in at least `QUORUM_FRACTION` of members to enter the core, with a
+ * full-union fallback when no token clears that bar (see
+ * {@link mergeCrossBlockClusters}'s JSDoc for why quorum beats strict
+ * intersection or full union). Exported so callers building a `ClusterReason`
+ * (`build-cluster-reason.ts`) can re-derive a final group's structural core
+ * from `finalGroupsByRoot` without duplicating this logic.
+ * @param memberDistinctiveTokens
+ */
+export declare function computeQuorumCore(memberDistinctiveTokens: readonly ReadonlySet<string>[]): ReadonlySet<string>;
+/**
+ * One root key's pooled member state after Stage B converges: every
+ * `tokenSets`/`landmarkInstances` entry folded in from every unit merged into
+ * this root (down-sampled to `capMembers` when the caller opts in, same as
+ * during merging). This is the exact state `mergeCrossBlockClusters` already
+ * builds internally to run quorum-core/shell comparisons each round — it was
+ * discarded once `keyToRoot` was returned. Exposing it lets a `ClusterReason`
+ * (`build-cluster-reason.ts`) be built from data Stage B already computed,
+ * with no extra pass over the corpus.
+ */
+export type FinalGroupMembers = {
+    readonly tokenSets: readonly ReadonlySet<string>[];
+    readonly landmarkInstances: readonly (readonly PerPageLandmarkInstance[])[];
+};
+/**
+ * `mergeCrossBlockClusters`'s result: the root-key mapping every caller
+ * needs for `clusterKey` resolution, plus each surviving root's final pooled
+ * member state for callers that also want to explain *why* (`ClusterReason`).
+ */
+export type MergeCrossBlockClustersResult = {
+    /** Every input unit's `key` mapped to its final root key. Units not absorbed into any other unit map to themselves. */
+    readonly rootByKey: ReadonlyMap<string, string>;
+    /** Every surviving root key's final pooled member state. */
+    readonly finalGroupsByRoot: ReadonlyMap<string, FinalGroupMembers>;
+};
 /**
  * Merges cross-block clusters (Stage B) via recursive quorum-core comparison.
  *
- * Returns a `Map` from each input unit's `key` to its final root key. Units
- * not absorbed into any other unit map to themselves.
+ * Returns the root-key mapping (see {@link MergeCrossBlockClustersResult}).
+ * Units not absorbed into any other unit map to themselves.
  *
  * Three merge mechanisms run per round, in order:
  * 1. **Fine stage** — complete-linkage at `CROSS_BLOCK_THRESHOLD` on quorum
@@ -58,4 +94,4 @@ export declare function mergeCrossBlockClusters(units: readonly CrossBlockUnit[]
      * full-membership merge behavior unchanged.
      */
     capMembers?: number;
-}): Map<string, string>;
+}): MergeCrossBlockClustersResult;

package/dist/merge-cross-block-clusters.js

@@ -83,10 +83,16 @@ const GENERIC_SEGMENTS = new Set([
 // Internal helpers
 // ---------------------------------------------------------------------------
 /**
- *
+ * Frequency-based token core of a group of member pages: a token must be
+ * present in at least `QUORUM_FRACTION` of members to enter the core, with a
+ * full-union fallback when no token clears that bar (see
+ * {@link mergeCrossBlockClusters}'s JSDoc for why quorum beats strict
+ * intersection or full union). Exported so callers building a `ClusterReason`
+ * (`build-cluster-reason.ts`) can re-derive a final group's structural core
+ * from `finalGroupsByRoot` without duplicating this logic.
  * @param memberDistinctiveTokens
  */
-function quorumCore(memberDistinctiveTokens) {
+export function computeQuorumCore(memberDistinctiveTokens) {
     const n = memberDistinctiveTokens.length;
     if (n === 0)
         return new Set();
@@ -158,14 +164,11 @@ function l2Contained(xSig, ySig) {
     }
     return true;
 }
-// ---------------------------------------------------------------------------
-// Main function
-// ---------------------------------------------------------------------------
 /**
  * Merges cross-block clusters (Stage B) via recursive quorum-core comparison.
  *
- * Returns a `Map` from each input unit's `key` to its final root key. Units
- * not absorbed into any other unit map to themselves.
+ * Returns the root-key mapping (see {@link MergeCrossBlockClustersResult}).
+ * Units not absorbed into any other unit map to themselves.
  *
  * Three merge mechanisms run per round, in order:
  * 1. **Fine stage** — complete-linkage at `CROSS_BLOCK_THRESHOLD` on quorum
@@ -191,7 +194,13 @@ function l2Contained(xSig, ySig) {
  */
 export function mergeCrossBlockClusters(units, options) {
     if (units.length <= 1) {
-        return new Map(units.map((u) => [u.key, u.key]));
+        return {
+            rootByKey: new Map(units.map((u) => [u.key, u.key])),
+            finalGroupsByRoot: new Map(units.map((u) => [
+                u.key,
+                { tokenSets: u.memberTokenSets, landmarkInstances: u.memberLandmarkInstances },
+            ])),
+        };
     }
     const threshold = options?.similarityThreshold ?? CROSS_BLOCK_THRESHOLD;
     const capMembers = options?.capMembers;
@@ -264,7 +273,7 @@ export function mergeCrossBlockClusters(units, options) {
         // Quorum core per group
         const cores = new Map();
         for (const key of groupKeys) {
-            cores.set(key, quorumCore(groupDistinctive.get(key) ?? []));
+            cores.set(key, computeQuorumCore(groupDistinctive.get(key) ?? []));
         }
         // ---------------------------------------------------------------
         // Fine stage: union-find over group indices
@@ -450,5 +459,9 @@ export function mergeCrossBlockClusters(units, options) {
             break; // fully converged
         applyMerges(l2Merges);
     }
-    return keyToRoot;
+    const finalGroupsByRoot = new Map([...groups.entries()].map(([root, g]) => [
+        root,
+        { tokenSets: g.tokenSets, landmarkInstances: g.landmarkInstances },
+    ]));
+    return { rootByKey: keyToRoot, finalGroupsByRoot };
 }