@d-zero/page-cluster 0.3.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +45 -8
- package/dist/build-cluster-reason.d.ts +119 -0
- package/dist/build-cluster-reason.js +63 -0
- package/dist/cli.d.ts +1 -0
- package/dist/cli.js +64 -6
- package/dist/derive-blocking-reason.d.ts +32 -0
- package/dist/derive-blocking-reason.js +1 -0
- package/dist/extract-landmarks.d.ts +75 -14
- package/dist/extract-landmarks.js +64 -4
- package/dist/is-chrome-landmark-instance.d.ts +41 -0
- package/dist/is-chrome-landmark-instance.js +50 -0
- package/dist/merge-cross-block-clusters.d.ts +39 -3
- package/dist/merge-cross-block-clusters.js +28 -107
- package/dist/offset-to-line-column.d.ts +38 -0
- package/dist/offset-to-line-column.js +49 -0
- package/dist/pass0-blocking.d.ts +10 -0
- package/dist/pass0-blocking.js +26 -46
- package/dist/per-page-landmark-signatures.d.ts +9 -1
- package/dist/per-page-landmark-signatures.js +16 -4
- package/dist/reassign-orphan-block-keys.d.ts +9 -0
- package/dist/reassign-orphan-block-keys.js +5 -2
- package/dist/resolve-blocking-group-keys.d.ts +10 -0
- package/dist/resolve-blocking-group-keys.js +16 -86
- package/dist/resolve-page-cluster-keys.d.ts +37 -1
- package/dist/resolve-page-cluster-keys.js +123 -17
- package/dist/shell-quorum.d.ts +70 -0
- package/dist/shell-quorum.js +110 -0
- package/package.json +10 -2
package/README.md
CHANGED
|
@@ -15,7 +15,7 @@ yarn add @d-zero/page-cluster
|
|
|
15
15
|
### CLI
|
|
16
16
|
|
|
17
17
|
```sh
|
|
18
|
-
page-cluster [--content-block-attribute <name>] < pages.jsonl > clusters.jsonl
|
|
18
|
+
page-cluster [--content-block-attribute <name>] [--cluster-reasons-file <path>] < pages.jsonl > clusters.jsonl
|
|
19
19
|
```
|
|
20
20
|
|
|
21
21
|
**入力**: JSONL 1 行 1 ページ。フィールドは以下。`html` 以外はすべて任意（`paths` / `stylesheetHrefs` がないと粗い分類になる）。
|
|
@@ -36,6 +36,39 @@ page-cluster [--content-block-attribute <name>] < pages.jsonl > clusters.jsonl
|
|
|
36
36
|
{ "id": "任意の識別子", "clusterKey": "..." }
|
|
37
37
|
```
|
|
38
38
|
|
|
39
|
+
`--cluster-reasons-file <path>` を指定すると、処理完了後に別ファイルとして「クラスタ選定理由」を書き出す。ページ単位ではなく**クラスタ単位**（`clusterKey` をキーにしたオブジェクト、1 クラスタにつき 1 エントリ）なので、ファイルサイズはページ数ではなくクラスタ数に比例する — ページ単位のレポートと違い、コーパスサイズの上限はない。各エントリは「なぜこのページ達が同じクラスタになったか」の根拠を構造化データとして返す: ブロッキング理由（共有 stylesheet 集合 or URL パスプレフィックス）、クラスタ内で共有されている DOM 構造トークンのコア、landmark タイプ（header/footer/nav/aside/form/search）ごとのクラスタ内共通性、そして同一ブロッキンググループ内で分岐した兄弟クラスタのキー一覧。
|
|
40
|
+
|
|
41
|
+
```json
|
|
42
|
+
{
|
|
43
|
+
"[\"path:news\",\"cluster:0\"]": {
|
|
44
|
+
"memberCount": 42,
|
|
45
|
+
"blocking": [
|
|
46
|
+
{ "blockKey": "path:news", "reason": { "kind": "path", "pathKey": "news" } }
|
|
47
|
+
],
|
|
48
|
+
"structuralCoreTokens": ["body>main>article", "..."],
|
|
49
|
+
"landmarks": {
|
|
50
|
+
"header": {
|
|
51
|
+
"presenceRate": 1,
|
|
52
|
+
"chromeRate": 1,
|
|
53
|
+
"shellTokens": ["..."],
|
|
54
|
+
"memberCountWithInstance": 42
|
|
55
|
+
},
|
|
56
|
+
"aside": {
|
|
57
|
+
"presenceRate": 0.3,
|
|
58
|
+
"chromeRate": 0,
|
|
59
|
+
"shellTokens": [],
|
|
60
|
+
"memberCountWithInstance": 13
|
|
61
|
+
}
|
|
62
|
+
},
|
|
63
|
+
"siblingClusterKeys": ["[\"path:news\",\"cluster:1\"]"]
|
|
64
|
+
}
|
|
65
|
+
}
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
上の例は、header はクラスタ全体で共通の chrome（`chromeRate: 1`）である一方、aside は 42 ページ中 13 ページにしか無く chrome とは判定されていない（`chromeRate: 0`）ことを示す。つまり「ヘッダーは共通だが、サイドナビの有無で分かれているページがある」という状況を数値で表している。`siblingClusterKeys` は同じブロッキンググループ内で Stage A/B が結局別クラスタのままにした相手のキーで、それぞれの `ClusterReason` を突き合わせれば「何が違って分かれたか」を呼び出し側で解釈できる。理由は構造化データのみで、文言化（「ヘッダーが共通です」等）は呼び出し側の責務。
|
|
69
|
+
|
|
70
|
+
landmark の位置情報そのもの（HTML 内のどこにあるか）が必要な場合は、ライブラリの `extractLandmarks`（ステートレス・公開 API）をページの HTML に対して自分で呼び、その結果と `ClusterReason.landmarks[type].shellTokens` を突き合わせて `isChromeLandmarkInstance`（同じく公開 API）で chrome 判定すればよい。詳細は [Library](#library) 節を参照。
|
|
71
|
+
|
|
39
72
|
クローラ出力が JSON 配列の場合は `jq` で line-delimited に変換して食わせる:
|
|
40
73
|
|
|
41
74
|
```sh
|
|
@@ -45,6 +78,7 @@ jq -c '.[]' crawl-output.json | page-cluster > clusters.jsonl
|
|
|
45
78
|
#### オプション
|
|
46
79
|
|
|
47
80
|
- `--content-block-attribute <name>` — CMS が自由編集コンテンツブロックに付与している属性名（例: `data-bgb`）が分かっている場合に指定する。指定すると比較前にその属性を持つ要素配下を無視するので、同じテンプレートで本文構成だけ違うページを混同しなくなる。唯一の site-specific なオプションで、未指定でも `<main>` / `role="main"` を起点にした自動深さキャップが常時働く（詳細は `resolve-page-cluster-keys.ts` の JSDoc を参照）
|
|
81
|
+
- `--cluster-reasons-file <path>` — 上記の「クラスタ選定理由」を `<path>` に JSON として書き出す。ページ数の上限はない。20,000 ページ以下のコーパスでは、指定すると進捗表示（後述）は出なくなる（進捗を出さない非ストリーミング経路に常に振り分けられるため。20,000 ページ超のストリーミング経路では進捗表示・クラスタ理由の両方が動く）
|
|
48
82
|
- `--help` / `-h` — ヘルプを表示する
|
|
49
83
|
- `--version` / `-v` — バージョンを表示する
|
|
50
84
|
|
|
@@ -76,12 +110,15 @@ silence したい場合は `2>/dev/null`。ログに残したい場合は `2> pr
|
|
|
76
110
|
|
|
77
111
|
サブパスエクスポート構成。import パスと提供関数の対応は以下。
|
|
78
112
|
|
|
79
|
-
| import パス | 提供関数
|
|
80
|
-
| ---------------------------------------------------- |
|
|
81
|
-
| `@d-zero/page-cluster` | `tokenize` — `<body>` 配下を構造トークン列に変換する低レベルプリミティブ
|
|
82
|
-
| `@d-zero/page-cluster/resolve-page-cluster-keys` | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array
|
|
83
|
-
| `@d-zero/page-cluster/
|
|
84
|
-
| `@d-zero/page-cluster/
|
|
113
|
+
| import パス | 提供関数 |
|
|
114
|
+
| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
115
|
+
| `@d-zero/page-cluster` | `tokenize` — `<body>` 配下を構造トークン列に変換する低レベルプリミティブ |
|
|
116
|
+
| `@d-zero/page-cluster/resolve-page-cluster-keys` | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array 入力）。`onClusterReason` コールバックを渡すと、確定したクラスタごとに 1 回だけ `ClusterReason` を通知する |
|
|
117
|
+
| `@d-zero/page-cluster/build-cluster-reason` | `ClusterReason` / `LandmarkClusterProfile` 型、`buildClusterReason` — クラスタ選定理由の型定義と組み立て関数（通常は `resolvePageClusterKeys` の `onClusterReason` 経由で使うので直接呼ぶ必要はない） |
|
|
118
|
+
| `@d-zero/page-cluster/extract-landmarks` | `extractLandmarks` — header / footer / nav / aside / form / search / main の 7 種を抽出し、インスタンスごとの生 HTML と HTML 内の位置（line/column・文字列オフセット）を返す |
|
|
119
|
+
| `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類 |
|
|
120
|
+
| `@d-zero/page-cluster/is-chrome-landmark-instance` | `isChromeLandmarkInstance` — 1 つの landmark インスタンスのトークン集合と `ClusterReason.landmarks[type].shellTokens` のようなシェルトークン集合を突き合わせて chrome/content を判定するステートレス関数 |
|
|
121
|
+
| `@d-zero/page-cluster/jaccard-similarity` | `jaccardSimilarity` — 2 つのトークン集合の Jaccard 類似度。`ClusterReason` 同士（`structuralCoreTokens` や `shellTokens`）を比較して兄弟クラスタとの差分を調べる用途などに使う |
|
|
85
122
|
|
|
86
123
|
```ts
|
|
87
124
|
import { resolvePageClusterKeysFromArray } from '@d-zero/page-cluster/resolve-page-cluster-keys';
|
|
@@ -135,7 +172,7 @@ flowchart TD
|
|
|
135
172
|
- **chrome discovery** — 全ページのランドマーク署名の度数分布に auto-cut を当て、閾値以上を「グローバル chrome」（サイト共通のヘッダー等）として比較から除外し、閾値未満かつ 2 ページ以上に出現するものを「ローカル chrome」（セクション固有のナビ等）としてトークン再注入する
|
|
136
173
|
- **Stage A（ブロック内クラスタリング）** — ブロックごとに直線的な処理。`<main>` の深さキャップ（候補深度を全走査して knee を探す自動選択）→ tokenize → complete-linkage 階層クラスタリング → max-gap auto-cut でカット高を決定 → 最後に包含関係にあるクラスタを吸収する包含割当（割当チェーンを辿り、循環はメンバー最大のクラスタをルートに選んで解決）
|
|
137
174
|
- **Pass 1b（ストリーミング時のみ）** — 20,000 ページ超では各ブロックをリザーバサンプリング（最大 100 ページ、ブロックキーをシードにした決定的乱数）で代表させ、サンプル外のページは Stage A 完了後に max-Jaccard で最寄りクラスタへ一括割当する。メモリ使用量はコーパス全体ではなくサンプルサイズに比例する
|
|
138
|
-
- **Stage B（ブロック越えマージ）** —
|
|
175
|
+
- **Stage B（ブロック越えマージ）** — ブロック分割はあくまで比較コスト削減のためなので、最後に同一テンプレートがブロックを跨いで分かれていないか再統合する。これが唯一の反復処理（次節）。収束後、`onClusterReason` が指定されていれば、確定した最終クラスタごとに `ClusterReason` を 1 回ずつ組み立てて通知する — 追加の全コーパススキャンではなく、Stage A/B が既に計算済みの中間データ（quorum core、landmark インスタンス、ブロッキング根拠）を再利用するだけなので、クラスタ数にしか比例しない
|
|
139
176
|
|
|
140
177
|
### Stage B: ブロック越え統合の不動点ループ
|
|
141
178
|
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
import type { BlockingReason } from './derive-blocking-reason.js';
|
|
2
|
+
import type { LandmarkType } from './extract-landmarks.js';
|
|
3
|
+
import type { PerPageLandmarkInstance } from './per-page-landmark-signatures.js';
|
|
4
|
+
/**
|
|
5
|
+
* One final cluster's common-vs-varying profile for a single landmark type
|
|
6
|
+
* (header/footer/nav/aside/form/search), derived by running
|
|
7
|
+
* {@link ./shell-quorum.js | shellQuorum} on just that type's instances across
|
|
8
|
+
* every member page. Absent from {@link ClusterReason}'s `landmarks` entirely
|
|
9
|
+
* when no member page carries an instance of the type.
|
|
10
|
+
*/
|
|
11
|
+
export type LandmarkClusterProfile = {
|
|
12
|
+
/** Fraction (0–1) of the cluster's member pages carrying at least one instance of this type. */
|
|
13
|
+
readonly presenceRate: number;
|
|
14
|
+
/**
|
|
15
|
+
* Fraction (0–1) of this type's instances, across all member pages, that
|
|
16
|
+
* {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance}
|
|
17
|
+
* classified as chrome against this cluster's own shell for the type —
|
|
18
|
+
* the "header/footer is shared chrome" half of the user-facing story.
|
|
19
|
+
*
|
|
20
|
+
* The denominator is per-page-deduplicated instances (same rule
|
|
21
|
+
* {@link ./per-page-landmark-signatures.js | computePerPageLandmarkInstances}
|
|
22
|
+
* always applies: two instances on the same page that tokenize to the
|
|
23
|
+
* same signature — a CMS-duplicated footer, or a `<header
|
|
24
|
+
* role="navigation">` matching both `header` and `nav` at the identical
|
|
25
|
+
* span — count once). A caller reconstructing a single instance's chrome
|
|
26
|
+
* verdict via `extractLandmarks` + `tokenize` +
|
|
27
|
+
* `isChromeLandmarkInstance` classifies that one instance correctly
|
|
28
|
+
* regardless of this rule; only a caller trying to reproduce this exact
|
|
29
|
+
* ratio by counting raw, un-deduplicated instances across a page would
|
|
30
|
+
* see a different number.
|
|
31
|
+
*/
|
|
32
|
+
readonly chromeRate: number;
|
|
33
|
+
/**
|
|
34
|
+
* The shell token set `shellQuorum` discovered for this type within this
|
|
35
|
+
* cluster. Exposed as raw evidence — comparing two sibling clusters'
|
|
36
|
+
* `shellTokens` for the same type (e.g. via
|
|
37
|
+
* {@link ./jaccard-similarity.js | jaccardSimilarity}) is how a caller
|
|
38
|
+
* answers "do these two clusters share the same footer but differ in
|
|
39
|
+
* whether they have a sidebar nav".
|
|
40
|
+
*/
|
|
41
|
+
readonly shellTokens: readonly string[];
|
|
42
|
+
/** How many member pages contributed at least one instance of this type. */
|
|
43
|
+
readonly memberCountWithInstance: number;
|
|
44
|
+
};
|
|
45
|
+
/**
|
|
46
|
+
* Structured, uninterpreted explanation of why a final cluster's member
|
|
47
|
+
* pages ended up together, and which sibling clusters (same Pass-0 block,
|
|
48
|
+
* different final cluster) they were nonetheless split from. Deliberately
|
|
49
|
+
* carries no human-readable text or verdicts ("these are the same template
|
|
50
|
+
* except for X") — that judgment belongs to the caller, which has the
|
|
51
|
+
* product context (and the localization requirements) `@d-zero/page-cluster`
|
|
52
|
+
* itself does not.
|
|
53
|
+
*/
|
|
54
|
+
export type ClusterReason = {
|
|
55
|
+
/** Number of pages this final cluster contains. */
|
|
56
|
+
readonly memberCount: number;
|
|
57
|
+
/**
|
|
58
|
+
* The distinct Pass-0 blocking keys that fed into this final cluster, and
|
|
59
|
+
* the evidence behind each. Usually one entry; more than one means Stage B
|
|
60
|
+
* merged pages that started in different blocks — itself a notable part
|
|
61
|
+
* of the explanation (e.g. two differently-styled URL sections turned out
|
|
62
|
+
* to share the same DOM template).
|
|
63
|
+
*/
|
|
64
|
+
readonly blocking: readonly {
|
|
65
|
+
readonly blockKey: string;
|
|
66
|
+
readonly reason: BlockingReason;
|
|
67
|
+
}[];
|
|
68
|
+
/**
|
|
69
|
+
* The frequency-quorum core of this cluster's DOM structural tokens (see
|
|
70
|
+
* {@link ./merge-cross-block-clusters.js | computeQuorumCore}) — the
|
|
71
|
+
* "this DOM structure is common" evidence, independent of CSS or
|
|
72
|
+
* landmarks.
|
|
73
|
+
*/
|
|
74
|
+
readonly structuralCoreTokens: readonly string[];
|
|
75
|
+
/** Per-landmark-type commonality. Types absent from every member page are omitted. */
|
|
76
|
+
readonly landmarks: {
|
|
77
|
+
readonly [K in LandmarkType]?: LandmarkClusterProfile;
|
|
78
|
+
};
|
|
79
|
+
/**
|
|
80
|
+
* Final cluster keys that share at least one Pass-0 block with this
|
|
81
|
+
* cluster but were not merged into it by Stage A/B — candidates for "why
|
|
82
|
+
* did these split" comparison. Does not include clusters that started in
|
|
83
|
+
* a different block from this one.
|
|
84
|
+
*/
|
|
85
|
+
readonly siblingClusterKeys: readonly string[];
|
|
86
|
+
};
|
|
87
|
+
/**
|
|
88
|
+
* Builds one final cluster's {@link ClusterReason} from data Stage A/B
|
|
89
|
+
* already computed for clustering itself — no re-tokenization, no re-running
|
|
90
|
+
* `shellQuorum`'s corpus-wide discovery pass, just re-deriving per-type shells
|
|
91
|
+
* and quorum cores from the same pooled member state
|
|
92
|
+
* {@link ./merge-cross-block-clusters.js | mergeCrossBlockClusters} already
|
|
93
|
+
* built and would otherwise have discarded.
|
|
94
|
+
* @param input
|
|
95
|
+
* @param input.tokenSets
|
|
96
|
+
* @param input.landmarkInstances
|
|
97
|
+
* @param input.blocking
|
|
98
|
+
* @param input.siblingClusterKeys
|
|
99
|
+
* @param input.chromeThreshold
|
|
100
|
+
* @example
|
|
101
|
+
* ```ts
|
|
102
|
+
* const reason = buildClusterReason({
|
|
103
|
+
* tokenSets: finalGroup.tokenSets,
|
|
104
|
+
* landmarkInstances: finalGroup.landmarkInstances,
|
|
105
|
+
* blocking: [{ blockKey: 'css:abc123', reason: { kind: 'css', distinctiveStylesheetHrefs: ['/a.css'] } }],
|
|
106
|
+
* siblingClusterKeys: ['["css:abc123","cluster:1"]'],
|
|
107
|
+
* });
|
|
108
|
+
* ```
|
|
109
|
+
*/
|
|
110
|
+
export declare function buildClusterReason(input: {
|
|
111
|
+
readonly tokenSets: readonly ReadonlySet<string>[];
|
|
112
|
+
readonly landmarkInstances: readonly (readonly PerPageLandmarkInstance[])[];
|
|
113
|
+
readonly blocking: readonly {
|
|
114
|
+
readonly blockKey: string;
|
|
115
|
+
readonly reason: BlockingReason;
|
|
116
|
+
}[];
|
|
117
|
+
readonly siblingClusterKeys: readonly string[];
|
|
118
|
+
readonly chromeThreshold?: number;
|
|
119
|
+
}): ClusterReason;
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
import { DEFAULT_CHROME_OVERLAP_THRESHOLD, isChromeLandmarkInstance, } from './is-chrome-landmark-instance.js';
|
|
2
|
+
import { computeQuorumCore } from './merge-cross-block-clusters.js';
|
|
3
|
+
import { ALL_LANDMARK_TYPES } from './per-page-landmark-signatures.js';
|
|
4
|
+
import { shellQuorum } from './shell-quorum.js';
|
|
5
|
+
/**
|
|
6
|
+
* Builds one final cluster's {@link ClusterReason} from data Stage A/B
|
|
7
|
+
* already computed for clustering itself — no re-tokenization, no re-running
|
|
8
|
+
* `shellQuorum`'s corpus-wide discovery pass, just re-deriving per-type shells
|
|
9
|
+
* and quorum cores from the same pooled member state
|
|
10
|
+
* {@link ./merge-cross-block-clusters.js | mergeCrossBlockClusters} already
|
|
11
|
+
* built and would otherwise have discarded.
|
|
12
|
+
* @param input
|
|
13
|
+
* @param input.tokenSets
|
|
14
|
+
* @param input.landmarkInstances
|
|
15
|
+
* @param input.blocking
|
|
16
|
+
* @param input.siblingClusterKeys
|
|
17
|
+
* @param input.chromeThreshold
|
|
18
|
+
* @example
|
|
19
|
+
* ```ts
|
|
20
|
+
* const reason = buildClusterReason({
|
|
21
|
+
* tokenSets: finalGroup.tokenSets,
|
|
22
|
+
* landmarkInstances: finalGroup.landmarkInstances,
|
|
23
|
+
* blocking: [{ blockKey: 'css:abc123', reason: { kind: 'css', distinctiveStylesheetHrefs: ['/a.css'] } }],
|
|
24
|
+
* siblingClusterKeys: ['["css:abc123","cluster:1"]'],
|
|
25
|
+
* });
|
|
26
|
+
* ```
|
|
27
|
+
*/
|
|
28
|
+
export function buildClusterReason(input) {
|
|
29
|
+
const memberCount = input.tokenSets.length;
|
|
30
|
+
const chromeThreshold = input.chromeThreshold ?? DEFAULT_CHROME_OVERLAP_THRESHOLD;
|
|
31
|
+
const structuralCoreTokens = [...computeQuorumCore(input.tokenSets)].toSorted();
|
|
32
|
+
const landmarks = {};
|
|
33
|
+
for (const type of ALL_LANDMARK_TYPES) {
|
|
34
|
+
const perMemberInstancesOfType = input.landmarkInstances.map((instances) => instances.filter((instance) => instance.type === type));
|
|
35
|
+
const membersWithInstance = perMemberInstancesOfType.filter((instances) => instances.length > 0);
|
|
36
|
+
if (membersWithInstance.length === 0)
|
|
37
|
+
continue;
|
|
38
|
+
const shellTokens = shellQuorum(perMemberInstancesOfType);
|
|
39
|
+
let chromeInstanceCount = 0;
|
|
40
|
+
let totalInstanceCount = 0;
|
|
41
|
+
for (const instances of membersWithInstance) {
|
|
42
|
+
for (const instance of instances) {
|
|
43
|
+
totalInstanceCount++;
|
|
44
|
+
if (isChromeLandmarkInstance(instance.tokens, shellTokens, chromeThreshold)) {
|
|
45
|
+
chromeInstanceCount++;
|
|
46
|
+
}
|
|
47
|
+
}
|
|
48
|
+
}
|
|
49
|
+
landmarks[type] = {
|
|
50
|
+
presenceRate: membersWithInstance.length / memberCount,
|
|
51
|
+
chromeRate: totalInstanceCount === 0 ? 0 : chromeInstanceCount / totalInstanceCount,
|
|
52
|
+
shellTokens: [...shellTokens].toSorted(),
|
|
53
|
+
memberCountWithInstance: membersWithInstance.length,
|
|
54
|
+
};
|
|
55
|
+
}
|
|
56
|
+
return {
|
|
57
|
+
memberCount,
|
|
58
|
+
blocking: input.blocking,
|
|
59
|
+
structuralCoreTokens,
|
|
60
|
+
landmarks,
|
|
61
|
+
siblingClusterKeys: input.siblingClusterKeys,
|
|
62
|
+
};
|
|
63
|
+
}
|
package/dist/cli.d.ts
CHANGED
package/dist/cli.js
CHANGED
|
@@ -4,11 +4,12 @@
|
|
|
4
4
|
// writes JSONL to stdout (one cluster assignment per line, in input order),
|
|
5
5
|
// and streams progress to stderr via `@d-zero/dealer`'s `Lanes` — in-place
|
|
6
6
|
// animated header on a TTY, appended `[page-cluster] …` lines otherwise.
|
|
7
|
+
import { writeFile } from 'node:fs/promises';
|
|
7
8
|
import process from 'node:process';
|
|
8
9
|
import { Lanes } from '@d-zero/dealer';
|
|
9
10
|
import { resolvePageClusterKeys } from './resolve-page-cluster-keys.js';
|
|
10
11
|
const HELP_TEXT = `Usage:
|
|
11
|
-
page-cluster [--content-block-attribute <name>] < pages.jsonl > clusters.jsonl
|
|
12
|
+
page-cluster [--content-block-attribute <name>] [--cluster-reasons-file <path>] < pages.jsonl > clusters.jsonl
|
|
12
13
|
|
|
13
14
|
Input (JSONL, one page per line):
|
|
14
15
|
{
|
|
@@ -22,10 +23,38 @@ Input (JSONL, one page per line):
|
|
|
22
23
|
Output (JSONL, one line per input page, in input order):
|
|
23
24
|
{ "id": "...", "clusterKey": "..." }
|
|
24
25
|
|
|
26
|
+
With --cluster-reasons-file <path>, a separate JSON file is written once
|
|
27
|
+
processing completes: an object keyed by clusterKey, one entry per final
|
|
28
|
+
cluster (not per page — a ClusterReason is sized by cluster count, so
|
|
29
|
+
this file has no page-count limit, unlike a per-page report would).
|
|
30
|
+
Each ClusterReason reports the blocking evidence that grouped the
|
|
31
|
+
cluster (shared stylesheet set or URL path prefix), the shared DOM-
|
|
32
|
+
structural token core, per-landmark-type (header/footer/nav/aside/form/
|
|
33
|
+
search) commonality within the cluster, and the sibling cluster keys it
|
|
34
|
+
was split from within the same blocking group — e.g. "header/footer are
|
|
35
|
+
common chrome across both clusters, but they differ in whether a
|
|
36
|
+
sidebar nav is present":
|
|
37
|
+
{
|
|
38
|
+
"[\\"path:news\\",\\"cluster:0\\"]": {
|
|
39
|
+
"memberCount": 42,
|
|
40
|
+
"blocking": [{ "blockKey": "path:news", "reason": { "kind": "path", "pathKey": "news" } }],
|
|
41
|
+
"structuralCoreTokens": ["body>main>article", "..."],
|
|
42
|
+
"landmarks": {
|
|
43
|
+
"header": { "presenceRate": 1, "chromeRate": 1, "shellTokens": ["..."], "memberCountWithInstance": 42 },
|
|
44
|
+
"aside": { "presenceRate": 0.3, "chromeRate": 0, "shellTokens": [], "memberCountWithInstance": 13 }
|
|
45
|
+
},
|
|
46
|
+
"siblingClusterKeys": ["[\\"path:news\\",\\"cluster:1\\"]"]
|
|
47
|
+
}
|
|
48
|
+
}
|
|
49
|
+
|
|
25
50
|
Options:
|
|
26
51
|
--content-block-attribute <name> CMS-provided attribute marking freeform
|
|
27
52
|
content blocks that should be stripped
|
|
28
53
|
before comparison (e.g. \`data-bgb\`).
|
|
54
|
+
--cluster-reasons-file <path> Write the per-cluster ClusterReason
|
|
55
|
+
object described above to <path> after
|
|
56
|
+
processing completes. No page-count
|
|
57
|
+
limit.
|
|
29
58
|
--help Print this help and exit.
|
|
30
59
|
--version Print the package version and exit.
|
|
31
60
|
|
|
@@ -82,6 +111,16 @@ export function parseArgs(argv) {
|
|
|
82
111
|
i++;
|
|
83
112
|
break;
|
|
84
113
|
}
|
|
114
|
+
case '--cluster-reasons-file': {
|
|
115
|
+
const next = argv[i + 1];
|
|
116
|
+
if (next === undefined) {
|
|
117
|
+
out.unknownFlag = `${arg} requires a value`;
|
|
118
|
+
return out;
|
|
119
|
+
}
|
|
120
|
+
out.clusterReasonsFile = next;
|
|
121
|
+
i++;
|
|
122
|
+
break;
|
|
123
|
+
}
|
|
85
124
|
default: {
|
|
86
125
|
out.unknownFlag = arg;
|
|
87
126
|
return out;
|
|
@@ -325,24 +364,43 @@ export async function runCli(options) {
|
|
|
325
364
|
return 1;
|
|
326
365
|
}
|
|
327
366
|
renderProgress(lanes, useTty, readingDoneLine(pages.length));
|
|
367
|
+
// Only worth collecting when the caller asked for the file — a
|
|
368
|
+
// ClusterReason Map costs bookkeeping proportional to cluster count,
|
|
369
|
+
// not page count, but there's no reason to pay even that when unused.
|
|
370
|
+
const reasonsByClusterKey = args.clusterReasonsFile
|
|
371
|
+
? new Map()
|
|
372
|
+
: undefined;
|
|
328
373
|
const resolveOptions = {
|
|
329
374
|
contentBlockAttribute: args.contentBlockAttribute,
|
|
330
375
|
onProgress: (event) => {
|
|
331
376
|
renderProgress(lanes, useTty, formatProgressLine(event, elapsed()));
|
|
332
377
|
},
|
|
378
|
+
onClusterReason: reasonsByClusterKey
|
|
379
|
+
? (key, reason) => reasonsByClusterKey.set(key, reason)
|
|
380
|
+
: undefined,
|
|
333
381
|
};
|
|
334
|
-
let
|
|
382
|
+
let clusterKeys;
|
|
335
383
|
try {
|
|
336
|
-
|
|
384
|
+
clusterKeys = await resolvePageClusterKeys(() => pages, resolveOptions);
|
|
337
385
|
}
|
|
338
386
|
catch (error) {
|
|
339
387
|
renderProgress(lanes, useTty, errorLine(error.message));
|
|
340
388
|
return 1;
|
|
341
389
|
}
|
|
342
|
-
const clusterCount = new Set(
|
|
390
|
+
const clusterCount = new Set(clusterKeys).size;
|
|
343
391
|
renderProgress(lanes, useTty, doneLine(pages.length, clusterCount, elapsed()));
|
|
344
|
-
for (const [index, key] of
|
|
345
|
-
|
|
392
|
+
for (const [index, key] of clusterKeys.entries()) {
|
|
393
|
+
const row = { id: ids[index] ?? index, clusterKey: key };
|
|
394
|
+
options.stdout.write(`${JSON.stringify(row)}\n`);
|
|
395
|
+
}
|
|
396
|
+
if (args.clusterReasonsFile && reasonsByClusterKey) {
|
|
397
|
+
try {
|
|
398
|
+
await writeFile(args.clusterReasonsFile, JSON.stringify(Object.fromEntries(reasonsByClusterKey), null, 2));
|
|
399
|
+
}
|
|
400
|
+
catch (error) {
|
|
401
|
+
renderProgress(lanes, useTty, errorLine(error.message));
|
|
402
|
+
return 1;
|
|
403
|
+
}
|
|
346
404
|
}
|
|
347
405
|
return 0;
|
|
348
406
|
}
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The evidence behind one Pass-0 blocking key — which signal
|
|
3
|
+
* {@link ./resolve-blocking-group-keys.js | resolveBlockingGroupKeys} (and,
|
|
4
|
+
* for the `orphanMerge` variant, {@link ./reassign-orphan-block-keys.js |
|
|
5
|
+
* reassignOrphanBlockKeys}) actually used to decide it, carried verbatim with
|
|
6
|
+
* no added interpretation. Every page sharing a `css:<hash>` blocking key
|
|
7
|
+
* shares the exact same `distinctiveStylesheetHrefs` set by construction (the
|
|
8
|
+
* hash is derived from that set), so one `BlockingReason` per distinct
|
|
9
|
+
* blocking key is enough — it does not need to vary per page.
|
|
10
|
+
*/
|
|
11
|
+
export type BlockingReason = {
|
|
12
|
+
readonly kind: 'css';
|
|
13
|
+
/**
|
|
14
|
+
* The sorted, deduplicated, first-party stylesheet hrefs left after
|
|
15
|
+
* corpus-wide chrome removal — the exact set
|
|
16
|
+
* {@link ./derive-stylesheet-group-key.js | deriveStylesheetGroupKey}
|
|
17
|
+
* hashed into this blocking key.
|
|
18
|
+
*/
|
|
19
|
+
readonly distinctiveStylesheetHrefs: readonly string[];
|
|
20
|
+
} | {
|
|
21
|
+
readonly kind: 'path';
|
|
22
|
+
/** The `derivePathGroupKey` result this blocking key was derived from. */
|
|
23
|
+
readonly pathKey: string;
|
|
24
|
+
} | {
|
|
25
|
+
readonly kind: 'orphanMerge';
|
|
26
|
+
/**
|
|
27
|
+
* The confined `path:` key this stylesheet-less page was folded into a
|
|
28
|
+
* same-section `css:` block under — see
|
|
29
|
+
* {@link ./reassign-orphan-block-keys.js | reassignOrphanBlockKeys}.
|
|
30
|
+
*/
|
|
31
|
+
readonly pathKey: string;
|
|
32
|
+
};
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
export {};
|
|
@@ -11,16 +11,50 @@
|
|
|
11
11
|
* have no implicit landmark role under HTML-AAM unless given an accessible
|
|
12
12
|
* name). `search` is matched via both the `<search>` element (WHATWG
|
|
13
13
|
* landmark shorthand) and `role="search"`.
|
|
14
|
+
*
|
|
15
|
+
* `main` is deliberately not a member of this union even though
|
|
16
|
+
* {@link ./extract-landmarks.js | extractLandmarks} reports it: this type is
|
|
17
|
+
* also the parameter type of `resolveLandmarkVariantKeys` and the vocabulary
|
|
18
|
+
* that chrome discovery (`computePerPageLandmarkInstances`'s
|
|
19
|
+
* `ALL_LANDMARK_TYPES`) iterates over. Admitting `'main'` here would let
|
|
20
|
+
* `resolveLandmarkVariantKeys(pages, 'main')` type-check while silently
|
|
21
|
+
* returning nothing (chrome discovery never looks at `main` instances) —
|
|
22
|
+
* exactly the kind of type-level lie this module avoids elsewhere. `main` is
|
|
23
|
+
* content, not chrome: it never participates in frequency-based chrome
|
|
24
|
+
* discovery, only in position reporting.
|
|
14
25
|
*/
|
|
15
26
|
export type LandmarkType = 'header' | 'footer' | 'nav' | 'aside' | 'form' | 'search';
|
|
27
|
+
/**
|
|
28
|
+
* An instance's location within the HTML string it was extracted from, in
|
|
29
|
+
* both string-index and 1-based line/column form. Computed once per page by
|
|
30
|
+
* {@link ./extract-landmarks.js | extractLandmarks} via
|
|
31
|
+
* {@link ./offset-to-line-column.js | buildLineColumnIndex}/`offsetToLineColumn`
|
|
32
|
+
* and carried downstream as plain numbers — nothing recomputes it.
|
|
33
|
+
*/
|
|
34
|
+
export type LandmarkPosition = {
|
|
35
|
+
readonly startOffset: number;
|
|
36
|
+
readonly endOffset: number;
|
|
37
|
+
readonly startLine: number;
|
|
38
|
+
readonly startColumn: number;
|
|
39
|
+
readonly endLine: number;
|
|
40
|
+
readonly endColumn: number;
|
|
41
|
+
};
|
|
42
|
+
/**
|
|
43
|
+
* One landmark instance: its raw HTML plus its {@link LandmarkPosition}
|
|
44
|
+
* within the page it was extracted from.
|
|
45
|
+
*/
|
|
46
|
+
export type LandmarkInstance = LandmarkPosition & {
|
|
47
|
+
readonly html: string;
|
|
48
|
+
};
|
|
16
49
|
/**
|
|
17
50
|
* Result of {@link ./extract-landmarks.js | extractLandmarks}. Each landmark
|
|
18
|
-
* field holds an array of
|
|
19
|
-
*
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
23
|
-
* span excised
|
|
51
|
+
* field holds an array of every genuinely-closed instance of that region on
|
|
52
|
+
* the page, in document order. Empty array if the page has none — or if
|
|
53
|
+
* every candidate found was malformed markup `extractLandmarks` declined to
|
|
54
|
+
* trust (see its JSDoc's note on discarded candidates). `remainderHtml` is
|
|
55
|
+
* the original HTML with every extracted `header`/`footer`/`nav`/`aside`/
|
|
56
|
+
* `form`/`search` span excised (`main` is never excised — see
|
|
57
|
+
* `extractLandmarks`'s "main handling" note), meant to be fed straight into
|
|
24
58
|
* {@link ./tokenize.js | tokenize} as the page's content-only signal.
|
|
25
59
|
*
|
|
26
60
|
* Multiple instances per type are the norm, not the exception: real crawl
|
|
@@ -33,12 +67,13 @@ export type LandmarkType = 'header' | 'footer' | 'nav' | 'aside' | 'form' | 'sea
|
|
|
33
67
|
* depth or ordering rule.
|
|
34
68
|
*/
|
|
35
69
|
export type ExtractLandmarksResult = {
|
|
36
|
-
header:
|
|
37
|
-
footer:
|
|
38
|
-
nav:
|
|
39
|
-
aside:
|
|
40
|
-
form:
|
|
41
|
-
search:
|
|
70
|
+
header: LandmarkInstance[];
|
|
71
|
+
footer: LandmarkInstance[];
|
|
72
|
+
nav: LandmarkInstance[];
|
|
73
|
+
aside: LandmarkInstance[];
|
|
74
|
+
form: LandmarkInstance[];
|
|
75
|
+
search: LandmarkInstance[];
|
|
76
|
+
main: LandmarkInstance[];
|
|
42
77
|
remainderHtml: string;
|
|
43
78
|
};
|
|
44
79
|
/**
|
|
@@ -113,14 +148,40 @@ export type ExtractLandmarksResult = {
|
|
|
113
148
|
* segment then disappears from the surviving paths, shortening them by one
|
|
114
149
|
* level. This is inherent to "delete the matched span, use whatever's
|
|
115
150
|
* left" and is not treated as a bug.
|
|
151
|
+
*
|
|
152
|
+
* ## Main handling
|
|
153
|
+
*
|
|
154
|
+
* `main` (the `<main>` tag or `role="main"`) is collected the same way as
|
|
155
|
+
* the other six types — one entry per genuinely-closed instance, in
|
|
156
|
+
* document order — but is kept out of every mechanism the other six feed:
|
|
157
|
+
*
|
|
158
|
+
* - It is **never excised**: its span is never added to `remainderHtml`'s
|
|
159
|
+
* excise list, because `main` is the page's actual content, not chrome.
|
|
160
|
+
* Removing it would gut `remainderHtml` down to whatever sits outside
|
|
161
|
+
* `<main>` (nothing, on most real pages).
|
|
162
|
+
* - Its `keepOutermost` nesting sweep runs **separately** from the other six
|
|
163
|
+
* types'. If it shared the sweep, a `<main>` that wraps most of the page
|
|
164
|
+
* (as it typically does) would make every `header`/`nav`/`aside` nested
|
|
165
|
+
* inside it look "contained by main" and get dropped — destroying the
|
|
166
|
+
* section-local chrome detection this module exists to enable (see "Why
|
|
167
|
+
* collect every instance" above). Only nested `<main>`s (an edge case —
|
|
168
|
+
* HTML discourages more than one) are deduplicated against each other.
|
|
169
|
+
* - It never contributes to chrome/shell-frequency analysis (`main` is
|
|
170
|
+
* absent from `computePerPageLandmarkInstances`'s `ALL_LANDMARK_TYPES`):
|
|
171
|
+
* its instances are reported for position purposes only, never treated as
|
|
172
|
+
* candidate chrome.
|
|
116
173
|
* @param html
|
|
117
174
|
* @example
|
|
118
175
|
* ```ts
|
|
119
176
|
* extractLandmarks('<body><header>H</header><main>M</main><footer>F</footer></body>');
|
|
120
177
|
* // {
|
|
121
|
-
* // header: ['<header>H</header>'
|
|
122
|
-
* //
|
|
178
|
+
* // header: [{ html: '<header>H</header>', startOffset: 6, endOffset: 24,
|
|
179
|
+
* // startLine: 1, startColumn: 7, endLine: 1, endColumn: 25 }],
|
|
180
|
+
* // footer: [{ html: '<footer>F</footer>', startOffset: 38, endOffset: 56,
|
|
181
|
+
* // startLine: 1, startColumn: 39, endLine: 1, endColumn: 57 }],
|
|
123
182
|
* // nav: [], aside: [], form: [], search: [],
|
|
183
|
+
* // main: [{ html: '<main>M</main>', startOffset: 24, endOffset: 38,
|
|
184
|
+
* // startLine: 1, startColumn: 25, endLine: 1, endColumn: 39 }],
|
|
124
185
|
* // remainderHtml: '<body><main>M</main></body>',
|
|
125
186
|
* // }
|
|
126
187
|
* ```
|