@d-zero/page-cluster 0.3.1 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +47 -7
- package/dist/build-page-landmark-report.d.ts +50 -0
- package/dist/build-page-landmark-report.js +67 -0
- package/dist/cli.d.ts +1 -0
- package/dist/cli.js +56 -6
- package/dist/extract-landmarks.d.ts +75 -14
- package/dist/extract-landmarks.js +64 -4
- package/dist/is-chrome-landmark-instance.d.ts +41 -0
- package/dist/is-chrome-landmark-instance.js +50 -0
- package/dist/merge-cross-block-clusters.js +5 -97
- package/dist/offset-to-line-column.d.ts +38 -0
- package/dist/offset-to-line-column.js +49 -0
- package/dist/per-page-landmark-signatures.d.ts +10 -1
- package/dist/per-page-landmark-signatures.js +16 -4
- package/dist/resolve-page-cluster-keys.d.ts +62 -1
- package/dist/resolve-page-cluster-keys.js +77 -84
- package/dist/shell-quorum.d.ts +70 -0
- package/dist/shell-quorum.js +110 -0
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -15,7 +15,7 @@ yarn add @d-zero/page-cluster
|
|
|
15
15
|
### CLI
|
|
16
16
|
|
|
17
17
|
```sh
|
|
18
|
-
page-cluster [--content-block-attribute <name>] < pages.jsonl > clusters.jsonl
|
|
18
|
+
page-cluster [--content-block-attribute <name>] [--include-landmark-positions] < pages.jsonl > clusters.jsonl
|
|
19
19
|
```
|
|
20
20
|
|
|
21
21
|
**入力**: JSONL 1 行 1 ページ。フィールドは以下。`html` 以外はすべて任意（`paths` / `stylesheetHrefs` がないと粗い分類になる）。
|
|
@@ -36,6 +36,45 @@ page-cluster [--content-block-attribute <name>] < pages.jsonl > clusters.jsonl
|
|
|
36
36
|
{ "id": "任意の識別子", "clusterKey": "..." }
|
|
37
37
|
```
|
|
38
38
|
|
|
39
|
+
`--include-landmark-positions` を指定すると、各行に `landmarks` フィールドが追加される。header / footer / nav / aside / form / search / main のインスタンスごとに、HTML 内の位置（1-based の line/column と文字列オフセットの両方）を返す。header 〜 search の 6 種は追加で、そのページが属する最終クラスタ内での頻度分析（`shellQuorum`）に基づく `isChrome`（サイト/セクション共通の chrome か、ページ固有のコンテンツか）を持つ。`main` は常にコンテンツなので `isChrome` を持たない。
|
|
40
|
+
|
|
41
|
+
```json
|
|
42
|
+
{
|
|
43
|
+
"id": "任意の識別子",
|
|
44
|
+
"clusterKey": "...",
|
|
45
|
+
"landmarks": {
|
|
46
|
+
"header": [
|
|
47
|
+
{
|
|
48
|
+
"startLine": 1,
|
|
49
|
+
"startColumn": 7,
|
|
50
|
+
"endLine": 1,
|
|
51
|
+
"endColumn": 30,
|
|
52
|
+
"startOffset": 6,
|
|
53
|
+
"endOffset": 29,
|
|
54
|
+
"isChrome": true
|
|
55
|
+
}
|
|
56
|
+
],
|
|
57
|
+
"footer": [],
|
|
58
|
+
"nav": [],
|
|
59
|
+
"aside": [],
|
|
60
|
+
"form": [],
|
|
61
|
+
"search": [],
|
|
62
|
+
"main": [
|
|
63
|
+
{
|
|
64
|
+
"startLine": 2,
|
|
65
|
+
"startColumn": 1,
|
|
66
|
+
"endLine": 10,
|
|
67
|
+
"endColumn": 8,
|
|
68
|
+
"startOffset": 40,
|
|
69
|
+
"endOffset": 120
|
|
70
|
+
}
|
|
71
|
+
]
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
20,000 ページを超えるコーパス（ストリーミング経路）では `--include-landmark-positions` は使えない（エラーで終了する）。ストリーミング経路はリザーバサンプリングと近似割当を使うため、ページ単位の chrome 判定に必要な「そのページが属する最終クラスタの shell トークン」という概念を持たないため。
|
|
77
|
+
|
|
39
78
|
クローラ出力が JSON 配列の場合は `jq` で line-delimited に変換して食わせる:
|
|
40
79
|
|
|
41
80
|
```sh
|
|
@@ -45,6 +84,7 @@ jq -c '.[]' crawl-output.json | page-cluster > clusters.jsonl
|
|
|
45
84
|
#### オプション
|
|
46
85
|
|
|
47
86
|
- `--content-block-attribute <name>` — CMS が自由編集コンテンツブロックに付与している属性名（例: `data-bgb`）が分かっている場合に指定する。指定すると比較前にその属性を持つ要素配下を無視するので、同じテンプレートで本文構成だけ違うページを混同しなくなる。唯一の site-specific なオプションで、未指定でも `<main>` / `role="main"` を起点にした自動深さキャップが常時働く（詳細は `resolve-page-cluster-keys.ts` の JSDoc を参照）
|
|
87
|
+
- `--include-landmark-positions` — 出力の各行に上記の `landmarks` フィールドを追加する。20,000 ページ超のコーパスでは使えない。指定すると進捗表示（後述）は出なくなる（進捗を出さない非ストリーミング経路に常に振り分けられるため）
|
|
48
88
|
- `--help` / `-h` — ヘルプを表示する
|
|
49
89
|
- `--version` / `-v` — バージョンを表示する
|
|
50
90
|
|
|
@@ -76,12 +116,12 @@ silence したい場合は `2>/dev/null`。ログに残したい場合は `2> pr
|
|
|
76
116
|
|
|
77
117
|
サブパスエクスポート構成。import パスと提供関数の対応は以下。
|
|
78
118
|
|
|
79
|
-
| import パス | 提供関数
|
|
80
|
-
| ---------------------------------------------------- |
|
|
81
|
-
| `@d-zero/page-cluster` | `tokenize` — `<body>` 配下を構造トークン列に変換する低レベルプリミティブ
|
|
82
|
-
| `@d-zero/page-cluster/resolve-page-cluster-keys` | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array
|
|
83
|
-
| `@d-zero/page-cluster/extract-landmarks` | `extractLandmarks` —
|
|
84
|
-
| `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類
|
|
119
|
+
| import パス | 提供関数 |
|
|
120
|
+
| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
121
|
+
| `@d-zero/page-cluster` | `tokenize` — `<body>` 配下を構造トークン列に変換する低レベルプリミティブ |
|
|
122
|
+
| `@d-zero/page-cluster/resolve-page-cluster-keys` | `resolvePageClusterKeys`（非同期・ファクトリ入力・メモリ有界のメインエントリー）、`resolvePageClusterKeysFromArray`（array 入力ラッパー）、`resolvePageClusterKeysInMemory`（同期・array 入力）。いずれも `includeLandmarkPositions: true` を渡すと `clusterKey` に加えて位置情報つきの `landmarks`（`PageLandmarkReport`）を返す |
|
|
123
|
+
| `@d-zero/page-cluster/extract-landmarks` | `extractLandmarks` — header / footer / nav / aside / form / search / main の 7 種を抽出し、インスタンスごとの生 HTML と HTML 内の位置（line/column・文字列オフセット）を返す |
|
|
124
|
+
| `@d-zero/page-cluster/resolve-landmark-variant-keys` | `resolveLandmarkVariantKeys` — 特定ランドマークのデザインバリアントでページを分類 |
|
|
85
125
|
|
|
86
126
|
```ts
|
|
87
127
|
import { resolvePageClusterKeysFromArray } from '@d-zero/page-cluster/resolve-page-cluster-keys';
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
import type { ExtractLandmarksResult, LandmarkPosition } from './extract-landmarks.js';
|
|
2
|
+
import type { TokenizeOptions } from './types.js';
|
|
3
|
+
/**
|
|
4
|
+
* A landmark instance's position plus whether
|
|
5
|
+
* {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance} judged
|
|
6
|
+
* it shared site/section chrome (`true`) or page-specific content (`false`),
|
|
7
|
+
* against the unit's {@link ./shell-quorum.js | shellQuorum} shell tokens.
|
|
8
|
+
*/
|
|
9
|
+
export type ReportedLandmarkInstance = LandmarkPosition & {
|
|
10
|
+
readonly isChrome: boolean;
|
|
11
|
+
};
|
|
12
|
+
/**
|
|
13
|
+
* Per-page landmark position report built by
|
|
14
|
+
* {@link ./build-page-landmark-report.js | buildPageLandmarkReport}. `main`
|
|
15
|
+
* carries no `isChrome` verdict — it never participates in chrome/shell
|
|
16
|
+
* discovery (see `extractLandmarks`'s "main handling" note) and is always
|
|
17
|
+
* content.
|
|
18
|
+
*/
|
|
19
|
+
export type PageLandmarkReport = {
|
|
20
|
+
header: ReportedLandmarkInstance[];
|
|
21
|
+
footer: ReportedLandmarkInstance[];
|
|
22
|
+
nav: ReportedLandmarkInstance[];
|
|
23
|
+
aside: ReportedLandmarkInstance[];
|
|
24
|
+
form: ReportedLandmarkInstance[];
|
|
25
|
+
search: ReportedLandmarkInstance[];
|
|
26
|
+
main: LandmarkPosition[];
|
|
27
|
+
};
|
|
28
|
+
/**
|
|
29
|
+
* Builds a page's landmark position report: every landmark instance's
|
|
30
|
+
* location, with `header`/`footer`/`nav`/`aside`/`form`/`search` instances
|
|
31
|
+
* additionally classified as chrome or content against `shellTokens`.
|
|
32
|
+
*
|
|
33
|
+
* Reads `landmarks` directly — the full, non-deduplicated instance list
|
|
34
|
+
* `extractLandmarks` produced — rather than going through
|
|
35
|
+
* {@link ./per-page-landmark-signatures.js | computePerPageLandmarkInstances}'s
|
|
36
|
+
* per-page-deduplicated `PerPageLandmarkInstance[]`: that dedupe collapses
|
|
37
|
+
* same-signature instances to one entry, which would silently drop the
|
|
38
|
+
* position of every duplicate instance a position report needs to include.
|
|
39
|
+
* @param landmarks
|
|
40
|
+
* @param shellTokens The unit-level shell token set from
|
|
41
|
+
* {@link ./shell-quorum.js | shellQuorum}, computed once per final cluster
|
|
42
|
+
* and shared across every member page's report.
|
|
43
|
+
* @param tokenizeOptions
|
|
44
|
+
* @example
|
|
45
|
+
* ```ts
|
|
46
|
+
* const shellTokens = shellQuorum(clusterPerPageInstances);
|
|
47
|
+
* const report = buildPageLandmarkReport(extractLandmarks(page.html), shellTokens);
|
|
48
|
+
* ```
|
|
49
|
+
*/
|
|
50
|
+
export declare function buildPageLandmarkReport(landmarks: ExtractLandmarksResult, shellTokens: ReadonlySet<string>, tokenizeOptions?: TokenizeOptions): PageLandmarkReport;
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
import { isChromeLandmarkInstance } from './is-chrome-landmark-instance.js';
|
|
2
|
+
import { ALL_LANDMARK_TYPES } from './per-page-landmark-signatures.js';
|
|
3
|
+
import { tokenize } from './tokenize.js';
|
|
4
|
+
/**
|
|
5
|
+
* Strips `html` off a {@link LandmarkInstance}, keeping only its position.
|
|
6
|
+
* `buildPageLandmarkReport`'s output is meant to be serialized per page
|
|
7
|
+
* across a whole corpus (the CLI's JSONL output), so the report
|
|
8
|
+
* deliberately excludes each instance's raw HTML to keep that payload from
|
|
9
|
+
* scaling with markup size — callers who also need the HTML already have
|
|
10
|
+
* `ExtractLandmarksResult` in hand.
|
|
11
|
+
* @param instance
|
|
12
|
+
*/
|
|
13
|
+
function toPosition(instance) {
|
|
14
|
+
return {
|
|
15
|
+
startOffset: instance.startOffset,
|
|
16
|
+
endOffset: instance.endOffset,
|
|
17
|
+
startLine: instance.startLine,
|
|
18
|
+
startColumn: instance.startColumn,
|
|
19
|
+
endLine: instance.endLine,
|
|
20
|
+
endColumn: instance.endColumn,
|
|
21
|
+
};
|
|
22
|
+
}
|
|
23
|
+
/**
|
|
24
|
+
* Builds a page's landmark position report: every landmark instance's
|
|
25
|
+
* location, with `header`/`footer`/`nav`/`aside`/`form`/`search` instances
|
|
26
|
+
* additionally classified as chrome or content against `shellTokens`.
|
|
27
|
+
*
|
|
28
|
+
* Reads `landmarks` directly — the full, non-deduplicated instance list
|
|
29
|
+
* `extractLandmarks` produced — rather than going through
|
|
30
|
+
* {@link ./per-page-landmark-signatures.js | computePerPageLandmarkInstances}'s
|
|
31
|
+
* per-page-deduplicated `PerPageLandmarkInstance[]`: that dedupe collapses
|
|
32
|
+
* same-signature instances to one entry, which would silently drop the
|
|
33
|
+
* position of every duplicate instance a position report needs to include.
|
|
34
|
+
* @param landmarks
|
|
35
|
+
* @param shellTokens The unit-level shell token set from
|
|
36
|
+
* {@link ./shell-quorum.js | shellQuorum}, computed once per final cluster
|
|
37
|
+
* and shared across every member page's report.
|
|
38
|
+
* @param tokenizeOptions
|
|
39
|
+
* @example
|
|
40
|
+
* ```ts
|
|
41
|
+
* const shellTokens = shellQuorum(clusterPerPageInstances);
|
|
42
|
+
* const report = buildPageLandmarkReport(extractLandmarks(page.html), shellTokens);
|
|
43
|
+
* ```
|
|
44
|
+
*/
|
|
45
|
+
export function buildPageLandmarkReport(landmarks, shellTokens, tokenizeOptions) {
|
|
46
|
+
const report = {
|
|
47
|
+
header: [],
|
|
48
|
+
footer: [],
|
|
49
|
+
nav: [],
|
|
50
|
+
aside: [],
|
|
51
|
+
form: [],
|
|
52
|
+
search: [],
|
|
53
|
+
main: landmarks.main.map(toPosition),
|
|
54
|
+
};
|
|
55
|
+
for (const type of ALL_LANDMARK_TYPES) {
|
|
56
|
+
for (const instance of landmarks[type]) {
|
|
57
|
+
const tokens = instance.html
|
|
58
|
+
? new Set(tokenize(`<body>${instance.html}</body>`, tokenizeOptions).tokens)
|
|
59
|
+
: new Set();
|
|
60
|
+
report[type].push({
|
|
61
|
+
...toPosition(instance),
|
|
62
|
+
isChrome: isChromeLandmarkInstance(tokens, shellTokens),
|
|
63
|
+
});
|
|
64
|
+
}
|
|
65
|
+
}
|
|
66
|
+
return report;
|
|
67
|
+
}
|
package/dist/cli.d.ts
CHANGED
package/dist/cli.js
CHANGED
|
@@ -8,7 +8,7 @@ import process from 'node:process';
|
|
|
8
8
|
import { Lanes } from '@d-zero/dealer';
|
|
9
9
|
import { resolvePageClusterKeys } from './resolve-page-cluster-keys.js';
|
|
10
10
|
const HELP_TEXT = `Usage:
|
|
11
|
-
page-cluster [--content-block-attribute <name>] < pages.jsonl > clusters.jsonl
|
|
11
|
+
page-cluster [--content-block-attribute <name>] [--include-landmark-positions] < pages.jsonl > clusters.jsonl
|
|
12
12
|
|
|
13
13
|
Input (JSONL, one page per line):
|
|
14
14
|
{
|
|
@@ -22,10 +22,35 @@ Input (JSONL, one page per line):
|
|
|
22
22
|
Output (JSONL, one line per input page, in input order):
|
|
23
23
|
{ "id": "...", "clusterKey": "..." }
|
|
24
24
|
|
|
25
|
+
With --include-landmark-positions, each line additionally carries a
|
|
26
|
+
\`landmarks\` field: every header/footer/nav/aside/form/search/main
|
|
27
|
+
instance's position (1-based line/column plus string offsets), with the
|
|
28
|
+
six excisable types (all but main) also carrying an \`isChrome\` verdict
|
|
29
|
+
against that page's final cluster:
|
|
30
|
+
{
|
|
31
|
+
"id": "...", "clusterKey": "...",
|
|
32
|
+
"landmarks": {
|
|
33
|
+
"header": [{ "startLine": 1, "startColumn": 7, "endLine": 1,
|
|
34
|
+
"endColumn": 30, "startOffset": 6, "endOffset": 29,
|
|
35
|
+
"isChrome": true }],
|
|
36
|
+
"footer": [...], "nav": [...], "aside": [...], "form": [...], "search": [...],
|
|
37
|
+
"main": [{ "startLine": 2, "startColumn": 1, "endLine": 10,
|
|
38
|
+
"endColumn": 8, "startOffset": 40, "endOffset": 120 }]
|
|
39
|
+
}
|
|
40
|
+
}
|
|
41
|
+
|
|
25
42
|
Options:
|
|
26
43
|
--content-block-attribute <name> CMS-provided attribute marking freeform
|
|
27
44
|
content blocks that should be stripped
|
|
28
45
|
before comparison (e.g. \`data-bgb\`).
|
|
46
|
+
--include-landmark-positions Add the \`landmarks\` field described
|
|
47
|
+
above to every output line. Not
|
|
48
|
+
supported for corpora over 20,000 pages
|
|
49
|
+
(throws instead of streaming). Disables
|
|
50
|
+
progress output on stderr — this option
|
|
51
|
+
always routes through the same
|
|
52
|
+
non-progress-emitting code path as a
|
|
53
|
+
run without progress.
|
|
29
54
|
--help Print this help and exit.
|
|
30
55
|
--version Print the package version and exit.
|
|
31
56
|
|
|
@@ -82,6 +107,10 @@ export function parseArgs(argv) {
|
|
|
82
107
|
i++;
|
|
83
108
|
break;
|
|
84
109
|
}
|
|
110
|
+
case '--include-landmark-positions': {
|
|
111
|
+
out.includeLandmarkPositions = true;
|
|
112
|
+
break;
|
|
113
|
+
}
|
|
85
114
|
default: {
|
|
86
115
|
out.unknownFlag = arg;
|
|
87
116
|
return out;
|
|
@@ -331,18 +360,39 @@ export async function runCli(options) {
|
|
|
331
360
|
renderProgress(lanes, useTty, formatProgressLine(event, elapsed()));
|
|
332
361
|
},
|
|
333
362
|
};
|
|
334
|
-
|
|
363
|
+
// `includeLandmarkPositions` always routes resolvePageClusterKeys
|
|
364
|
+
// through its non-progress-emitting sync path (see that option's own
|
|
365
|
+
// JSDoc), so the onProgress callback above is set but never invoked
|
|
366
|
+
// in this branch — no separate "quiet" resolveOptions variant needed.
|
|
367
|
+
let clusterKeys;
|
|
368
|
+
let landmarksByIndex;
|
|
335
369
|
try {
|
|
336
|
-
|
|
370
|
+
if (args.includeLandmarkPositions) {
|
|
371
|
+
const results = await resolvePageClusterKeys(() => pages, {
|
|
372
|
+
...resolveOptions,
|
|
373
|
+
includeLandmarkPositions: true,
|
|
374
|
+
});
|
|
375
|
+
clusterKeys = results.map((r) => r.clusterKey);
|
|
376
|
+
landmarksByIndex = results.map((r) => r.landmarks);
|
|
377
|
+
}
|
|
378
|
+
else {
|
|
379
|
+
clusterKeys = await resolvePageClusterKeys(() => pages, resolveOptions);
|
|
380
|
+
}
|
|
337
381
|
}
|
|
338
382
|
catch (error) {
|
|
339
383
|
renderProgress(lanes, useTty, errorLine(error.message));
|
|
340
384
|
return 1;
|
|
341
385
|
}
|
|
342
|
-
const clusterCount = new Set(
|
|
386
|
+
const clusterCount = new Set(clusterKeys).size;
|
|
343
387
|
renderProgress(lanes, useTty, doneLine(pages.length, clusterCount, elapsed()));
|
|
344
|
-
for (const [index, key] of
|
|
345
|
-
|
|
388
|
+
for (const [index, key] of clusterKeys.entries()) {
|
|
389
|
+
const row = {
|
|
390
|
+
id: ids[index] ?? index,
|
|
391
|
+
clusterKey: key,
|
|
392
|
+
};
|
|
393
|
+
if (landmarksByIndex)
|
|
394
|
+
row.landmarks = landmarksByIndex[index];
|
|
395
|
+
options.stdout.write(`${JSON.stringify(row)}\n`);
|
|
346
396
|
}
|
|
347
397
|
return 0;
|
|
348
398
|
}
|
|
@@ -11,16 +11,50 @@
|
|
|
11
11
|
* have no implicit landmark role under HTML-AAM unless given an accessible
|
|
12
12
|
* name). `search` is matched via both the `<search>` element (WHATWG
|
|
13
13
|
* landmark shorthand) and `role="search"`.
|
|
14
|
+
*
|
|
15
|
+
* `main` is deliberately not a member of this union even though
|
|
16
|
+
* {@link ./extract-landmarks.js | extractLandmarks} reports it: this type is
|
|
17
|
+
* also the parameter type of `resolveLandmarkVariantKeys` and the vocabulary
|
|
18
|
+
* that chrome discovery (`computePerPageLandmarkInstances`'s
|
|
19
|
+
* `ALL_LANDMARK_TYPES`) iterates over. Admitting `'main'` here would let
|
|
20
|
+
* `resolveLandmarkVariantKeys(pages, 'main')` type-check while silently
|
|
21
|
+
* returning nothing (chrome discovery never looks at `main` instances) —
|
|
22
|
+
* exactly the kind of type-level lie this module avoids elsewhere. `main` is
|
|
23
|
+
* content, not chrome: it never participates in frequency-based chrome
|
|
24
|
+
* discovery, only in position reporting.
|
|
14
25
|
*/
|
|
15
26
|
export type LandmarkType = 'header' | 'footer' | 'nav' | 'aside' | 'form' | 'search';
|
|
27
|
+
/**
|
|
28
|
+
* An instance's location within the HTML string it was extracted from, in
|
|
29
|
+
* both string-index and 1-based line/column form. Computed once per page by
|
|
30
|
+
* {@link ./extract-landmarks.js | extractLandmarks} via
|
|
31
|
+
* {@link ./offset-to-line-column.js | buildLineColumnIndex}/`offsetToLineColumn`
|
|
32
|
+
* and carried downstream as plain numbers — nothing recomputes it.
|
|
33
|
+
*/
|
|
34
|
+
export type LandmarkPosition = {
|
|
35
|
+
readonly startOffset: number;
|
|
36
|
+
readonly endOffset: number;
|
|
37
|
+
readonly startLine: number;
|
|
38
|
+
readonly startColumn: number;
|
|
39
|
+
readonly endLine: number;
|
|
40
|
+
readonly endColumn: number;
|
|
41
|
+
};
|
|
42
|
+
/**
|
|
43
|
+
* One landmark instance: its raw HTML plus its {@link LandmarkPosition}
|
|
44
|
+
* within the page it was extracted from.
|
|
45
|
+
*/
|
|
46
|
+
export type LandmarkInstance = LandmarkPosition & {
|
|
47
|
+
readonly html: string;
|
|
48
|
+
};
|
|
16
49
|
/**
|
|
17
50
|
* Result of {@link ./extract-landmarks.js | extractLandmarks}. Each landmark
|
|
18
|
-
* field holds an array of
|
|
19
|
-
*
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
23
|
-
* span excised
|
|
51
|
+
* field holds an array of every genuinely-closed instance of that region on
|
|
52
|
+
* the page, in document order. Empty array if the page has none — or if
|
|
53
|
+
* every candidate found was malformed markup `extractLandmarks` declined to
|
|
54
|
+
* trust (see its JSDoc's note on discarded candidates). `remainderHtml` is
|
|
55
|
+
* the original HTML with every extracted `header`/`footer`/`nav`/`aside`/
|
|
56
|
+
* `form`/`search` span excised (`main` is never excised — see
|
|
57
|
+
* `extractLandmarks`'s "main handling" note), meant to be fed straight into
|
|
24
58
|
* {@link ./tokenize.js | tokenize} as the page's content-only signal.
|
|
25
59
|
*
|
|
26
60
|
* Multiple instances per type are the norm, not the exception: real crawl
|
|
@@ -33,12 +67,13 @@ export type LandmarkType = 'header' | 'footer' | 'nav' | 'aside' | 'form' | 'sea
|
|
|
33
67
|
* depth or ordering rule.
|
|
34
68
|
*/
|
|
35
69
|
export type ExtractLandmarksResult = {
|
|
36
|
-
header:
|
|
37
|
-
footer:
|
|
38
|
-
nav:
|
|
39
|
-
aside:
|
|
40
|
-
form:
|
|
41
|
-
search:
|
|
70
|
+
header: LandmarkInstance[];
|
|
71
|
+
footer: LandmarkInstance[];
|
|
72
|
+
nav: LandmarkInstance[];
|
|
73
|
+
aside: LandmarkInstance[];
|
|
74
|
+
form: LandmarkInstance[];
|
|
75
|
+
search: LandmarkInstance[];
|
|
76
|
+
main: LandmarkInstance[];
|
|
42
77
|
remainderHtml: string;
|
|
43
78
|
};
|
|
44
79
|
/**
|
|
@@ -113,14 +148,40 @@ export type ExtractLandmarksResult = {
|
|
|
113
148
|
* segment then disappears from the surviving paths, shortening them by one
|
|
114
149
|
* level. This is inherent to "delete the matched span, use whatever's
|
|
115
150
|
* left" and is not treated as a bug.
|
|
151
|
+
*
|
|
152
|
+
* ## Main handling
|
|
153
|
+
*
|
|
154
|
+
* `main` (the `<main>` tag or `role="main"`) is collected the same way as
|
|
155
|
+
* the other six types — one entry per genuinely-closed instance, in
|
|
156
|
+
* document order — but is kept out of every mechanism the other six feed:
|
|
157
|
+
*
|
|
158
|
+
* - It is **never excised**: its span is never added to `remainderHtml`'s
|
|
159
|
+
* excise list, because `main` is the page's actual content, not chrome.
|
|
160
|
+
* Removing it would gut `remainderHtml` down to whatever sits outside
|
|
161
|
+
* `<main>` (nothing, on most real pages).
|
|
162
|
+
* - Its `keepOutermost` nesting sweep runs **separately** from the other six
|
|
163
|
+
* types'. If it shared the sweep, a `<main>` that wraps most of the page
|
|
164
|
+
* (as it typically does) would make every `header`/`nav`/`aside` nested
|
|
165
|
+
* inside it look "contained by main" and get dropped — destroying the
|
|
166
|
+
* section-local chrome detection this module exists to enable (see "Why
|
|
167
|
+
* collect every instance" above). Only nested `<main>`s (an edge case —
|
|
168
|
+
* HTML discourages more than one) are deduplicated against each other.
|
|
169
|
+
* - It never contributes to chrome/shell-frequency analysis (`main` is
|
|
170
|
+
* absent from `computePerPageLandmarkInstances`'s `ALL_LANDMARK_TYPES`):
|
|
171
|
+
* its instances are reported for position purposes only, never treated as
|
|
172
|
+
* candidate chrome.
|
|
116
173
|
* @param html
|
|
117
174
|
* @example
|
|
118
175
|
* ```ts
|
|
119
176
|
* extractLandmarks('<body><header>H</header><main>M</main><footer>F</footer></body>');
|
|
120
177
|
* // {
|
|
121
|
-
* // header: ['<header>H</header>'
|
|
122
|
-
* //
|
|
178
|
+
* // header: [{ html: '<header>H</header>', startOffset: 6, endOffset: 24,
|
|
179
|
+
* // startLine: 1, startColumn: 7, endLine: 1, endColumn: 25 }],
|
|
180
|
+
* // footer: [{ html: '<footer>F</footer>', startOffset: 38, endOffset: 56,
|
|
181
|
+
* // startLine: 1, startColumn: 39, endLine: 1, endColumn: 57 }],
|
|
123
182
|
* // nav: [], aside: [], form: [], search: [],
|
|
183
|
+
* // main: [{ html: '<main>M</main>', startOffset: 24, endOffset: 38,
|
|
184
|
+
* // startLine: 1, startColumn: 25, endLine: 1, endColumn: 39 }],
|
|
124
185
|
* // remainderHtml: '<body><main>M</main></body>',
|
|
125
186
|
* // }
|
|
126
187
|
* ```
|
|
@@ -1,11 +1,13 @@
|
|
|
1
1
|
import { excise } from './excise.js';
|
|
2
2
|
import { findMatchingElements, } from './find-shallowest-elements.js';
|
|
3
|
+
import { buildLineColumnIndex, offsetToLineColumn } from './offset-to-line-column.js';
|
|
3
4
|
const TAG_TO_TYPE = {
|
|
4
5
|
header: 'header',
|
|
5
6
|
footer: 'footer',
|
|
6
7
|
nav: 'nav',
|
|
7
8
|
aside: 'aside',
|
|
8
9
|
search: 'search',
|
|
10
|
+
main: 'main',
|
|
9
11
|
};
|
|
10
12
|
const ROLE_TO_TYPE = {
|
|
11
13
|
banner: 'header',
|
|
@@ -14,6 +16,7 @@ const ROLE_TO_TYPE = {
|
|
|
14
16
|
complementary: 'aside',
|
|
15
17
|
form: 'form',
|
|
16
18
|
search: 'search',
|
|
19
|
+
main: 'main',
|
|
17
20
|
};
|
|
18
21
|
/**
|
|
19
22
|
* Determines which landmark type(s) an element matches by tag name or
|
|
@@ -40,6 +43,16 @@ function matchLandmarkTypes(tagName, role) {
|
|
|
40
43
|
}
|
|
41
44
|
return types;
|
|
42
45
|
}
|
|
46
|
+
/**
|
|
47
|
+
* Type guard splitting `findMatchingElements`' combined match list into the
|
|
48
|
+
* excisable six landmark types vs `main`. `main` is content, not chrome, and
|
|
49
|
+
* must never be mixed into the same `keepOutermost` sweep as the other six
|
|
50
|
+
* — see `extractLandmarks`'s "main handling" note for why.
|
|
51
|
+
* @param match
|
|
52
|
+
*/
|
|
53
|
+
function isMainMatch(match) {
|
|
54
|
+
return match.type === 'main';
|
|
55
|
+
}
|
|
43
56
|
/**
|
|
44
57
|
* Filters out any match whose whole-element span is strictly contained by
|
|
45
58
|
* another match's span, keeping only outermost instances. Runs across all
|
|
@@ -151,21 +164,50 @@ function keepOutermost(matches) {
|
|
|
151
164
|
* segment then disappears from the surviving paths, shortening them by one
|
|
152
165
|
* level. This is inherent to "delete the matched span, use whatever's
|
|
153
166
|
* left" and is not treated as a bug.
|
|
167
|
+
*
|
|
168
|
+
* ## Main handling
|
|
169
|
+
*
|
|
170
|
+
* `main` (the `<main>` tag or `role="main"`) is collected the same way as
|
|
171
|
+
* the other six types — one entry per genuinely-closed instance, in
|
|
172
|
+
* document order — but is kept out of every mechanism the other six feed:
|
|
173
|
+
*
|
|
174
|
+
* - It is **never excised**: its span is never added to `remainderHtml`'s
|
|
175
|
+
* excise list, because `main` is the page's actual content, not chrome.
|
|
176
|
+
* Removing it would gut `remainderHtml` down to whatever sits outside
|
|
177
|
+
* `<main>` (nothing, on most real pages).
|
|
178
|
+
* - Its `keepOutermost` nesting sweep runs **separately** from the other six
|
|
179
|
+
* types'. If it shared the sweep, a `<main>` that wraps most of the page
|
|
180
|
+
* (as it typically does) would make every `header`/`nav`/`aside` nested
|
|
181
|
+
* inside it look "contained by main" and get dropped — destroying the
|
|
182
|
+
* section-local chrome detection this module exists to enable (see "Why
|
|
183
|
+
* collect every instance" above). Only nested `<main>`s (an edge case —
|
|
184
|
+
* HTML discourages more than one) are deduplicated against each other.
|
|
185
|
+
* - It never contributes to chrome/shell-frequency analysis (`main` is
|
|
186
|
+
* absent from `computePerPageLandmarkInstances`'s `ALL_LANDMARK_TYPES`):
|
|
187
|
+
* its instances are reported for position purposes only, never treated as
|
|
188
|
+
* candidate chrome.
|
|
154
189
|
* @param html
|
|
155
190
|
* @example
|
|
156
191
|
* ```ts
|
|
157
192
|
* extractLandmarks('<body><header>H</header><main>M</main><footer>F</footer></body>');
|
|
158
193
|
* // {
|
|
159
|
-
* // header: ['<header>H</header>'
|
|
160
|
-
* //
|
|
194
|
+
* // header: [{ html: '<header>H</header>', startOffset: 6, endOffset: 24,
|
|
195
|
+
* // startLine: 1, startColumn: 7, endLine: 1, endColumn: 25 }],
|
|
196
|
+
* // footer: [{ html: '<footer>F</footer>', startOffset: 38, endOffset: 56,
|
|
197
|
+
* // startLine: 1, startColumn: 39, endLine: 1, endColumn: 57 }],
|
|
161
198
|
* // nav: [], aside: [], form: [], search: [],
|
|
199
|
+
* // main: [{ html: '<main>M</main>', startOffset: 24, endOffset: 38,
|
|
200
|
+
* // startLine: 1, startColumn: 25, endLine: 1, endColumn: 39 }],
|
|
162
201
|
* // remainderHtml: '<body><main>M</main></body>',
|
|
163
202
|
* // }
|
|
164
203
|
* ```
|
|
165
204
|
*/
|
|
166
205
|
export function extractLandmarks(html) {
|
|
167
206
|
const allMatches = findMatchingElements(html, matchLandmarkTypes);
|
|
168
|
-
const
|
|
207
|
+
const mainMatches = allMatches.filter(isMainMatch);
|
|
208
|
+
const excisableMatches = allMatches.filter((match) => !isMainMatch(match));
|
|
209
|
+
const outermost = keepOutermost(excisableMatches);
|
|
210
|
+
const outermostMain = keepOutermost(mainMatches);
|
|
169
211
|
const result = {
|
|
170
212
|
header: [],
|
|
171
213
|
footer: [],
|
|
@@ -173,13 +215,31 @@ export function extractLandmarks(html) {
|
|
|
173
215
|
aside: [],
|
|
174
216
|
form: [],
|
|
175
217
|
search: [],
|
|
218
|
+
main: [],
|
|
176
219
|
remainderHtml: html,
|
|
177
220
|
};
|
|
178
221
|
const spans = [];
|
|
222
|
+
const lineColumnIndex = buildLineColumnIndex(html);
|
|
223
|
+
const toInstance = (match) => {
|
|
224
|
+
const start = offsetToLineColumn(lineColumnIndex, match.startOffset);
|
|
225
|
+
const end = offsetToLineColumn(lineColumnIndex, match.endOffset);
|
|
226
|
+
return {
|
|
227
|
+
html: html.slice(match.startOffset, match.endOffset),
|
|
228
|
+
startOffset: match.startOffset,
|
|
229
|
+
endOffset: match.endOffset,
|
|
230
|
+
startLine: start.line,
|
|
231
|
+
startColumn: start.column,
|
|
232
|
+
endLine: end.line,
|
|
233
|
+
endColumn: end.column,
|
|
234
|
+
};
|
|
235
|
+
};
|
|
179
236
|
for (const match of outermost) {
|
|
180
|
-
result[match.type].push(
|
|
237
|
+
result[match.type].push(toInstance(match));
|
|
181
238
|
spans.push({ start: match.startOffset, end: match.endOffset });
|
|
182
239
|
}
|
|
240
|
+
for (const match of outermostMain) {
|
|
241
|
+
result.main.push(toInstance(match));
|
|
242
|
+
}
|
|
183
243
|
result.remainderHtml = excise(html, spans);
|
|
184
244
|
return result;
|
|
185
245
|
}
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Default containment threshold for {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance}.
|
|
3
|
+
* Chosen to match the quorum/shell fractions used elsewhere in this pipeline
|
|
4
|
+
* ({@link ./merge-cross-block-clusters.js | mergeCrossBlockClusters}'s
|
|
5
|
+
* `QUORUM_FRACTION`, {@link ./shell-quorum.js | SHELL_QUORUM_FALLBACK_FRACTION}) —
|
|
6
|
+
* unlike those, this exact value has not been validated against real crawl
|
|
7
|
+
* data; it is a starting point carried over by convention, adjustable via
|
|
8
|
+
* this function's `threshold` parameter.
|
|
9
|
+
*/
|
|
10
|
+
export declare const DEFAULT_CHROME_OVERLAP_THRESHOLD = 0.8;
|
|
11
|
+
/**
|
|
12
|
+
* Classifies a single landmark instance as chrome (shared site/section
|
|
13
|
+
* furniture) or content, given the instance's own tokens and the shell
|
|
14
|
+
* token set {@link ./shell-quorum.js | shellQuorum} discovered for its unit.
|
|
15
|
+
*
|
|
16
|
+
* ## Why containment (`|instance ∩ shell| / |instance|`) and not Jaccard
|
|
17
|
+
*
|
|
18
|
+
* `shellTokens` is the union of chrome tokens across an entire unit's
|
|
19
|
+
* landmark instances (header + nav + footer + …, corpus-wide), so it is
|
|
20
|
+
* usually far larger than any single instance's own token set. Jaccard's
|
|
21
|
+
* denominator is the *union* of both sets, which stays shell-sized even when
|
|
22
|
+
* the instance is 100% shell tokens — driving the score down regardless of
|
|
23
|
+
* how purely "shell" the instance is. Containment instead asks "of this
|
|
24
|
+
* instance's own tokens, how many are shell tokens", which is the question
|
|
25
|
+
* that actually matters for classifying one instance.
|
|
26
|
+
*
|
|
27
|
+
* An instance with zero tokens is never chrome (there is nothing to
|
|
28
|
+
* corroborate) — matches {@link ./per-page-landmark-signatures.js | computePerPageLandmarkInstances}'s
|
|
29
|
+
* own choice to drop zero-token instances before they ever reach a
|
|
30
|
+
* `PerPageLandmarkInstance`, kept here as a defensive default rather than an
|
|
31
|
+
* assumption about every caller.
|
|
32
|
+
* @param instanceTokens
|
|
33
|
+
* @param shellTokens
|
|
34
|
+
* @param threshold
|
|
35
|
+
* @example
|
|
36
|
+
* ```ts
|
|
37
|
+
* const shellTokens = shellQuorum(unitPerPageInstances);
|
|
38
|
+
* const isChrome = isChromeLandmarkInstance(instance.tokens, shellTokens);
|
|
39
|
+
* ```
|
|
40
|
+
*/
|
|
41
|
+
export declare function isChromeLandmarkInstance(instanceTokens: ReadonlySet<string>, shellTokens: ReadonlySet<string>, threshold?: number): boolean;
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Default containment threshold for {@link ./is-chrome-landmark-instance.js | isChromeLandmarkInstance}.
|
|
3
|
+
* Chosen to match the quorum/shell fractions used elsewhere in this pipeline
|
|
4
|
+
* ({@link ./merge-cross-block-clusters.js | mergeCrossBlockClusters}'s
|
|
5
|
+
* `QUORUM_FRACTION`, {@link ./shell-quorum.js | SHELL_QUORUM_FALLBACK_FRACTION}) —
|
|
6
|
+
* unlike those, this exact value has not been validated against real crawl
|
|
7
|
+
* data; it is a starting point carried over by convention, adjustable via
|
|
8
|
+
* this function's `threshold` parameter.
|
|
9
|
+
*/
|
|
10
|
+
export const DEFAULT_CHROME_OVERLAP_THRESHOLD = 0.8;
|
|
11
|
+
/**
|
|
12
|
+
* Classifies a single landmark instance as chrome (shared site/section
|
|
13
|
+
* furniture) or content, given the instance's own tokens and the shell
|
|
14
|
+
* token set {@link ./shell-quorum.js | shellQuorum} discovered for its unit.
|
|
15
|
+
*
|
|
16
|
+
* ## Why containment (`|instance ∩ shell| / |instance|`) and not Jaccard
|
|
17
|
+
*
|
|
18
|
+
* `shellTokens` is the union of chrome tokens across an entire unit's
|
|
19
|
+
* landmark instances (header + nav + footer + …, corpus-wide), so it is
|
|
20
|
+
* usually far larger than any single instance's own token set. Jaccard's
|
|
21
|
+
* denominator is the *union* of both sets, which stays shell-sized even when
|
|
22
|
+
* the instance is 100% shell tokens — driving the score down regardless of
|
|
23
|
+
* how purely "shell" the instance is. Containment instead asks "of this
|
|
24
|
+
* instance's own tokens, how many are shell tokens", which is the question
|
|
25
|
+
* that actually matters for classifying one instance.
|
|
26
|
+
*
|
|
27
|
+
* An instance with zero tokens is never chrome (there is nothing to
|
|
28
|
+
* corroborate) — matches {@link ./per-page-landmark-signatures.js | computePerPageLandmarkInstances}'s
|
|
29
|
+
* own choice to drop zero-token instances before they ever reach a
|
|
30
|
+
* `PerPageLandmarkInstance`, kept here as a defensive default rather than an
|
|
31
|
+
* assumption about every caller.
|
|
32
|
+
* @param instanceTokens
|
|
33
|
+
* @param shellTokens
|
|
34
|
+
* @param threshold
|
|
35
|
+
* @example
|
|
36
|
+
* ```ts
|
|
37
|
+
* const shellTokens = shellQuorum(unitPerPageInstances);
|
|
38
|
+
* const isChrome = isChromeLandmarkInstance(instance.tokens, shellTokens);
|
|
39
|
+
* ```
|
|
40
|
+
*/
|
|
41
|
+
export function isChromeLandmarkInstance(instanceTokens, shellTokens, threshold = DEFAULT_CHROME_OVERLAP_THRESHOLD) {
|
|
42
|
+
if (instanceTokens.size === 0)
|
|
43
|
+
return false;
|
|
44
|
+
let hit = 0;
|
|
45
|
+
for (const token of instanceTokens) {
|
|
46
|
+
if (shellTokens.has(token))
|
|
47
|
+
hit++;
|
|
48
|
+
}
|
|
49
|
+
return hit / instanceTokens.size >= threshold;
|
|
50
|
+
}
|