npm - @d-zero/page-cluster - Versions diffs - 0.2.0 - Mend

@d-zero/page-cluster 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (80) hide show

package/README.md +68 -0
package/dist/array-edit-distance.d.ts +20 -0
package/dist/array-edit-distance.js +52 -0
package/dist/build-segment.d.ts +18 -0
package/dist/build-segment.js +27 -0
package/dist/cap-content-depth.d.ts +69 -0
package/dist/cap-content-depth.js +161 -0
package/dist/compute-document-frequency.d.ts +33 -0
package/dist/compute-document-frequency.js +40 -0
package/dist/create-frame.d.ts +16 -0
package/dist/create-frame.js +29 -0
package/dist/derive-path-group-key.d.ts +44 -0
package/dist/derive-path-group-key.js +51 -0
package/dist/derive-stylesheet-group-key.d.ts +36 -0
package/dist/derive-stylesheet-group-key.js +41 -0
package/dist/detect-content-depth-cap.d.ts +114 -0
package/dist/detect-content-depth-cap.js +137 -0
package/dist/escape-reg-exp.d.ts +11 -0
package/dist/escape-reg-exp.js +13 -0
package/dist/excise.d.ts +13 -0
package/dist/excise.js +24 -0
package/dist/extract-landmarks.d.ts +82 -0
package/dist/extract-landmarks.js +104 -0
package/dist/filter-first-party-stylesheet-hrefs.d.ts +73 -0
package/dist/filter-first-party-stylesheet-hrefs.js +118 -0
package/dist/find-shallowest-elements.d.ts +39 -0
package/dist/find-shallowest-elements.js +121 -0
package/dist/foldable-tags.d.ts +8 -0
package/dist/foldable-tags.js +8 -0
package/dist/format-bracket.d.ts +11 -0
package/dist/format-bracket.js +17 -0
package/dist/hash-content.d.ts +22 -0
package/dist/hash-content.js +26 -0
package/dist/html-region-utils.d.ts +74 -0
package/dist/html-region-utils.js +96 -0
package/dist/is-fold-candidate.d.ts +13 -0
package/dist/is-fold-candidate.js +16 -0
package/dist/is-genuine-close.d.ts +23 -0
package/dist/is-genuine-close.js +27 -0
package/dist/is-noise-class.d.ts +6 -0
package/dist/is-noise-class.js +8 -0
package/dist/jaccard-similarity.d.ts +23 -0
package/dist/jaccard-similarity.js +36 -0
package/dist/merge-landmark-affined-clusters.d.ts +179 -0
package/dist/merge-landmark-affined-clusters.js +544 -0
package/dist/merge-spans.d.ts +15 -0
package/dist/merge-spans.js +22 -0
package/dist/noise-class-patterns.d.ts +21 -0
package/dist/noise-class-patterns.js +74 -0
package/dist/normalize-for-hash.d.ts +10 -0
package/dist/normalize-for-hash.js +12 -0
package/dist/opaque-tags.d.ts +17 -0
package/dist/opaque-tags.js +18 -0
package/dist/parse-class-list.d.ts +10 -0
package/dist/parse-class-list.js +23 -0
package/dist/reassign-orphan-block-keys.d.ts +81 -0
package/dist/reassign-orphan-block-keys.js +159 -0
package/dist/remove-content-blocks.d.ts +67 -0
package/dist/remove-content-blocks.js +150 -0
package/dist/resolve-blocking-group-keys.d.ts +116 -0
package/dist/resolve-blocking-group-keys.js +120 -0
package/dist/resolve-closed-frame.d.ts +26 -0
package/dist/resolve-closed-frame.js +33 -0
package/dist/resolve-landmark-variant-keys.d.ts +66 -0
package/dist/resolve-landmark-variant-keys.js +71 -0
package/dist/resolve-options.d.ts +6 -0
package/dist/resolve-options.js +10 -0
package/dist/resolve-page-cluster-keys.d.ts +222 -0
package/dist/resolve-page-cluster-keys.js +198 -0
package/dist/resolve-structural-cluster-keys.d.ts +50 -0
package/dist/resolve-structural-cluster-keys.js +287 -0
package/dist/run-tokenizer.d.ts +33 -0
package/dist/run-tokenizer.js +152 -0
package/dist/split-tokens-by-frequency.d.ts +46 -0
package/dist/split-tokens-by-frequency.js +88 -0
package/dist/tokenize.d.ts +58 -0
package/dist/tokenize.js +60 -0
package/dist/types.d.ts +85 -0
package/dist/types.js +1 -0
package/package.json +102 -0

package/README.md ADDED Viewed

@@ -0,0 +1,68 @@
+# `@d-zero/page-cluster`
+大量クローリングした HTML の重複・類似ページ検出のためのパッケージ。`tokenize()` は `<body>` 配下のHTMLを、テキストを除去した構造トークンに変換する。用途・設計判断のWHYは `src/tokenize.ts` の JSDoc を参照。
+## Installation
+```sh
+yarn add @d-zero/page-cluster
+```
+## Usage
+```ts
+import { tokenize } from '@d-zero/page-cluster';
+const { tokens, bodyClassList } = tokenize(
+	'<body class="law-page"><div class="card"><ul><li>A</li><li>B</li></ul></div></body>',
+);
+// tokens: ["body>.card>ul>li", "body>.card>ul>li"]
+// bodyClassList: ["law-page"]
+```
+### オプション
+```ts
+tokenize(html, {
+	filterNoiseClasses: true, // 既定値。ハッシュ的自動生成class名を除外する
+	includeComments: false, // 既定値。コメントノードをトークン化しない
+});
+```
+### クラスタリング
+クロールしたページ群から最終的なクラスタキーを得るには `resolvePageClusterKeys()` を使う。ブロッキング（URLパス/スタイルシートによる粗い絞り込み）と構造クラスタリング（ブロック内でのcomplete-linkage階層的クラスタリング）を内部で連結し、ブロックを跨いで一意なキーを返す。既定で各ページの`<header>`/`<footer>`/`<nav>`/`<aside>`（タグ名またはARIAランドマークロール）を比較対象から除外し、共通chromeの影響を受けにくくする。また、スタイルシート参照が記録されていない「孤児」ページを、同一URLセクションに閉じたスタイルシート・ブロックへ再割当する処理（`reassignOrphanBlockKeys()`）や、埋め込みコンテンツが引き込むサードパーティCSS参照をブロッキング判定から除外する処理（`filterFirstPartyStylesheetHrefs()`）も既定で有効。挙動の詳細・トレードオフはそれぞれのJSDocを参照。
+自由編集ブロックエディタ（CMSが各コンテンツブロックに固有のdata属性を付与するタイプ）を使うサイトでは、`contentBlockAttribute` オプションでその属性名を指定すると、ページごとに異なるブロック構成が構造比較のノイズになるのを防げる（既定は未指定＝無効、サイトごとの属性名を推測できないため）。詳細は `removeContentBlocks()` のJSDocを参照。
+CMSのブロック属性名が分からない・サイトごとに違う場合は `autoCapMainDepth: true` を使う。`<main>`/`role="main"`という標準タグを起点に、構造クラスタ数が急増する直前の深さをブロックごとに実データから自動検出して打ち切るため、サイト固有の設定が一切不要（既定はfalse。実データ検証では`contentBlockAttribute`より良い結果になる場合もあった一方、計算コストが実測で数倍〜1桁台後半になる。倍率はコーパスのブロック構成に依存する）。詳細は `detectContentDepthCap()` のJSDocを参照。
+header/footer/nav/asideが一致するページ同士をさらに合流させたい場合は `mergeRareLandmarkClusters: true` を使う。ただし単純な一致判定は実データで過剰融合を招くことが分かっているため（header/footer/navは99%以上のページに存在し判別力を持たない）、コーパス全体で希少なランドマークバリアントが一致した場合に限り、より緩いコンテンツ類似度閾値（`landmarkGateSimilarityThreshold`）での合流を許可する（既定はfalse。実データでの検証は未実施で、合成フィクスチャでの単体・回帰テストのみ）。詳細・コスト特性は `mergeLandmarkAffinedClusters()` のJSDocを参照。
+```ts
+import { resolvePageClusterKeys } from '@d-zero/page-cluster/resolve-page-cluster-keys';
+const keys = resolvePageClusterKeys(
+	pages.map((page) => ({
+		paths: page.urlPathSegments,
+		stylesheetHrefs: page.stylesheetHrefs,
+		html: page.html,
+	})),
+	{ contentBlockAttribute: 'data-bgb' }, // 使っているCMSのブロック属性名に合わせて指定
+);
+// pagesと同じ順序・同じ長さ。同じキーのページが同一テンプレートと判定されたページ群
+```
+### ランドマークバリアント分類
+「同一テンプレートか」ではなく「このページはどのヘッダー/フッター/ナビ/サイドナビを持っているか」というメタプロパティを個別に知りたい場合は `resolveLandmarkVariantKeys()` を使う。`resolvePageClusterKeys()` とは独立した戻り値で、両者の合成は呼び出し側の責務。
+```ts
+import { resolveLandmarkVariantKeys } from '@d-zero/page-cluster/resolve-landmark-variant-keys';
+const headerVariantKeys = resolveLandmarkVariantKeys(
+	pages.map((page) => page.html),
+	'header',
+);
+// pagesと同じ順序・同じ長さ。同じキーのページが同じヘッダーデザインを持つページ群
+```

package/dist/array-edit-distance.d.ts ADDED Viewed

@@ -0,0 +1,20 @@
+/**
+ * Element-wise Levenshtein distance between two `tokenize()` outputs, for
+ * the small set of comparisons that need order/nesting to matter (refining
+ * a merge distance between near-duplicate candidates, spot-checking cluster
+ * quality) rather than the set-based similarity used for bulk narrowing.
+ * Operates on whole array elements, not characters, so a single differing
+ * path costs 1 edit regardless of its string length. This is the same
+ * O(n*m) dynamic-programming shape as tree edit distance, but requires no
+ * tree: run directly on the flat leaf-path arrays `tokenize()` already
+ * produces, which is why it stays viable at the scale this comparison is
+ * meant for (small numbers of already-narrowed candidates, not all-pairs).
+ * @param a
+ * @param b
+ * @example
+ * ```ts
+ * arrayEditDistance(['body>ul>li', 'body>ul>li'], ['body>ul>li']);
+ * // 1
+ * ```
+ */
+export declare function arrayEditDistance(a: readonly string[], b: readonly string[]): number;

package/dist/array-edit-distance.js ADDED Viewed

@@ -0,0 +1,52 @@
+/**
+ * Reads `row[index]`, throwing instead of returning `undefined`. The DP loop
+ * below only ever indexes within bounds it just built, so the thrown branch
+ * is unreachable in practice; it exists to satisfy `noUncheckedIndexedAccess`
+ * without a non-null assertion. Named `readDpValue` rather than `at` to avoid
+ * reading as (and being confused for) `Array.prototype.at`, whose negative-
+ * index-from-end semantics this helper does not share.
+ * @param row
+ * @param index
+ */
+function readDpValue(row, index) {
+    const value = row[index];
+    if (value === undefined) {
+        throw new Error('arrayEditDistance: DP row index out of bounds');
+    }
+    return value;
+}
+/**
+ * Element-wise Levenshtein distance between two `tokenize()` outputs, for
+ * the small set of comparisons that need order/nesting to matter (refining
+ * a merge distance between near-duplicate candidates, spot-checking cluster
+ * quality) rather than the set-based similarity used for bulk narrowing.
+ * Operates on whole array elements, not characters, so a single differing
+ * path costs 1 edit regardless of its string length. This is the same
+ * O(n*m) dynamic-programming shape as tree edit distance, but requires no
+ * tree: run directly on the flat leaf-path arrays `tokenize()` already
+ * produces, which is why it stays viable at the scale this comparison is
+ * meant for (small numbers of already-narrowed candidates, not all-pairs).
+ * @param a
+ * @param b
+ * @example
+ * ```ts
+ * arrayEditDistance(['body>ul>li', 'body>ul>li'], ['body>ul>li']);
+ * // 1
+ * ```
+ */
+export function arrayEditDistance(a, b) {
+    const rowCount = a.length;
+    const colCount = b.length;
+    let previousRow = Array.from({ length: colCount + 1 }, (_, index) => index);
+    for (let row = 1; row <= rowCount; row++) {
+        const currentRow = [row];
+        for (let col = 1; col <= colCount; col++) {
+            currentRow.push(a[row - 1] === b[col - 1]
+                ? readDpValue(previousRow, col - 1)
+                : 1 +
+                    Math.min(readDpValue(previousRow, col), readDpValue(currentRow, col - 1), readDpValue(previousRow, col - 1)));
+        }
+        previousRow = currentRow;
+    }
+    return readDpValue(previousRow, colCount);
+}

package/dist/build-segment.d.ts ADDED Viewed

@@ -0,0 +1,18 @@
+/**
+ * Builds the path segment string for one element.
+ *
+ * `div`/`span` drop their tag name in favor of `.class` when they carry a
+ * class list, since the class (not the generic wrapper tag) is what carries
+ * meaning for those two tags. Every other tag always keeps its name because
+ * the tag itself is semantically meaningful (e.g. `ul`, `table`, `button`).
+ *
+ * `role`/`type` are appended as a bracket suffix rather than folded into the
+ * class list: unlike `class`, they are single-valued attributes with their
+ * own semantics (ARIA role, form control kind), so keeping them visually
+ * distinct avoids collisions with an actual class named e.g. `button`.
+ * @param tagName
+ * @param classList
+ * @param role
+ * @param type
+ */
+export declare function buildSegment(tagName: string, classList: readonly string[], role?: string, type?: string): string;

package/dist/build-segment.js ADDED Viewed

@@ -0,0 +1,27 @@
+import { FOLDABLE_TAGS } from './foldable-tags.js';
+import { formatBracket } from './format-bracket.js';
+/**
+ * Builds the path segment string for one element.
+ *
+ * `div`/`span` drop their tag name in favor of `.class` when they carry a
+ * class list, since the class (not the generic wrapper tag) is what carries
+ * meaning for those two tags. Every other tag always keeps its name because
+ * the tag itself is semantically meaningful (e.g. `ul`, `table`, `button`).
+ *
+ * `role`/`type` are appended as a bracket suffix rather than folded into the
+ * class list: unlike `class`, they are single-valued attributes with their
+ * own semantics (ARIA role, form control kind), so keeping them visually
+ * distinct avoids collisions with an actual class named e.g. `button`.
+ * @param tagName
+ * @param classList
+ * @param role
+ * @param type
+ */
+export function buildSegment(tagName, classList, role, type) {
+    const base = classList.length > 0
+        ? FOLDABLE_TAGS.has(tagName)
+            ? `.${classList.join('.')}`
+            : `${tagName}.${classList.join('.')}`
+        : tagName;
+    return `${base}${formatBracket({ role, type })}`;
+}

package/dist/cap-content-depth.d.ts ADDED Viewed

@@ -0,0 +1,69 @@
+/**
+ * The only landmark this function knows how to depth-cap. A closed union
+ * (not an open `string`) because, unlike
+ * {@link ./remove-content-blocks.js | removeContentBlocks}'s caller-supplied
+ * CMS attribute, `<main>`/`role="main"` is an HTML5/ARIA standard — there is
+ * exactly one vocabulary to support, not one per site.
+ */
+export type ContentDepthLandmark = 'main';
+/**
+ * @see capContentDepth
+ */
+export type CapContentDepthOptions = {
+    landmark: ContentDepthLandmark;
+    /**
+     * How many levels of elements *inside* the landmark to keep, counting the
+     * landmark's own direct children as depth 1. Must be a non-negative
+     * integer (0 keeps none of the landmark's content, only its own opening/
+     * closing tags). See {@link ./cap-content-depth.js | capContentDepth}'s
+     * JSDoc for how to choose this.
+     */
+    maxDepth: number;
+};
+/**
+ * Result of {@link ./cap-content-depth.js | capContentDepth}.
+ */
+export type CapContentDepthResult = {
+    remainderHtml: string;
+};
+/**
+ * Excises the deepest content inside `options.landmark` (currently only
+ * `'main'`/`role="main"` is supported — see `ContentDepthLandmark`'s JSDoc),
+ * keeping up to `options.maxDepth` levels of nesting and returning what's
+ * left.
+ *
+ * Built for the same real-crawl finding {@link ./remove-content-blocks.js |
+ * removeContentBlocks} addresses — freeform CMS-block content dominating a
+ * page's token set and defeating structural similarity — but without
+ * needing the caller to know their CMS's own block-marker attribute.
+ * `<main>` is HTML5-standard, so this works without any per-site
+ * configuration at all. Confirmed on two unrelated real crawls (302 and
+ * ~4,100 pages): once nesting depth inside `<main>` passes a threshold
+ * (3, on both), the number of distinct structural clusters explodes (14x
+ * and 9x, respectively) — the "skeleton" (which template a page uses) lives
+ * in the shallow levels; the free-edited content that varies page-to-page
+ * lives deeper. See {@link ./detect-content-depth-cap.js |
+ * detectContentDepthCap} to find that threshold automatically instead of
+ * hardcoding `maxDepth`.
+ *
+ * `<main>`'s own opening/closing tags (and their attributes, e.g. a class
+ * that itself differs between a list-page and a detail-page `<main>`) are
+ * always kept — only what's *between* them past `maxDepth` is excised, for
+ * the same reason `extractLandmarks` never adds a placeholder: the tag
+ * itself is real structural signal, not something to erase.
+ *
+ * A page with no `<main>` and no `role="main"` element has nothing to cap;
+ * `remainderHtml` is returned unchanged, matching this package's convention
+ * for "nothing to do" (see `extractLandmarks`, `removeContentBlocks`).
+ * @param html
+ * @param options
+ * @example
+ * ```ts
+ * capContentDepth(
+ * 	'<body><main><div><div><div><div>too deep</div></div></div></div></main></body>',
+ * 	{ landmark: 'main', maxDepth: 2 },
+ * );
+ * // { remainderHtml: '<body><main><div><div></div></div></main></body>' }
+ * ```
+ */
+export declare function capContentDepth(html: string, options: CapContentDepthOptions): CapContentDepthResult;

package/dist/cap-content-depth.js ADDED Viewed

@@ -0,0 +1,161 @@
+import { Parser } from 'htmlparser2';
+import { excise } from './excise.js';
+import { findShallowestElements } from './find-shallowest-elements.js';
+import { isGenuineClose } from './is-genuine-close.js';
+import { isOpaqueTagName } from './opaque-tags.js';
+const TAG_TO_LANDMARK = { main: 'main' };
+const ROLE_TO_LANDMARK = { main: 'main' };
+/**
+ * Finds the single shallowest `<main>`/`role="main"` element in `html` via
+ * {@link ./find-shallowest-elements.js | findShallowestElements} (same
+ * "shallowest wins" rule as {@link ./extract-landmarks.js | extractLandmarks}
+ * uses for its own four landmark types, for the same reason: the site-wide,
+ * outermost instance is the real one), and returns the offsets of its
+ * content (excluding its own opening/closing tags) — or `undefined` if there
+ * is no genuine one.
+ * @param html
+ * @param landmark
+ */
+function findShallowestLandmarkContent(html, landmark) {
+    const [winner] = findShallowestElements(html, (name, role) => TAG_TO_LANDMARK[name] === landmark ||
+        (role !== undefined && ROLE_TO_LANDMARK[role] === landmark)
+        ? [landmark]
+        : []);
+    return winner
+        ? { contentStart: winner.contentStart, contentEnd: winner.contentEnd }
+        : undefined;
+}
+/**
+ * Within `html.slice(contentStart, contentEnd)`, finds every element whose
+ * nesting depth (the landmark's own direct children are depth 1) exceeds
+ * `maxDepth`, and returns their `[start, end)` spans (absolute offsets into
+ * the original `html`) for excision. Once a too-deep element is found, its
+ * subtree is not explored further — same reasoning as
+ * {@link ./remove-content-blocks.js | removeContentBlocks} not diving into
+ * an already-matched block: nothing inside a span already marked for
+ * removal needs its own depth checked.
+ * @param html
+ * @param contentStart
+ * @param contentEnd
+ * @param maxDepth
+ */
+function collectDeepSpans(html, contentStart, contentEnd, maxDepth) {
+    const spans = [];
+    const stack = [];
+    let opaque = null;
+    // Set once a too-deep element opens; cleared when that same element
+    // closes. While set, every nested open/close (other than matching
+    // closes of the capped tag itself) is ignored, same shape as the
+    // `opaque` tracking above.
+    let cappedAt = null;
+    const parser = new Parser({
+        onopentag(name) {
+            if (opaque) {
+                if (name === opaque.tagName)
+                    opaque.depth++;
+                return;
+            }
+            if (cappedAt) {
+                if (name === cappedAt.tagName)
+                    cappedAt.depth++;
+                return;
+            }
+            if (isOpaqueTagName(name)) {
+                opaque = { tagName: name, depth: 1 };
+                return;
+            }
+            const depth = stack.length + 1;
+            if (depth > maxDepth) {
+                cappedAt = { tagName: name, depth: 1 };
+                spans.push({ start: contentStart + parser.startIndex, end: -1 });
+                return;
+            }
+            stack.push({ tagName: name, startOffset: parser.startIndex });
+        },
+        onclosetag(name) {
+            if (opaque) {
+                if (name === opaque.tagName) {
+                    opaque.depth--;
+                    if (opaque.depth === 0)
+                        opaque = null;
+                }
+                return;
+            }
+            if (cappedAt) {
+                if (name === cappedAt.tagName) {
+                    cappedAt.depth--;
+                    if (cappedAt.depth === 0) {
+                        const endOffset = contentStart + parser.endIndex + 1;
+                        const open = spans.at(-1);
+                        if (open) {
+                            open.end = isGenuineClose(html, endOffset, name) ? endOffset : open.start;
+                        }
+                        cappedAt = null;
+                    }
+                }
+                return;
+            }
+            const frame = stack.pop();
+            if (!frame)
+                return;
+        },
+    }, { decodeEntities: false });
+    parser.end(html.slice(contentStart, contentEnd));
+    // A capped span whose genuine close was never confirmed (malformed
+    // markup) collapses to a zero-length span at its own start — excise()
+    // treats start === end as a no-op slice, so nothing is corrupted, and
+    // that one candidate is simply not removed, the same safety trade-off
+    // extractLandmarks/removeContentBlocks make for unclosed tags.
+    return spans.filter((span) => span.end > span.start);
+}
+/**
+ * Excises the deepest content inside `options.landmark` (currently only
+ * `'main'`/`role="main"` is supported — see `ContentDepthLandmark`'s JSDoc),
+ * keeping up to `options.maxDepth` levels of nesting and returning what's
+ * left.
+ *
+ * Built for the same real-crawl finding {@link ./remove-content-blocks.js |
+ * removeContentBlocks} addresses — freeform CMS-block content dominating a
+ * page's token set and defeating structural similarity — but without
+ * needing the caller to know their CMS's own block-marker attribute.
+ * `<main>` is HTML5-standard, so this works without any per-site
+ * configuration at all. Confirmed on two unrelated real crawls (302 and
+ * ~4,100 pages): once nesting depth inside `<main>` passes a threshold
+ * (3, on both), the number of distinct structural clusters explodes (14x
+ * and 9x, respectively) — the "skeleton" (which template a page uses) lives
+ * in the shallow levels; the free-edited content that varies page-to-page
+ * lives deeper. See {@link ./detect-content-depth-cap.js |
+ * detectContentDepthCap} to find that threshold automatically instead of
+ * hardcoding `maxDepth`.
+ *
+ * `<main>`'s own opening/closing tags (and their attributes, e.g. a class
+ * that itself differs between a list-page and a detail-page `<main>`) are
+ * always kept — only what's *between* them past `maxDepth` is excised, for
+ * the same reason `extractLandmarks` never adds a placeholder: the tag
+ * itself is real structural signal, not something to erase.
+ *
+ * A page with no `<main>` and no `role="main"` element has nothing to cap;
+ * `remainderHtml` is returned unchanged, matching this package's convention
+ * for "nothing to do" (see `extractLandmarks`, `removeContentBlocks`).
+ * @param html
+ * @param options
+ * @example
+ * ```ts
+ * capContentDepth(
+ * 	'<body><main><div><div><div><div>too deep</div></div></div></div></main></body>',
+ * 	{ landmark: 'main', maxDepth: 2 },
+ * );
+ * // { remainderHtml: '<body><main><div><div></div></div></main></body>' }
+ * ```
+ */
+export function capContentDepth(html, options) {
+    if (!(Number.isInteger(options.maxDepth) && options.maxDepth >= 0)) {
+        throw new RangeError(`capContentDepth: maxDepth must be a non-negative integer, got ${options.maxDepth}`);
+    }
+    const content = findShallowestLandmarkContent(html, options.landmark);
+    if (!content) {
+        return { remainderHtml: html };
+    }
+    const spans = collectDeepSpans(html, content.contentStart, content.contentEnd, options.maxDepth);
+    return { remainderHtml: excise(html, spans) };
+}

package/dist/compute-document-frequency.d.ts ADDED Viewed

@@ -0,0 +1,33 @@
+import type { DocumentFrequency } from './types.js';
+/**
+ * Counts, for each token, how many of the given per-page token sets contain
+ * it. This is the first half of separating a page's shared site chrome
+ * (header/nav/footer) from its page-specific content: a token that recurs
+ * across nearly every page in `tokenSets` is chrome, one that shows up on
+ * only a handful of pages is content — see `splitTokensByFrequency`, which
+ * consumes this result to make that call per token.
+ *
+ * `tokenSets` must be a *homogeneous* page collection (typically one site,
+ * or one section of a large multi-template site), not an arbitrary pool.
+ * Real-data validation against a small single-layout corporate site (a few
+ * hundred pages) found a clean bimodal frequency split (site chrome tokens
+ * showed up on 95%+ of pages, content tokens on well under 50%, with
+ * nothing in between). The same computation against the *whole* crawl of a
+ * much larger site that turned out to be a federation of independent
+ * sub-sections (the largest covering under half of all pages) found no
+ * token crossing even a 50% document-frequency threshold: with no single
+ * dominant layout, frequency-based chrome detection needs a mostly-
+ * homogeneous input to work at all. Splitting such a site into its
+ * sections first (by URL path, or by a coarse structural clustering pass)
+ * and calling this function per section recovered the same clean bimodal
+ * split. Grouping heterogeneous pages before calling this function is the
+ * caller's responsibility; this function has no way to detect that its
+ * input mixes multiple layouts.
+ * @param tokenSets
+ * @example
+ * ```ts
+ * computeDocumentFrequency([new Set(['body>header>a']), new Set(['body>header>a'])]);
+ * // { documentFrequency: Map { 'body>header>a' => 2 }, pageCount: 2 }
+ * ```
+ */
+export declare function computeDocumentFrequency(tokenSets: readonly ReadonlySet<string>[]): DocumentFrequency;

package/dist/compute-document-frequency.js ADDED Viewed

@@ -0,0 +1,40 @@
+/**
+ * Counts, for each token, how many of the given per-page token sets contain
+ * it. This is the first half of separating a page's shared site chrome
+ * (header/nav/footer) from its page-specific content: a token that recurs
+ * across nearly every page in `tokenSets` is chrome, one that shows up on
+ * only a handful of pages is content — see `splitTokensByFrequency`, which
+ * consumes this result to make that call per token.
+ *
+ * `tokenSets` must be a *homogeneous* page collection (typically one site,
+ * or one section of a large multi-template site), not an arbitrary pool.
+ * Real-data validation against a small single-layout corporate site (a few
+ * hundred pages) found a clean bimodal frequency split (site chrome tokens
+ * showed up on 95%+ of pages, content tokens on well under 50%, with
+ * nothing in between). The same computation against the *whole* crawl of a
+ * much larger site that turned out to be a federation of independent
+ * sub-sections (the largest covering under half of all pages) found no
+ * token crossing even a 50% document-frequency threshold: with no single
+ * dominant layout, frequency-based chrome detection needs a mostly-
+ * homogeneous input to work at all. Splitting such a site into its
+ * sections first (by URL path, or by a coarse structural clustering pass)
+ * and calling this function per section recovered the same clean bimodal
+ * split. Grouping heterogeneous pages before calling this function is the
+ * caller's responsibility; this function has no way to detect that its
+ * input mixes multiple layouts.
+ * @param tokenSets
+ * @example
+ * ```ts
+ * computeDocumentFrequency([new Set(['body>header>a']), new Set(['body>header>a'])]);
+ * // { documentFrequency: Map { 'body>header>a' => 2 }, pageCount: 2 }
+ * ```
+ */
+export function computeDocumentFrequency(tokenSets) {
+    const documentFrequency = new Map();
+    for (const tokens of tokenSets) {
+        for (const token of tokens) {
+            documentFrequency.set(token, (documentFrequency.get(token) ?? 0) + 1);
+        }
+    }
+    return { documentFrequency, pageCount: tokenSets.length };
+}

package/dist/create-frame.d.ts ADDED Viewed

@@ -0,0 +1,16 @@
+import type { Frame, ResolvedOptions } from './types.js';
+/**
+ * Builds the stack frame for a newly-opened element. `id`/`data-*`/every
+ * `aria-*` other than `role` are read from `attribs` implicitly by never
+ * being looked at: `id`/`data-*` tend to be per-instance/per-page values
+ * (breaking similarity detection the same way raw text would), and
+ * non-`role` `aria-*` attributes are either free-text (`aria-label`,
+ * `aria-describedby` — inconsistent with dropping visible text) or render
+ * state (`aria-current`, `aria-expanded`, `aria-selected` — the same
+ * per-page-varying-position problem as a `current`/`active` class, see
+ * `tokenize.ts`).
+ * @param tagName
+ * @param attribs
+ * @param options
+ */
+export declare function createFrame(tagName: string, attribs: Record<string, string>, options: ResolvedOptions): Frame;

package/dist/create-frame.js ADDED Viewed

@@ -0,0 +1,29 @@
+import { buildSegment } from './build-segment.js';
+import { isFoldCandidate } from './is-fold-candidate.js';
+import { parseClassList } from './parse-class-list.js';
+/**
+ * Builds the stack frame for a newly-opened element. `id`/`data-*`/every
+ * `aria-*` other than `role` are read from `attribs` implicitly by never
+ * being looked at: `id`/`data-*` tend to be per-instance/per-page values
+ * (breaking similarity detection the same way raw text would), and
+ * non-`role` `aria-*` attributes are either free-text (`aria-label`,
+ * `aria-describedby` — inconsistent with dropping visible text) or render
+ * state (`aria-current`, `aria-expanded`, `aria-selected` — the same
+ * per-page-varying-position problem as a `current`/`active` class, see
+ * `tokenize.ts`).
+ * @param tagName
+ * @param attribs
+ * @param options
+ */
+export function createFrame(tagName, attribs, options) {
+    const classList = parseClassList(attribs.class, options.filterNoiseClasses);
+    const role = attribs.role || undefined;
+    const type = attribs.type || undefined;
+    return {
+        tagName,
+        segment: buildSegment(tagName, classList, role, type),
+        isFoldCandidate: isFoldCandidate(tagName, classList, role, type),
+        childElementCount: 0,
+        pendingPaths: [],
+    };
+}

package/dist/derive-path-group-key.d.ts ADDED Viewed

@@ -0,0 +1,44 @@
+/**
+ * Derives a coarse grouping key from a page's URL path segments (e.g. the
+ * `paths` field of `@d-zero/shared/parse-url`'s `ExURL`), keeping only the
+ * leading `depth` segments. This is a *blocking key* in the record-linkage
+ * sense: a cheap, coarse partition applied before any expensive structural
+ * comparison (`jaccardSimilarity`/`arrayEditDistance` on `tokenize()`
+ * output), not a similarity score by itself. Real-data validation on a large
+ * multi-section site found that a single site-wide document-frequency
+ * computation ({@link ./compute-document-frequency.js | computeDocumentFrequency})
+ * fails when the site is actually a federation of independently-templated
+ * sub-sections; splitting pages by their top-level URL segment first and
+ * computing frequency per group recovered a working split. This function
+ * produces that split key.
+ *
+ * Deliberately returns only this one signal rather than merging it with
+ * other blocking signals (e.g. a stylesheet-based key) into a single
+ * composite key: literature on entity-resolution blocking (e.g. Michelson &
+ * Knoblock's DNF blocking scheme) finds that combining independent blocking
+ * predicates with AND into one key is inferior to keeping them independent
+ * and combining candidate pairs with OR — that combination decision belongs
+ * to the caller that actually groups pages, not to this function.
+ *
+ * Empty segments anywhere in `paths` are dropped before slicing, so the
+ * trailing `''` that `ExURL.paths` produces for a directory-style URL (one
+ * ending in `/`) doesn't fragment a section's key from the same section's
+ * non-trailing-slash URLs.
+ *
+ * `depth` must be a positive integer. A non-positive or fractional depth has
+ * no sensible interpretation as "how many leading segments to keep", so it
+ * is rejected eagerly rather than silently coerced or left to produce a
+ * confusing result (e.g. slicing with a negative or fractional length).
+ * @param paths
+ * @param depth
+ * @example
+ * ```ts
+ * derivePathGroupKey(['dept-a', 'news', '123']);
+ * // 'dept-a'
+ * derivePathGroupKey(['dept-a', 'news', '123'], 2);
+ * // 'dept-a/news'
+ * derivePathGroupKey([]);
+ * // ''
+ * ```
+ */
+export declare function derivePathGroupKey(paths: readonly string[], depth?: number): string;

package/dist/derive-path-group-key.js ADDED Viewed

@@ -0,0 +1,51 @@
+/**
+ * Derives a coarse grouping key from a page's URL path segments (e.g. the
+ * `paths` field of `@d-zero/shared/parse-url`'s `ExURL`), keeping only the
+ * leading `depth` segments. This is a *blocking key* in the record-linkage
+ * sense: a cheap, coarse partition applied before any expensive structural
+ * comparison (`jaccardSimilarity`/`arrayEditDistance` on `tokenize()`
+ * output), not a similarity score by itself. Real-data validation on a large
+ * multi-section site found that a single site-wide document-frequency
+ * computation ({@link ./compute-document-frequency.js | computeDocumentFrequency})
+ * fails when the site is actually a federation of independently-templated
+ * sub-sections; splitting pages by their top-level URL segment first and
+ * computing frequency per group recovered a working split. This function
+ * produces that split key.
+ *
+ * Deliberately returns only this one signal rather than merging it with
+ * other blocking signals (e.g. a stylesheet-based key) into a single
+ * composite key: literature on entity-resolution blocking (e.g. Michelson &
+ * Knoblock's DNF blocking scheme) finds that combining independent blocking
+ * predicates with AND into one key is inferior to keeping them independent
+ * and combining candidate pairs with OR — that combination decision belongs
+ * to the caller that actually groups pages, not to this function.
+ *
+ * Empty segments anywhere in `paths` are dropped before slicing, so the
+ * trailing `''` that `ExURL.paths` produces for a directory-style URL (one
+ * ending in `/`) doesn't fragment a section's key from the same section's
+ * non-trailing-slash URLs.
+ *
+ * `depth` must be a positive integer. A non-positive or fractional depth has
+ * no sensible interpretation as "how many leading segments to keep", so it
+ * is rejected eagerly rather than silently coerced or left to produce a
+ * confusing result (e.g. slicing with a negative or fractional length).
+ * @param paths
+ * @param depth
+ * @example
+ * ```ts
+ * derivePathGroupKey(['dept-a', 'news', '123']);
+ * // 'dept-a'
+ * derivePathGroupKey(['dept-a', 'news', '123'], 2);
+ * // 'dept-a/news'
+ * derivePathGroupKey([]);
+ * // ''
+ * ```
+ */
+export function derivePathGroupKey(paths, depth = 1) {
+    if (!(Number.isInteger(depth) && depth > 0)) {
+        throw new RangeError(`derivePathGroupKey: depth must be a positive integer, got ${depth}`);
+    }
+    // See the trailing-slash note above.
+    const segments = paths.filter((segment) => segment !== '');
+    return segments.slice(0, depth).join('/');
+}