@d-zero/beholder 3.1.6 → 4.0.0

package/CHANGELOG.md

@@ -3,6 +3,16 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [4.0.0](https://github.com/d-zero-dev/tools/compare/@d-zero/beholder@3.1.6...@d-zero/beholder@4.0.0) (2026-07-21)
+
+- feat(beholder)!: enrich PageData with mainContents metrics and scrollHeight ([8a0fc72](https://github.com/d-zero-dev/tools/commit/8a0fc72412295a92850b2c2fa2fa8cd8145eb5f6))
+
+### BREAKING CHANGES
+
+- PageData now requires mainContents and scrollHeight fields.
+
+Co-authored-by: Cursor <cursoragent@cursor.com>
+
 ## [3.1.6](https://github.com/d-zero-dev/tools/compare/@d-zero/beholder@3.1.5...@d-zero/beholder@3.1.6) (2026-07-14)
 
 **Note:** Version bump only for package @d-zero/beholder

package/README.md

@@ -1,6 +1,6 @@
 # `@d-zero/beholder`
 
-Puppeteer の `Page` を受け取り、単一ページのメタデータ・リンク・画像・ネットワークリソースを収集するインプロセス型スクレイパー。結果は `ScrapeResult` として戻り値で返却される（イベント経由ではない）。ブラウザ管理は呼び出し側の責任。
+Puppeteer の `Page` を受け取り、単一ページのメタデータ・リンク・画像・ネットワークリソースに加え、メインコンテンツの定量データと desktop/mobile の `scrollHeight` を収集するインプロセス型スクレイパー。結果は `ScrapeResult` として戻り値で返却される（イベント経由ではない）。ブラウザ管理は呼び出し側の責任。
 
 ## Installation
 
@@ -26,13 +26,48 @@ const result = await scraper.scrapeStart(page, parseUrl('https://example.com'),
 	isExternal: false,
 });
 
-if (result.type === 'success') {
-	console.log(result.pageData?.meta.title);
+if (result.type === 'success' && result.pageData) {
+	console.log(result.pageData.meta.title);
+	console.log(result.pageData.mainContents?.wordCount);
+	console.log(result.pageData.scrollHeight);
 }
 ```
 
 設計判断（イベントではなく戻り値で返す理由、`page` のライフサイクル責務、リトライ機構など）は `src/scraper.ts` の JSDoc を参照。
 
+## 取得フロー
+
+```mermaid
+flowchart TD
+  goto[page.goto]
+  ct{content-type is text/html?}
+  early[both null]
+  html[get HTML]
+  ext{isExternal?}
+  extOut[title-only; both null]
+  idle[networkidle]
+  mc["page.evaluate: mainContents"]
+  anchors[getAnchorList / getMeta]
+  imgs{captureImages?}
+  fetchImg["#fetchImages + collect scrollHeight"]
+  light["measureScrollHeight light path"]
+  done[return PageData]
+
+  goto --> ct
+  ct -->|no| early
+  ct -->|yes| html --> ext
+  ext -->|yes| extOut
+  ext -->|no| idle --> mc --> anchors --> imgs
+  imgs -->|yes| fetchImg --> done
+  imgs -->|no| light --> done
+```
+
+## 返却データ（`PageData` の追加分）
+
+- `mainContents`: メイン領域の定量（`wordCount` / `bodyWordCount`、見出し・画像・表・ボタン・iframe / video / audio / canvas の配列、検出した `main` の識別情報）。メイン未検出時もオブジェクトは返り、配列は空・`wordCount` は 0（非 HTML 等で取得しなかったときだけ `null`）
+- `scrollHeight`: `{ desktop, mobile }`（各 `number | null`）。未計測時はフィールド全体が `null`
+- 配列要素のフィールド詳細は型（`MainContents*` / `ScrollHeightData`）を参照
+
 ## DOM 文字列からメタ抽出（Puppeteer なし）
 
 HTML 文字列を jsdom などでパースしてから `Meta` を取り出したい場合、`extractMetaFromDocument` を使う。`Scraper` が内部で呼ぶ `collectHead → detectTags → classify` パイプラインと同じ実装を再利用するため、戻り値の `Meta` 形状は `scrapeStart` と同一。DOM ライブラリ（jsdom 等）はユーザランドの責務。

package/dist/get-main-contents.d.ts

@@ -0,0 +1,48 @@
+/**
+ * Main-content quantitative extraction for scraped HTML pages.
+ *
+ * Runs entirely against a `Document` (Puppeteer page realm or jsdom) so the
+ * same logic is unit-tested without a browser. Puppeteer callers use
+ * {@link getMainContents}, which passes this function into `page.evaluate`.
+ *
+ * WHY no `@medv/finder`: selector strings are diagnostic only; a tag+id+class // cspell:disable-line
+ * path is enough and avoids a Node-only dependency inside the page realm.
+ * @module
+ */
+import type { MainContentsData } from './types.js';
+import type { Page } from 'puppeteer';
+/**
+ * Extract main-content metrics from a `Document`.
+ *
+ * This function must remain free of closure over imports so Puppeteer can
+ * serialize it into `page.evaluate`.
+ *
+ * Argument order matches `page.evaluate(fn, mainContentSelector)`: the selector
+ * is the first argument; `doc` defaults to the page-realm global `document`
+ * when omitted (jsdom tests pass an explicit `Document` as the second argument).
+ * @param mainContentSelector - Optional selector prepended to the default list.
+ * @param doc - The document to inspect (defaults to the global `document`).
+ * @returns Quantitative main-content data (never `null`; empty when no main region).
+ * @example
+ * ```ts
+ * // In page.evaluate / browser:
+ * const data = extractMainContentsFromDocument('#page-body');
+ * // In jsdom tests:
+ * const data = extractMainContentsFromDocument('#page-body', document);
+ * ```
+ */
+export declare function extractMainContentsFromDocument(mainContentSelector?: string | null, doc?: Document): MainContentsData;
+/**
+ * Extract main-content metrics from a Puppeteer page via a single `page.evaluate`.
+ * @param page - Puppeteer page whose DOM has finished loading.
+ * @param options - Optional main-content selector override.
+ * @param options.mainContentSelector
+ * @returns Quantitative main-content data.
+ * @example
+ * ```ts
+ * const mainContents = await getMainContents(page, { mainContentSelector: '#page-body' });
+ * ```
+ */
+export declare function getMainContents(page: Page, options?: {
+    mainContentSelector?: string | null;
+}): Promise<MainContentsData>;

package/dist/get-main-contents.js

@@ -0,0 +1,270 @@
+/**
+ * Main-content quantitative extraction for scraped HTML pages.
+ *
+ * Runs entirely against a `Document` (Puppeteer page realm or jsdom) so the
+ * same logic is unit-tested without a browser. Puppeteer callers use
+ * {@link getMainContents}, which passes this function into `page.evaluate`.
+ *
+ * WHY no `@medv/finder`: selector strings are diagnostic only; a tag+id+class // cspell:disable-line
+ * path is enough and avoids a Node-only dependency inside the page realm.
+ * @module
+ */
+/**
+ * Extract main-content metrics from a `Document`.
+ *
+ * This function must remain free of closure over imports so Puppeteer can
+ * serialize it into `page.evaluate`.
+ *
+ * Argument order matches `page.evaluate(fn, mainContentSelector)`: the selector
+ * is the first argument; `doc` defaults to the page-realm global `document`
+ * when omitted (jsdom tests pass an explicit `Document` as the second argument).
+ * @param mainContentSelector - Optional selector prepended to the default list.
+ * @param doc - The document to inspect (defaults to the global `document`).
+ * @returns Quantitative main-content data (never `null`; empty when no main region).
+ * @example
+ * ```ts
+ * // In page.evaluate / browser:
+ * const data = extractMainContentsFromDocument('#page-body');
+ * // In jsdom tests:
+ * const data = extractMainContentsFromDocument('#page-body', document);
+ * ```
+ */
+export function extractMainContentsFromDocument(mainContentSelector = null, doc = document) {
+    /**
+     * @param text
+     */
+    function removeSpaces(text) {
+        return text?.trim().replaceAll(/\s+/g, '') || null;
+    }
+    /**
+     * @param value
+     */
+    function cssEscape(value) {
+        if (typeof CSS !== 'undefined' && typeof CSS.escape === 'function') {
+            return CSS.escape(value);
+        }
+        return value.replaceAll(/([ !"#$%&'()*+,./:;<=>?@[\\\]^`{|}~])/g, '\\$1');
+    }
+    /**
+     * @param el
+     */
+    function buildSelector(el) {
+        const tag = el.nodeName.toLowerCase();
+        const id = el.id ? `#${cssEscape(el.id)}` : '';
+        const classes = [...el.classList].map((c) => `.${cssEscape(c)}`).join('');
+        return `${tag}${id}${classes}`;
+    }
+    /**
+     * @param media
+     */
+    function resolveMediaSrc(media) {
+        if (media.currentSrc) {
+            return media.currentSrc;
+        }
+        const attrSrc = media.getAttribute('src');
+        if (attrSrc) {
+            try {
+                return new URL(attrSrc, doc.baseURI).href;
+            }
+            catch {
+                return attrSrc;
+            }
+        }
+        const source = media.querySelector('source[src]');
+        const sourceSrc = source?.getAttribute('src');
+        if (sourceSrc) {
+            try {
+                return new URL(sourceSrc, doc.baseURI).href;
+            }
+            catch {
+                return sourceSrc;
+            }
+        }
+        return '';
+    }
+    const title = doc.title?.trim() ?? '';
+    const bodyWordCount = removeSpaces(doc.body?.textContent ?? null)?.length ?? 0;
+    const selectors = [
+        'main',
+        '[role="main"]',
+        '#main',
+        '.main',
+        '#content',
+        '.content',
+        '#contents',
+        '.contents',
+        '#main-content',
+        '.main-content',
+        '#main_content',
+        '.main_content',
+        '#mainContent',
+        '.mainContent',
+    ];
+    if (mainContentSelector) {
+        selectors.unshift(mainContentSelector);
+    }
+    let $main = null;
+    try {
+        $main = doc.querySelector(selectors.join(','));
+    }
+    catch {
+        // Invalid custom selector: retry without it so built-in selectors still run.
+        if (mainContentSelector) {
+            selectors.shift();
+            $main = doc.querySelector(selectors.join(','));
+        }
+    }
+    if (!$main) {
+        const fallbackSelectors = [
+            '[id*="main" i]',
+            '[class*="main" i]',
+            '[id*="content" i]',
+            '[class*="content" i]',
+        ];
+        for (const sel of fallbackSelectors) {
+            const candidate = doc.querySelector(sel);
+            if (candidate && candidate !== doc.body && candidate !== doc.documentElement) {
+                $main = candidate;
+                break;
+            }
+        }
+    }
+    if (!$main) {
+        return {
+            title,
+            main: null,
+            wordCount: 0,
+            bodyWordCount,
+            headings: [],
+            images: [],
+            tables: [],
+            buttons: [],
+            iframes: [],
+            videos: [],
+            audios: [],
+            canvases: [],
+        };
+    }
+    const headings = [];
+    for (const $heading of $main.querySelectorAll('h1, h2, h3, h4, h5, h6')) {
+        headings.push({
+            text: removeSpaces($heading.textContent),
+            level: Number.parseInt($heading.nodeName.replace(/h/i, ''), 10),
+        });
+    }
+    const images = [];
+    for (const $img of $main.querySelectorAll('img, input[type="image"]')) {
+        images.push({
+            src: $img.src,
+            alt: $img.alt,
+        });
+    }
+    const tables = [];
+    for (const $table of $main.querySelectorAll('table')) {
+        tables.push({
+            rows: $table.querySelectorAll('tr').length,
+            cols: $table.querySelector('tr')?.querySelectorAll('th, td').length || 0,
+            hasHeader: !!$table.querySelector('thead'),
+            hasFooter: !!$table.querySelector('tfoot'),
+            hasMergedCell: !!$table.querySelector('[colspan], [rowspan]'),
+        });
+    }
+    const win = doc.defaultView;
+    const HTMLButton = win?.HTMLButtonElement;
+    const HTMLInput = win?.HTMLInputElement;
+    const buttons = [];
+    for (const $el of $main.querySelectorAll('button, [role="button"], [class*="button"], [class*="btn"]')) {
+        const isButton = HTMLButton ? $el instanceof HTMLButton : $el.nodeName === 'BUTTON';
+        const isInput = HTMLInput ? $el instanceof HTMLInput : $el.nodeName === 'INPUT';
+        const isFormControl = isButton || isInput;
+        let text = removeSpaces($el.textContent);
+        if (!text && isInput) {
+            text = removeSpaces($el.getAttribute('value'));
+        }
+        const disabledProp = isFormControl && 'disabled' in $el
+            ? Boolean($el.disabled)
+            : false;
+        buttons.push({
+            nodeName: $el.nodeName,
+            role: $el.getAttribute('role'),
+            type: isFormControl ? $el.type : null,
+            text,
+            disabled: disabledProp || $el.getAttribute('aria-disabled') === 'true',
+        });
+    }
+    const iframes = [];
+    for (const $el of $main.querySelectorAll('iframe')) {
+        iframes.push({
+            src: $el.src,
+            title: $el.hasAttribute('title') ? $el.getAttribute('title') : null,
+            width: $el.getAttribute('width'),
+            height: $el.getAttribute('height'),
+        });
+    }
+    const videos = [];
+    for (const $el of $main.querySelectorAll('video')) {
+        const posterAttr = $el.getAttribute('poster');
+        let poster = null;
+        if (posterAttr) {
+            try {
+                poster = new URL(posterAttr, doc.baseURI).href;
+            }
+            catch {
+                poster = posterAttr;
+            }
+        }
+        videos.push({
+            src: resolveMediaSrc($el),
+            poster,
+            width: $el.width,
+            height: $el.height,
+        });
+    }
+    const audios = [];
+    for (const $el of $main.querySelectorAll('audio')) {
+        audios.push({
+            src: resolveMediaSrc($el),
+        });
+    }
+    const canvases = [];
+    for (const $el of $main.querySelectorAll('canvas')) {
+        canvases.push({
+            width: $el.width,
+            height: $el.height,
+        });
+    }
+    return {
+        title,
+        main: {
+            nodeName: $main.nodeName,
+            id: $main.id || null,
+            classList: [...$main.classList],
+            role: $main.getAttribute('role'),
+            selector: buildSelector($main),
+        },
+        wordCount: removeSpaces($main.textContent)?.length ?? 0,
+        bodyWordCount,
+        headings,
+        images,
+        tables,
+        buttons,
+        iframes,
+        videos,
+        audios,
+        canvases,
+    };
+}
+/**
+ * Extract main-content metrics from a Puppeteer page via a single `page.evaluate`.
+ * @param page - Puppeteer page whose DOM has finished loading.
+ * @param options - Optional main-content selector override.
+ * @param options.mainContentSelector
+ * @returns Quantitative main-content data.
+ * @example
+ * ```ts
+ * const mainContents = await getMainContents(page, { mainContentSelector: '#page-body' });
+ * ```
+ */
+export async function getMainContents(page, options) {
+    return page.evaluate(extractMainContentsFromDocument, options?.mainContentSelector ?? null);
+}

package/dist/index.d.ts

@@ -20,4 +20,4 @@ export { detectCDN } from '@d-zero/shared/detect-cdn';
 export type { CDNType } from '@d-zero/shared/detect-cdn';
 export type { ScrapeResult, ResourceEntry, PageData } from './types.js';
 export type { ScraperOptions, ChangePhaseEvent, ScraperEventTypes } from './types.js';
-export type { Resource, AnchorData, Meta, ImageElement, SkippedPageData, NetworkLog, OpenGraphMeta, OgArticleMeta, OgBookMeta, OgProfileMeta, OgMusicMeta, OgVideoNsMeta, TwitterMeta, FbMeta, FediverseMeta, AppleMeta, MsApplicationMeta, VerificationMeta, GoogleMeta, GeoMeta, CitationMeta, RdfaMeta, MicrodataMeta, AmpMeta, LegacyMeta, MobileMeta, MicroformatsMeta, PinterestMeta, SlackMeta, LinkedInMeta, ExperimentalMeta, WikiMeta, LinkMeta, LinkEntry, JsonLdEntry, OthersBucket, ScriptEntry, IframeEntry, TagsMeta, TagDetail, TagEntry, TagSource, ViewportMeta, RobotsMeta, ReferrerMeta, FormatDetectionMeta, HttpEquivMeta, HttpEquivRefresh, RawHeadEntry, } from './types.js';
+export type { Resource, AnchorData, Meta, ImageElement, SkippedPageData, NetworkLog, ScrollHeightData, MainContentsData, MainContentsMainTag, MainContentsHeading, MainContentsImage, MainContentsTable, MainContentsButton, MainContentsIframe, MainContentsVideo, MainContentsAudio, MainContentsCanvas, OpenGraphMeta, OgArticleMeta, OgBookMeta, OgProfileMeta, OgMusicMeta, OgVideoNsMeta, TwitterMeta, FbMeta, FediverseMeta, AppleMeta, MsApplicationMeta, VerificationMeta, GoogleMeta, GeoMeta, CitationMeta, RdfaMeta, MicrodataMeta, AmpMeta, LegacyMeta, MobileMeta, MicroformatsMeta, PinterestMeta, SlackMeta, LinkedInMeta, ExperimentalMeta, WikiMeta, LinkMeta, LinkEntry, JsonLdEntry, OthersBucket, ScriptEntry, IframeEntry, TagsMeta, TagDetail, TagEntry, TagSource, ViewportMeta, RobotsMeta, ReferrerMeta, FormatDetectionMeta, HttpEquivMeta, HttpEquivRefresh, RawHeadEntry, } from './types.js';

package/dist/is-html-content-type.d.ts

@@ -0,0 +1,18 @@
+/**
+ * Determine whether a Content-Type media type is HTML.
+ *
+ * MIME types are case-insensitive (RFC 2045). Surrounding whitespace after
+ * parameter stripping is also tolerated so `text/html `, `Text/HTML`, and
+ * `text/html; charset=utf-8` (after `;` split) all classify as HTML.
+ * @param contentType - The media type portion of a Content-Type header
+ *   (parameters already stripped), or `null` when unknown.
+ * @returns `true` when the media type is `text/html` in any letter case.
+ * @example
+ * ```ts
+ * isHtmlContentType('text/html'); // true
+ * isHtmlContentType('Text/HTML'); // true
+ * isHtmlContentType('application/pdf'); // false
+ * // Callers should strip parameters first (e.g. split on ';') before calling.
+ * ```
+ */
+export declare function isHtmlContentType(contentType: string | null): boolean;

package/dist/is-html-content-type.js

@@ -0,0 +1,20 @@
+/**
+ * Determine whether a Content-Type media type is HTML.
+ *
+ * MIME types are case-insensitive (RFC 2045). Surrounding whitespace after
+ * parameter stripping is also tolerated so `text/html `, `Text/HTML`, and
+ * `text/html; charset=utf-8` (after `;` split) all classify as HTML.
+ * @param contentType - The media type portion of a Content-Type header
+ *   (parameters already stripped), or `null` when unknown.
+ * @returns `true` when the media type is `text/html` in any letter case.
+ * @example
+ * ```ts
+ * isHtmlContentType('text/html'); // true
+ * isHtmlContentType('Text/HTML'); // true
+ * isHtmlContentType('application/pdf'); // false
+ * // Callers should strip parameters first (e.g. split on ';') before calling.
+ * ```
+ */
+export function isHtmlContentType(contentType) {
+    return contentType !== null && contentType.trim().toLowerCase() === 'text/html';
+}

package/dist/measure-scroll-height.d.ts

@@ -0,0 +1,23 @@
+/**
+ * Lightweight scrollHeight measurement without `scrollAllOver`.
+ *
+ * Used when `captureImages` is false so callers still receive desktop/mobile
+ * heights without the cost of full-page scrolling for image extraction.
+ * @module
+ */
+import type { ScrollHeightData } from './types.js';
+import type { Page } from 'puppeteer';
+/**
+ * Measure `document.body.scrollHeight` at desktop-compact and mobile-small viewports.
+ *
+ * Sets each viewport, reads height once, and does **not** call `scrollAllOver`.
+ * A failed preset yields `null` for that side only.
+ * @param page - Puppeteer page whose DOM has finished loading.
+ * @returns Heights for desktop and mobile presets.
+ * @example
+ * ```ts
+ * const scrollHeight = await measureScrollHeight(page);
+ * // { desktop: 2400, mobile: 5200 }
+ * ```
+ */
+export declare function measureScrollHeight(page: Page): Promise<ScrollHeightData>;

package/dist/measure-scroll-height.js

@@ -0,0 +1,44 @@
+/**
+ * Lightweight scrollHeight measurement without `scrollAllOver`.
+ *
+ * Used when `captureImages` is false so callers still receive desktop/mobile
+ * heights without the cost of full-page scrolling for image extraction.
+ * @module
+ */
+import { devicePresets } from '@d-zero/puppeteer-page-scan';
+/**
+ * Measure `document.body.scrollHeight` at desktop-compact and mobile-small viewports.
+ *
+ * Sets each viewport, reads height once, and does **not** call `scrollAllOver`.
+ * A failed preset yields `null` for that side only.
+ * @param page - Puppeteer page whose DOM has finished loading.
+ * @returns Heights for desktop and mobile presets.
+ * @example
+ * ```ts
+ * const scrollHeight = await measureScrollHeight(page);
+ * // { desktop: 2400, mobile: 5200 }
+ * ```
+ */
+export async function measureScrollHeight(page) {
+    const desktop = await measureAtPreset(page, 'desktop-compact');
+    const mobile = await measureAtPreset(page, 'mobile-small');
+    return { desktop, mobile };
+}
+/**
+ * @param page
+ * @param key
+ */
+async function measureAtPreset(page, key) {
+    const preset = devicePresets[key];
+    try {
+        await page.setViewport({
+            width: preset.width,
+            height: 800,
+            deviceScaleFactor: preset.resolution ?? 1,
+        });
+        return await page.evaluate(() => document.body.scrollHeight);
+    }
+    catch {
+        return null;
+    }
+}