@d-zero/beholder 2.1.5 → 3.0.0

package/CHANGELOG.md

@@ -3,6 +3,50 @@
 All notable changes to this project will be documented in this file.
 See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
 
+# [3.0.0](https://github.com/d-zero-dev/tools/compare/@d-zero/beholder@2.1.6...@d-zero/beholder@3.0.0) (2026-06-16)
+
+### Bug Fixes
+
+- **beholder:** warn loudly and tripwire-test puppeteer Page.\_client() coverage ([97a07ea](https://github.com/d-zero-dev/tools/commit/97a07ea273e90d50bfede1d68f594ddee9c33268))
+
+- feat(beholder)!: expand meta extraction with frontmatter-keys schema and Wappalyzer tag detection ([6ee7861](https://github.com/d-zero-dev/tools/commit/6ee78617aac3fe3d5c022ccfd0df265de0c5310b))
+
+### Features
+
+- **beholder:** rewrite getAnchorList with single AX tree + parallel describeNode ([#876](https://github.com/d-zero-dev/tools/issues/876)) ([7e5b089](https://github.com/d-zero-dev/tools/commit/7e5b089695bd1e605d63c6faef2e8bf927bd861f))
+
+### BREAKING CHANGES
+
+- `Meta` is restructured from flat keys (`noindex`, `canonical`,
+  `'og:type'`, `'twitter:card'`, ...) into a nested shape backed by
+  `frontmatter-keys.md`. New required fields: `title`, `jsonLd`,
+  `speculationRules`, `originTrial`, `tags`, `others`. `getMeta(page)` now takes
+  a context object `getMeta(page, { url, html?, statusCode?, headers? }, timeout?)`.
+  Old top-level shortcuts (`canonical`, `alternate`, `noindex`, `nofollow`,
+  `noarchive`, `'og:*'`, `'twitter:card'`) are removed; values move to
+  `meta.link.canonical`, `meta.robots.*`, `meta.og.*`, `meta.twitter.*` etc.
+
+Changes:
+
+- New `src/meta/` module: `types.ts`, `keys.ts`, `parsers.ts`, `classify.ts`,
+  `id-extractors.ts`, `tag-detection.ts`, plus ambient `simple-wappalyzer.d.ts`
+- Browser-side `collectHead()` serializes every `<meta>`, `<link>`, structured-data
+  `<script>`, `<base>`, `<iframe>` plus a curated set of `window` globals into
+  `RawHeadEntry[]`; Node-side `classify()` maps these to typed Meta fields
+- `simple-wappalyzer` (MIT) added as a dependency for technology detection;
+  detected providers run through `id-extractors.ts` for real ID extraction
+  (GA4, GTM, UA, FB Pixel, Hotjar, Clarity, ...)
+- Unknown markup is preserved under `Meta.others` (meta/property/httpEquiv/
+  itemprop/link/script/iframe buckets) so nothing is silently dropped
+- Tests: parsers/classify/id-extractors/tag-detection units + getMeta
+  error/timeout fallback
+
+## [2.1.6](https://github.com/d-zero-dev/tools/compare/@d-zero/beholder@2.1.5...@d-zero/beholder@2.1.6) (2026-06-15)
+
+### Bug Fixes
+
+- **beholder:** prevent getMeta from hanging on unresponsive pages ([f55bb26](https://github.com/d-zero-dev/tools/commit/f55bb261c1868b40709cbae6aa17d273c5516e74)), closes [#874](https://github.com/d-zero-dev/tools/issues/874)
+
 ## [2.1.5](https://github.com/d-zero-dev/tools/compare/@d-zero/beholder@2.1.4...@d-zero/beholder@2.1.5) (2026-05-27)
 
 **Note:** Version bump only for package @d-zero/beholder

package/README.md

@@ -1,58 +1,16 @@
 # `@d-zero/beholder`
 
-Puppeteer を使用してWebページをスクレイピングし、メタデータ・リンク・画像・ネットワークリソースを収集するライブラリです。
+Puppeteer の `Page` を受け取り、単一ページのメタデータ・リンク・画像・ネットワークリソースを収集するインプロセス型スクレイパー。結果は `ScrapeResult` として戻り値で返却される（イベント経由ではない）。ブラウザ管理は呼び出し側の責任。
 
-**名前の由来**: **[Beholder](https://forgottenrealms.fandom.com/wiki/Beholder)** という名前は、多くの目を持つ神話上の生物に由来しています。この生物のように、このツールはWebページを正確かつ徹底的にスクレイピングして記録し、詳細を見逃しません。
+## Installation
 
-## インストール
-
-```bash
+```sh
 yarn add @d-zero/beholder
 ```
 
-## 概要
-
-`@d-zero/beholder` は、Puppeteer の `Page` オブジェクトを受け取り、単一ページのスクレイピングを行うインプロセス型のスクレイパーです。
-
-**主な特徴:**
-
-- 結果は `ScrapeResult` として戻り値で返却（イベント経由ではない）
-- ストリーミングイベント（`changePhase`, `resourceResponse`）で進捗を監視可能
-- キーワード・パス除外によるページスキップ
-- 複数デバイスプリセット対応のレスポンシブ画像キャプチャ
-- ブラウザ管理は呼び出し側の責任（Scraperはページ操作のみ）
-
-## エクスポートされるAPI
+## Usage
 
-### `Scraper`（デフォルトエクスポート）
-
-ページレベルのスクレイパークラスです。`TypedAwaitEventEmitter` を継承しています。
-
-```typescript
-import Scraper from '@d-zero/beholder';
-```
-
-#### `scrapeStart(page, url, options?, isSkip?)`
-
-Puppeteer ページ上でスクレイピングを実行します。
-
-**パラメータ:**
-
-- `page` (`Page`) — Puppeteer ページインスタンス
-- `url` (`ExURL`) — スクレイピング対象のURL
-- `options` (`Partial<ScraperOptions>`, 省略可) — スクレイピングオプション
-- `isSkip` (`boolean`, 省略可) — `true` でネットワークリクエストなしにスキップ
-
-**戻り値:** `Promise<ScrapeResult>`
-
-- `type: "success"` — `pageData` にスクレイピング結果を格納
-- `type: "skipped"` — `ignored` にスキップ理由を格納
-- `type: "error"` — `error` にエラー詳細を格納
-- `failedRequests` — ネットワーク切断等で失敗したサブリソースリクエストの一覧（存在する場合のみ）
-
-**使用例:**
-
-```typescript
+```ts
 import Scraper from '@d-zero/beholder';
 import { parseUrl } from '@d-zero/shared/parse-url';
 import { launch } from 'puppeteer';
@@ -61,241 +19,16 @@ const browser = await launch();
 const page = await browser.newPage();
 
 const scraper = new Scraper();
+scraper.on('changePhase', (event) => console.log(event.message));
 
-// 進捗イベントを監視
-scraper.on('changePhase', async (event) => {
-	console.log(`[${event.pid}] ${event.name}: ${event.message}`);
-});
-
-// スクレイピングを実行
-const url = parseUrl('https://example.com');
-const result = await scraper.scrapeStart(page, url, {
+const result = await scraper.scrapeStart(page, parseUrl('https://example.com'), {
 	captureImages: true,
-	excludeKeywords: ['広告'],
 	isExternal: false,
 });
 
 if (result.type === 'success') {
-	console.log('タイトル:', result.pageData?.meta.title);
-	console.log('リンク数:', result.pageData?.anchorList.length);
-	console.log('画像数:', result.pageData?.imageList.length);
-	console.log('サブリソース数:', result.resources.length);
+	console.log(result.pageData?.meta.title);
 }
-
-// クリーンアップはブラウザレベルで行う
-await page.close();
-await browser.close();
-```
-
-#### イベント
-
-| イベント名         | 説明                                           |
-| ------------------ | ---------------------------------------------- |
-| `changePhase`      | スクレイピングフェーズが遷移した場合           |
-| `resourceResponse` | サブリソースのレスポンスがキャプチャされた場合 |
-
-### `ScraperOptions`
-
-| プロパティ          | 型         | デフォルト | 説明                                                             |
-| ------------------- | ---------- | ---------- | ---------------------------------------------------------------- |
-| `isExternal`        | `boolean`  | `false`    | 外部URLかどうか                                                  |
-| `captureImages`     | `boolean`  | `true`     | 画像データを取得するかどうか                                     |
-| `excludeKeywords`   | `string[]` | `[]`       | HTML内にマッチしたらスキップするキーワード                       |
-| `metadataOnly`      | `boolean`  | `false`    | メタデータのみ取得（ブラウザスクレイピングなし）                 |
-| `imageLoadTimeout`  | `number`   | `5000`     | 画像読み込み待機のタイムアウト（ms）                             |
-| `disableQueries`    | `boolean`  | -          | URLパース時にクエリパラメータを除去するかどうか                  |
-| `retries`           | `number`   | -          | ネットワーク操作のリトライ回数                                   |
-| `headCheckResult`   | `PageData` | -          | 事前取得したHEADチェック結果（省略時はHEADリクエストをスキップ） |
-| `navigationTimeout` | `number`   | `60000`    | `page.goto()` のタイムアウト（ms）                               |
-
-### ユーティリティ関数
-
-#### `isError(status)`
-
-HTTPステータスコードがエラーかどうかを判定します。200-399 は成功、それ以外はエラーです。
-
-```typescript
-import { isError } from '@d-zero/beholder';
-
-isError(200); // false
-isError(404); // true
-```
-
-#### `detectCompress(headers)` / `detectCDN(headers)`
-
-レスポンスヘッダーから圧縮方式・CDNプロバイダを検出します（`@d-zero/shared` からの再エクスポート）。
-
-### 型定義
-
-#### `ScrapeResult`
-
-スクレイピング操作の結果を表します。
-
-```typescript
-type ScrapeResult = {
-	type: 'success' | 'skipped' | 'error';
-	pageData?: PageData;
-	resources: ResourceEntry[];
-	ignored?: { url: ExURL; matchedText: string; excludeKeywords: string[] };
-	error?: { name: string; message: string; stack?: string; shutdown: boolean };
-	failedRequests?: ReadonlyArray<{ url: string; errorText: string }>;
-};
 ```
 
-#### `PageData`
-
-スクレイピング成功時のページデータです。
-
-```typescript
-type PageData = {
-	url: ExURL;
-	redirectPaths: string[];
-	isTarget: boolean;
-	isExternal: boolean;
-	status: number;
-	statusText: string;
-	contentType: string | null;
-	contentLength: number | null;
-	responseHeaders: Record<string, string | string[] | undefined> | null;
-	meta: Meta;
-	anchorList: AnchorData[];
-	imageList: ImageElement[];
-	html: string;
-	isSkipped: false;
-};
-```
-
-#### `Meta`
-
-ページの `<head>` から抽出されたメタデータです。
-
-```typescript
-type Meta = {
-	lang?: string;
-	title: string;
-	description?: string;
-	keywords?: string;
-	noindex?: boolean;
-	nofollow?: boolean;
-	noarchive?: boolean;
-	canonical?: string;
-	alternate?: string;
-	'og:type'?: string;
-	'og:title'?: string;
-	'og:site_name'?: string;
-	'og:description'?: string;
-	'og:url'?: string;
-	'og:image'?: string;
-	'twitter:card'?: string;
-};
-```
-
-#### `AnchorData`
-
-アンカー要素（`<a>` / `<area>`）のデータです。
-
-```typescript
-type AnchorData = {
-	href: ExURL;
-	textContent: string;
-	isExternal?: boolean;
-};
-```
-
-#### `ImageElement`
-
-画像要素のデータです。デバイスプリセットごとにキャプチャされます。
-
-```typescript
-type ImageElement = {
-	src: string;
-	currentSrc: string;
-	alt: string;
-	width: number;
-	height: number;
-	naturalWidth: number;
-	naturalHeight: number;
-	isLazy: boolean;
-	viewportWidth: number;
-	sourceCode: string;
-};
-```
-
-#### `ResourceEntry`
-
-ページ読み込み中にキャプチャされたサブリソースです。
-
-```typescript
-type ResourceEntry = {
-	log: NetworkLog;
-	resource: Omit<Resource, 'uid'>;
-	pageUrl: string;
-};
-```
-
-#### `NetworkLog`
-
-ネットワークリクエスト/レスポンスのログエントリです。
-
-```typescript
-type NetworkLog = {
-	url: ExURL;
-	status: number | null;
-	contentLength: number;
-	contentType: string;
-	isError: boolean;
-	request: { ts: number; headers: Record<string, string>; method: string };
-	response?: {
-		ts: number;
-		status: number;
-		statusText: string;
-		fromCache: boolean;
-		headers: Record<string, string>;
-	};
-};
-```
-
-#### `Resource`
-
-ネットワークリソースのメタデータです。
-
-```typescript
-type Resource = {
-	url: ExURL;
-	isExternal: boolean;
-	isError: boolean;
-	status: number | null;
-	statusText: string | null;
-	contentType: string | null;
-	contentLength: number | null;
-	compress: false | CompressType;
-	cdn: false | CDNType;
-	headers: Record<string, string | string[] | undefined> | null;
-};
-```
-
-#### `ChangePhaseEvent`
-
-スクレイピングライフサイクルのフェーズ遷移イベントです。
-
-主なフェーズ: `scrapeStart` → `openPage` → `loadDOMContent` → `waitNetworkIdle` → `getHTML` → `getAnchors` → `getMeta` → `extractImages` → `getImages` → `scrapeEnd`
-
-その他のフェーズ: `launchBrowser`, `headRequest`, `headRequestTimeout`, `newPage`, `setViewport`, `scrollToBottom`, `waitImageLoad`, `pageSkipped`, `retryWait`, `retryExhausted`, `beforeCleanup`, `cleanedUp`
-
-#### `SkippedPageData`
-
-キーワードまたはパス除外によりスキップされたページのデータです。
-
-```typescript
-type SkippedPageData = {
-	isSkipped: true;
-	url: ExURL;
-	matched:
-		| { type: 'keyword'; text: string; excludeKeywords: string[] }
-		| { type: 'path'; excludes: string[] };
-};
-```
-
-## ライセンス
-
-MIT
+設計判断（イベントではなく戻り値で返す理由、`page` のライフサイクル責務、リトライ機構など）は `src/scraper.ts` の JSDoc を参照。

package/dist/dom-evaluation.d.ts

@@ -3,10 +3,28 @@
  *
  * These functions are called by {@link ./scraper.ts | Scraper.#fetchData} to extract
  * anchors, images, and meta information after page navigation completes.
+ *
+ * WHY timeouts everywhere: A page whose main thread is blocked (heavy JS, autoplay
+ * video players, infinite loops) makes every CDP round-trip hang. `getMeta` and
+ * `getImageList` therefore collect all data in a single `page.evaluate` and wrap it
+ * in {@link raceWithTimeout} so a blocked thread is abandoned after a bounded budget
+ * instead of accumulating per-property timeouts up to the caller's global timeout.
+ * Note that `page.evaluate` itself runs on the page's main thread and has no built-in
+ * timeout, so the surrounding race is what actually bounds the hang.
  * @see {@link ./types.ts} for the data types returned by these functions
  */
-import type { AnchorData, ImageElement, ParseURLOptions } from './types.js';
+import type { AnchorData, ImageElement, Meta, ParseURLOptions } from './types.js';
 import type { ElementHandle, Page } from 'puppeteer';
+/**
+ * Default timeout (ms) applied to DOM evaluation operations when the caller does not
+ * specify one. Bounds how long a single `page.evaluate` / property read may hang on a
+ * page whose main thread is unresponsive.
+ *
+ * WHY 180s: Aligned with the upstream `Scraper#fetchData` retryable timeout (3 min) so
+ * a single phase does not exceed the retry budget while still tolerating large pages
+ * (e.g., 1000+ anchors) and slow main threads.
+ */
+export declare const DEFAULT_DOM_EVALUATION_TIMEOUT = 180000;
 /**
  * Parameters for {@link getProp}.
  * @template T - The expected type of the property value.
@@ -22,88 +40,108 @@ export interface GetPropParams<T> {
 /**
  * Retrieves a DOM property value from a Puppeteer element handle with a timeout.
  *
- * Races the actual property retrieval against a 10-second timeout.
+ * Races the actual property retrieval against a timeout via {@link raceWithTimeout},
+ * which clears the loser-side timer so it cannot keep the event loop alive.
  * If the property cannot be read or the timeout expires, the fallback value is returned.
  * @template T - The expected type of the property value.
  * @param params - Parameters containing the element, property name, and fallback.
- * @returns The property value, or the fallback if retrieval fails.
- */
-export declare function getProp<T>(params: GetPropParams<T>): Promise<T>;
-/**
- * Parameters for {@link getPropBySelector}.
- * @template T - The expected type of the property value.
- */
-export interface GetPropBySelectorParams<T> {
-    /** The Puppeteer page to query. */
-    readonly page: Page;
-    /** A CSS selector to find the target element. */
-    readonly selector: string;
-    /** The DOM property name to read from the matched element. */
-    readonly propName: string;
-    /** The default value if no element matches or the property cannot be read. */
-    readonly fallback: T;
-}
-/**
- * Retrieves a DOM property value from the first element matching a CSS selector.
- *
- * Combines `page.$()` with {@link getProp} for convenient single-element lookups.
- * @template T - The expected type of the property value.
- * @param params - Parameters containing the page, selector, property name, and fallback.
- * @returns The property value, or the fallback if the element is not found or retrieval fails.
+ * @param timeout - Timeout in ms before falling back. Defaults to {@link DEFAULT_DOM_EVALUATION_TIMEOUT}.
+ * @returns The property value, or the fallback if retrieval fails or times out.
  */
-export declare function getPropBySelector<T>(params: GetPropBySelectorParams<T>): Promise<T>;
+export declare function getProp<T>(params: GetPropParams<T>, timeout?: number): Promise<T>;
 /**
  * Extracts all `<img>` elements from the page and returns their properties.
  *
- * For each image, collects the `src`, `currentSrc`, `alt`, bounding box dimensions,
- * natural dimensions, lazy-loading status, and the outer HTML source code.
+ * Collects every image's `src`, `currentSrc`, `alt`, layout dimensions,
+ * natural dimensions, lazy-loading status, and outer HTML in a single
+ * `page.evaluate` call, wrapped in {@link raceWithTimeout}. On timeout (an
+ * unresponsive page) an empty array is returned rather than hanging.
  * @param page - The Puppeteer page to extract images from.
  * @param viewportWidth - The current viewport width in pixels, recorded alongside each image entry.
+ * @param timeout - Timeout in ms for the evaluation. Defaults to {@link DEFAULT_DOM_EVALUATION_TIMEOUT}.
  * @returns An array of {@link ImageElement} objects describing each image on the page.
  */
-export declare function getImageList(page: Page, viewportWidth: number): Promise<ImageElement[]>;
+export declare function getImageList(page: Page, viewportWidth: number, timeout?: number): Promise<ImageElement[]>;
 /**
  * Extracts all anchor (`<a>` and `<area>`) elements with `href` attributes from the page.
  *
  * For each anchor, resolves the `href` to an `ExURL` via `parseUrl`, retrieves
  * the accessible name (from the accessibility tree, falling back to `textContent`),
  * and filters out non-HTTP links.
+ *
+ * WHY Strategy F (single AX-tree fetch + parallel `DOM.describeNode`): the old
+ * implementation called `page.accessibility.snapshot({ root })` per anchor, which
+ * triggers a CDP round-trip *and* a Chrome-side AX subtree computation (~42ms
+ * each). On a page with 1181 anchors that compounded to ~53s. By fetching the
+ * full AX tree once and using `DOM.describeNode` in parallel to map element
+ * handles back to AX nodes by `backendDOMNodeId`, the same data is collected in
+ * ~150ms on the same page — a ~350× speed-up while preserving the original
+ * accessible-name semantics. See issue #876 for measurements.
+ *
+ * WHY the whole operation is wrapped in `raceWithTimeout`: even with bounded
+ * per-CDP-call timeouts, a degenerate page (blocked main thread, thousands of
+ * anchors, runaway describeNode latency) could chain enough sub-timeouts to
+ * exceed the caller's `timeout` budget. The outer race guarantees the function
+ * returns within `timeout`, surfacing whatever anchors were collected so far so
+ * the upstream scrape phase can continue rather than tripping a retryable retry.
  * @param page - The Puppeteer page to extract anchors from.
  * @param options - Optional URL parsing options (e.g., `disableQueries`).
+ * @param timeout - Total time budget in ms for the whole extraction. Defaults to {@link DEFAULT_DOM_EVALUATION_TIMEOUT}.
  * @returns An array of {@link AnchorData} objects for all HTTP(S) links found on the page.
  */
-export declare function getAnchorList(page: Page, options?: ParseURLOptions): Promise<AnchorData[]>;
+export declare function getAnchorList(page: Page, options?: ParseURLOptions, timeout?: number): Promise<AnchorData[]>;
 /**
- * Extracts comprehensive meta information from the page's `<head>`.
+ * Required context for {@link getMeta}. Provided by the scraper from data it
+ * already has on hand (URL it navigated to, response status/headers it received).
+ *
+ * `html` is optional: when omitted, `getMeta` falls back to `page.content()`
+ * to obtain the rendered HTML for the third-party tag detection pass.
+ */
+export type GetMetaContext = {
+    /** The fully resolved URL of the page (after redirects). */
+    readonly url: string;
+    /** Rendered HTML. Falls back to `page.content()` when omitted. */
+    readonly html?: string;
+    /** Response status code, surfaced to the Wappalyzer driver. */
+    readonly statusCode?: number;
+    /** Response headers; case is preserved by the caller, lowercased internally. */
+    readonly headers?: Record<string, string | string[] | undefined>;
+    /**
+     * When `true`, the returned `Meta` includes `_raw: RawHeadEntry[]` for
+     * debugging. Default `false` to keep the serialized payload small.
+     */
+    readonly includeRaw?: boolean;
+};
+/**
+ * Extracts comprehensive metadata from the page.
+ *
+ * Two passes happen in parallel:
+ * 1. Browser-side `collectHead()` serializes every `<meta>`, `<link>`,
+ *    relevant `<script>`, `<base>`, `<noscript>`/`<iframe>` and a curated
+ *    set of `window` globals into a `RawHeadEntry[]`. Node-side `classify()`
+ *    then maps those entries to typed `Meta` fields using the lookup tables
+ *    in `./meta/keys.ts`, with unknown entries preserved in `Meta.others`.
+ * 2. `detectTags()` runs `simple-wappalyzer` over the page HTML to produce
+ *    `Meta.tags` (technology detection + real-ID extraction).
  *
- * Collects the following metadata:
- * - `title` - The document title.
- * - `lang` - The `lang` attribute of the `<html>` element.
- * - `description` - The `<meta name="description">` content.
- * - `keywords` - The `<meta name="keywords">` content.
- * - `noindex` / `nofollow` / `noarchive` - Parsed from the `<meta name="robots">` directives.
- * - `canonical` - The `<link rel="canonical">` content.
- * - `alternate` - The `<link rel="alternate">` content.
- * - Open Graph tags: `og:type`, `og:title`, `og:site_name`, `og:description`, `og:url`, `og:image`.
- * - `twitter:card` - The Twitter Card type.
- * @param page - The Puppeteer page to extract meta information from.
- * @returns An object containing all extracted meta properties.
+ * The whole call is wrapped in `raceWithTimeout`. On timeout an empty `Meta`
+ * (with `title: ''` and empty required arrays/objects) is returned.
+ * @param page
+ * @param context
+ * @param timeout
+ * @example
+ * ```ts
+ * const meta = await getMeta(page, {
+ *   url: 'https://example.com/',
+ *   html: await page.content(),
+ *   statusCode: response.status,
+ *   headers: response.headers,
+ * });
+ * console.log(meta.title);                         // <title> text
+ * console.log(meta.og?.image);                     // og:image[] array
+ * console.log(meta.robots?.noindex);               // parsed robots
+ * console.log(meta.tags.detected.Analytics);       // Wappalyzer hits
+ * console.log(meta.tags.entries.find(e => e.provider === 'Google Analytics')?.id);
+ * ```
  */
-export declare function getMeta(page: Page): Promise<{
-    title: string;
-    lang: string;
-    description: string;
-    keywords: string;
-    noindex: boolean;
-    nofollow: boolean;
-    noarchive: boolean;
-    canonical: string;
-    alternate: string;
-    'og:type': string;
-    'og:title': string;
-    'og:site_name': string;
-    'og:description': string;
-    'og:url': string;
-    'og:image': string;
-    'twitter:card': string;
-}>;
+export declare function getMeta(page: Page, context: GetMetaContext, timeout?: number): Promise<Meta>;