@pagepocket/lib 0.8.5 → 0.9.0

package/README.md

@@ -1,357 +0,0 @@
-# @pagepocket/lib
-
-Core library for capturing a page via NetworkInterceptorAdapter events and
-producing a virtual snapshot (HTML/CSS/JS rewritten to absolute snapshot paths).
-No network fetch happens in the core library.
-
-## Install
-
-```bash
-pnpm add @pagepocket/lib
-```
-
-## Quick Start
-
-```ts
-import { PagePocket } from "@pagepocket/lib";
-import { CaptureHttpCdpUnit } from "@pagepocket/capture-http-cdp-unit";
-import { BuildSnapshotUnit } from "@pagepocket/build-snapshot-unit";
-import { WriteDownUnit } from "@pagepocket/write-down-unit";
-
-const result = await PagePocket.fromCDPTab(123).capture({
-  units: [new CaptureHttpCdpUnit(), new BuildSnapshotUnit(), new WriteDownUnit({ type: "raw", outputPath: "./out" })]
-});
-
-// result.kind === "raw" | "zip" | ...
-```
-
-## API
-
-```ts
-class PagePocket {
-  static fromURL(url: string, options?: PagePocketOptions): PagePocket;
-  static fromTarget(target: InterceptTarget, options?: PagePocketOptions): PagePocket;
-// Network progress events are available via the units runner channel (network@1).
-  capture(options?: CaptureOptions): Promise<PageSnapshot>;
-}
-```
-
-### CaptureOptions (core)
-
-````ts
-interface CaptureOptions {
-  interceptor: NetworkInterceptorAdapter;
-  completion?: CompletionStrategy | CompletionStrategy[];
-  filter?: ResourceFilter;
-  blacklist?: RegExp[];
-  pathResolver?: PathResolver;
-  contentStore?: ContentStore;
-  rewriteEntry?: boolean;
-  rewriteCSS?: boolean;
-  limits?: {
-    maxTotalBytes?: number;
-    maxSingleResourceBytes?: number;
-    maxResources?: number;
-  };
-}
-
-type NetworkEventStream = AsyncIterable<NetworkEvent>;
-
-### Built-in blacklist
-
-```ts
-import { PagePocket, gaBlacklist } from "@pagepocket/lib";
-
-const snapshot = await PagePocket.fromURL("https://example.com").capture({
-  interceptor,
-  blacklist: gaBlacklist.ga
-});
-````
-
-You can combine multiple built-in lists:
-
-```ts
-import { PagePocket, builtinBlacklist } from "@pagepocket/lib";
-
-const snapshot = await PagePocket.fromURL("https://example.com").capture({
-  interceptor,
-  blacklist: [...builtinBlacklist.ga]
-});
-```
-
-````
-
-### PageSnapshot output
-
-```ts
-interface PageSnapshot {
-  version: "1.0";
-  createdAt: number;
-  url: string;
-  entry: string;
-  files: SnapshotFile[];
-  toDirectory(outDir: string, options?: WriteFSOptions): Promise<WriteResult>;
-  toZip(options?: ZipOptions): Promise<ZipResult>;
-}
-
-interface WriteFSOptions {
-  clearCache?: boolean;
-  overwrite?: boolean;
-  suffix?: string;
-}
-
-interface ZipOptions {
-  asBlob?: boolean;
-  clearCache?: boolean;
-  overwrite?: boolean;
-  suffix?: string;
-  outputPath?: string;
-}
-
-interface ZipWriteResult {
-  data: Uint8Array | Blob;
-  outputPath: string;
-}
-
-type ZipResult = Uint8Array | Blob | ZipWriteResult;
-````
-
-Snapshot layout:
-
-```
-/index.html
-/api.json
-/<same-origin paths>
-/external_resources/<cross-origin paths>
-```
-
-If multiple documents are captured, each document is written to its own output
-directory based on the document URL path (e.g. `foo/bar/index.html`).
-
-## Notes
-
-- Uses `@pagepocket/uni-fs` for file IO so it works in Node and OPFS contexts.
-- Network data comes only from the interceptor events.
-
-## HTML element replacement (replaceElements)
-
-During capture, `@pagepocket/lib` rewrites the captured HTML (the raw Document response body) before writing the final snapshot HTML.
-
-`replaceElements` lets you declaratively or programmatically replace parts of that HTML.
-
-Typical use-cases:
-
-- Replace embedded players/widgets with a placeholder (e.g. YouTube iframe → `<div>Player</div>`)
-- Rename tags at scale (e.g. all `<div>` → `<p>`)
-- Replace a specific element (e.g. `<div id="foo">` → `<span>hello</span>`)
-
-### Where this runs
-
-`replaceElements` runs during the **HTML rewrite stage (Cheerio)**, not on a live browser DOM.
-
-- Input is the captured HTML string.
-- You can match using CSS selectors.
-- Function rules receive a Cheerio element context.
-
-If you need to modify the live DOM after scripts run, that is a different capture mode and is not supported by this option.
-
-### Configuration
-
-Add `replaceElements` to `CaptureOptions`:
-
-```ts
-import type {
-  CaptureOptions,
-  ReplaceElementsConfig,
-  ReplaceAction,
-  ReplaceElementContext,
-  ReplaceElementRule
-} from "@pagepocket/lib";
-
-const replaceElements: ReplaceElementsConfig = [
-  {
-    name: "replace-youtube-embed",
-    match: 'iframe[src*="youtube.com/embed"]',
-    replace: { type: "replaceWithHtml", html: "<div>Player</div>" }
-  }
-];
-
-const options: CaptureOptions = {
-  interceptor,
-  replaceElements
-};
-```
-
-### Data structure
-
-`replaceElements` is an array of items. Each item can be:
-
-1. A **rule object** (declarative)
-2. A **function** (imperative; convenience form)
-3. A **function rule with query** (imperative; recommended for performance)
-
-```ts
-export type ReplaceElementsConfig = Array<
-  ReplaceElementRule | ReplaceElementFn | ReplaceElementFnWithQuery
->;
-
-export interface ReplaceElementRule {
-  name?: string;
-  match: MatchQuery;
-  replace: ReplaceAction;
-  apply?: ApplyOptions;
-}
-
-export type ReplaceElementFn = (
-  ctx: ReplaceElementContext
-) => void | ReplaceAction | ReplaceAction[] | Promise<void | ReplaceAction | ReplaceAction[]>;
-
-export interface ReplaceElementFnWithQuery {
-  name?: string;
-  query: string;
-  run: ReplaceElementFn;
-  apply?: ApplyOptions;
-}
-```
-
-#### MatchQuery
-
-First-class support is CSS selectors (Cheerio selectors).
-
-```ts
-export type MatchQuery =
-  | string
-  | {
-      selector?: string;
-      tagName?: string;
-      id?: string;
-      attrs?: Record<string, string | RegExp | true>;
-    };
-```
-
-#### ReplaceAction
-
-```ts
-export type ReplaceAction =
-  | { type: "replaceWithHtml"; html: string }
-  | {
-      type: "replaceWithElement";
-      tagName: string;
-      textContent?: string;
-      html?: string;
-      attrs?: Record<string, string | null>;
-    }
-  | {
-      type: "renameTag";
-      to: string;
-      keepAttributes?: boolean;
-      keepChildren?: boolean;
-    }
-  | { type: "remove" };
-```
-
-#### ApplyOptions
-
-```ts
-export interface ApplyOptions {
-  /** default: "document" */
-  scope?: "document" | "allFrames";
-  /** default: "all" */
-  limit?: number | "all";
-  /** default: "stop" */
-  onReplaced?: "stop" | "continue";
-}
-```
-
-Notes:
-
-- `scope: "allFrames"` applies to multiple captured documents (if your interceptor captures frames / subdocuments). Cross-origin frames may not be available depending on capture.
-- `limit` limits how many matched elements are replaced for that rule.
-
-### Semantics
-
-- Rules run **in array order**.
-- Each rule matches against the **current HTML state** (earlier replacements affect later matches).
-- Default `onReplaced: "stop"` makes results predictable: once an element is replaced by a rule, later rules won't attempt to re-process that same original element.
-
-### Examples
-
-Replace YouTube iframe embeds with a placeholder:
-
-```ts
-replaceElements: [
-  {
-    match: 'iframe[src*="youtube.com/embed"]',
-    replace: { type: "replaceWithHtml", html: "<div>Player</div>" }
-  }
-];
-```
-
-Rename all `div` tags to `p`:
-
-```ts
-replaceElements: [
-  {
-    match: "div",
-    replace: { type: "renameTag", to: "p", keepAttributes: true, keepChildren: true }
-  }
-];
-```
-
-Replace `<div id="foo">` with `<span>hello</span>`:
-
-```ts
-replaceElements: [
-  {
-    match: "div#foo",
-    replace: { type: "replaceWithElement", tagName: "span", textContent: "hello" }
-  }
-];
-```
-
-Complex matching via function (recommended form with `query`):
-
-```ts
-replaceElements: [
-  {
-    query: "iframe",
-    run: (ctx) => {
-      const src = ctx.$el.attr("src") || "";
-      if (!/youtube\.com\/embed/.test(src)) return;
-
-      return { type: "replaceWithHtml", html: "<div>Player</div>" };
-    }
-  }
-];
-```
-
-### Function context
-
-Function rules receive a Cheerio-aware context:
-
-```ts
-export interface ReplaceElementContext {
-  /** Cheerio root ($) for the current document */
-  $: CheerioAPI;
-  /** Cheerio wrapper for the matched element */
-  $el: Cheerio<any>;
-
-  /** Document URL being rewritten */
-  url: string;
-  /** Entry URL (top-level) */
-  entryUrl: string;
-
-  ruleIndex: number;
-  matchIndex: number;
-}
-```
-
-### Manual validation
-
-Run the CLI against a page containing the target elements:
-
-```bash
-pnpm -F @pagepocket/cli start -- https://example.com
-```
-
-Then inspect the output `*.html` for the expected replacements.