@tiny-codes/web-clip-extractor 0.2.0-alpha.0

package/README.md

@@ -0,0 +1,124 @@
+# web-clip-extractor
+
+The Node.js content-extraction engine for the web-to-obsidian Hermes plugin. It
+fetches a public web page over a hardened network policy and returns normalized
+article metadata plus Markdown.
+
+See `../CHANGELOG.md` for version history and `../plugin/README.md` for how the
+plugin drives this extractor.
+
+## Requirements
+
+- Node.js 18+ (package `engines.node: >=18`)
+- Locked dependencies from `package-lock.json` (`npm ci --ignore-scripts`)
+- Playwright Chromium for the browser fallback (`npx playwright install chromium`
+  after `npm ci` — `--ignore-scripts` intentionally skips the download that
+  Playwright's postinstall normally performs).
+
+## Dependencies
+
+| Package      | Purpose                                   |
+| ------------ | ----------------------------------------- |
+| `defuddle`   | Primary static article extraction         |
+| `linkedom`   | Lightweight DOM parsing (no browser)      |
+| `playwright` | Isolated Chromium fallback for weak pages |
+
+## CLI
+
+```bash
+node src/cli.mjs <url> [--no-browser]
+```
+
+- `<url>` — exactly one `http://` or `https://` URL.
+- `--no-browser` — skip the Chromium fallback; fail with `QUALITY_GATE` if the
+  static pass yields no substantial article.
+
+Output is a single JSON line on stdout. On success:
+
+```json
+{ "ok": true, "title": "...", "author": "...", "published": "...",
+  "description": "...", "site": "...", "canonicalUrl": "...",
+  "keywords": [...], "markdown": "...", "wordCount": 123, "method": "static" }
+```
+
+`method` is `static` (Defuddle over the pinned fetch) or `playwright`
+(Chromium fallback). On failure, exit code is `1` and stdout carries:
+
+```json
+{ "ok": false, "error": "<user-facing message>", "code": "<CODE>" }
+```
+
+### Error codes
+
+| Code                       | Meaning                                         |
+| -------------------------- | ----------------------------------------------- |
+| `INVALID_URL`              | The URL is malformed.                           |
+| `UNSUPPORTED_SCHEME`       | Only HTTP and HTTPS are allowed.                |
+| `URL_CREDENTIALS`          | URLs containing credentials are rejected.       |
+| `NON_DEFAULT_PORT`         | Non-default ports are rejected.                 |
+| `DNS_FAILED`               | The hostname could not be resolved.             |
+| `BLOCKED_ADDRESS`          | Destination blocked by network policy.          |
+| `TOO_MANY_REDIRECTS`       | The page redirected too many times.             |
+| `INVALID_REDIRECT`         | The page returned an invalid redirect.          |
+| `TIMEOUT`                  | Extraction timed out.                           |
+| `NETWORK_ERROR`            | The page request failed.                        |
+| `HTTP_STATUS`              | The server returned an unsuccessful status.     |
+| `UNSUPPORTED_CONTENT_TYPE` | The response is not HTML.                       |
+| `UNSUPPORTED_ENCODING`     | The response uses an unsupported encoding.      |
+| `BODY_TOO_LARGE`           | The response body is too large.                 |
+| `INVALID_RESPONSE`         | The server returned an invalid response.        |
+| `EXTRACTION_FAILED`        | Article extraction failed.                      |
+| `QUALITY_GATE`             | The page did not contain a substantial article. |
+| `BROWSER_FAILED`           | Browser extraction failed.                      |
+| `USAGE`                    | Bad command-line invocation.                    |
+
+## Extraction pipeline
+
+1. **Static pass** — `extractHtml()` normalizes links (`absolutizeLinks`),
+   canonicalizes the URL, then runs Defuddle to produce Markdown. No page
+   scripts execute and Defuddle network fallbacks are disabled.
+2. **Quality gate** — a result passes only if it has a meaningful non-generic
+   title and ≥ 200 Markdown characters (`MIN_MARKDOWN_CHARS`).
+3. **Playwright fallback** (unless `--no-browser`) — if the static pass fails
+   the quality gate, an isolated headless Chromium re-renders the page. All
+   browser HTTP(S) is deferred to the pinned Node fetch layer via
+   `buildSecureRouteHandler()` (see [Network safety](#network-safety)).
+
+### Site-specific handling
+
+- **Netease (`c.m.163.com`)** — author and published date live only in the
+  embedded `window.__INITIAL_STATE__` JSON, not in standard meta tags.
+  `extractNeteaseMeta()` reads the balanced `main` object (source /
+  `sourceinfo.tname` / `ptime`), scoped to `163.com` hostnames.
+- **Lazy-loaded images** — Netease and similar sites render images with a
+  placeholder `src` (`empty.png`) and the real URL in `data-echo` /
+  `data-src` / `data-original`. `extractLazyImageSrc()` prefers those
+  attributes (in that order) and rewrites `src` to the real, same-page
+  absolute URL before Defuddle.
+
+## Network safety
+
+- Every request is validated and pinned: non-default ports rejected, redirects
+  revalidated, DNS answers must all be public, and each request is bound to an
+  approved address while preserving TLS SNI/hostname checks.
+- Static responses require an HTML media type and a bounded body.
+- In the browser, **Chromium native DNS and HTTP(S) are disabled**
+  (`--host-resolver-rules=MAP * ~NOTFOUND`). Every HTTP(S) page resource is
+  fetched by the pinned Node layer and injected via `route.fulfill`;
+  WebSockets, service workers, and downloads are blocked. Request count,
+  per-resource bytes, total bytes, and wall time are bounded.
+- The naked `node src/cli.mjs` process may print page-controlled diagnostics
+  only internally; the CLI protocol and consuming logs stay free of untrusted
+  stack traces.
+
+See `src/network-policy.mjs` for the full policy implementation.
+
+## Tests
+
+```bash
+npm test        # node --test (requires locked deps installed)
+npm run check   # node --check on each src module
+```
+
+Test files live in `test/` and cover the CLI protocol, extraction, network
+policy, and security regressions.

package/package.json

@@ -0,0 +1,38 @@
+{
+  "name": "@tiny-codes/web-clip-extractor",
+  "version": "0.2.0-alpha.0",
+  "type": "module",
+  "engines": {
+    "node": ">=18"
+  },
+  "keywords": [
+    "web",
+    "clip",
+    "extractor",
+    "obsidian"
+  ],
+  "homepage": "https://github.com/shijistar/url-to-obsidian",
+  "bugs": {
+    "url": "https://github.com/shijistar/url-to-obsidian/issues"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/shijistar/url-to-obsidian.git"
+  },
+  "license": "MIT",
+  "author": "Fengbao Li <shijistar@gmail.com>",
+  "sideEffects": false,
+  "files": [
+    "src",
+    "README.md"
+  ],
+  "scripts": {
+    "test": "node --test",
+    "check": "node --check src/cli.mjs && node --check src/extractor.mjs && node --check src/network-policy.mjs"
+  },
+  "dependencies": {
+    "defuddle": "^0.19.2",
+    "linkedom": "^0.18.13",
+    "playwright": "^1.61.1"
+  }
+}

package/src/cli.mjs

@@ -0,0 +1,54 @@
+#!/usr/bin/env node
+
+import { extractUrl } from './extractor.mjs';
+
+const PUBLIC_ERRORS = {
+  INVALID_URL: 'The URL is malformed.',
+  UNSUPPORTED_SCHEME: 'Only HTTP and HTTPS URLs are allowed.',
+  URL_CREDENTIALS: 'URLs containing credentials are not allowed.',
+  NON_DEFAULT_PORT: 'Non-default ports are not allowed.',
+  DNS_FAILED: 'The hostname could not be resolved.',
+  BLOCKED_ADDRESS: 'The destination is blocked by network policy.',
+  TOO_MANY_REDIRECTS: 'The page redirected too many times.',
+  INVALID_REDIRECT: 'The page returned an invalid redirect.',
+  TIMEOUT: 'The extraction timed out.',
+  NETWORK_ERROR: 'The page request failed.',
+  HTTP_STATUS: 'The server returned an unsuccessful status.',
+  UNSUPPORTED_CONTENT_TYPE: 'The response is not HTML.',
+  UNSUPPORTED_ENCODING: 'The response uses an unsupported encoding.',
+  BODY_TOO_LARGE: 'The response body is too large.',
+  INVALID_RESPONSE: 'The server returned an invalid response.',
+  EXTRACTION_FAILED: 'Article extraction failed.',
+  QUALITY_GATE: 'The page did not contain a substantial article.',
+  BROWSER_FAILED: 'Browser extraction failed.',
+  USAGE: 'Usage: node src/cli.mjs <url> [--no-browser]',
+};
+
+function output(value) {
+  process.stdout.write(`${JSON.stringify(value)}\n`);
+}
+
+async function main(argv) {
+  const args = [...argv];
+  const noBrowserIndex = args.indexOf('--no-browser');
+  const noBrowser = noBrowserIndex !== -1;
+  if (noBrowser) args.splice(noBrowserIndex, 1);
+  if (args.length !== 1 || args[0].startsWith('--')) {
+    output({ ok: false, error: PUBLIC_ERRORS.USAGE, code: 'USAGE' });
+    process.exitCode = 1;
+    return;
+  }
+
+  try {
+    const result = await extractUrl(args[0], { allowBrowser: !noBrowser });
+    output({ ok: true, ...result });
+  } catch (cause) {
+    const code = typeof cause?.code === 'string' && Object.hasOwn(PUBLIC_ERRORS, cause.code)
+      ? cause.code
+      : 'EXTRACTION_FAILED';
+    output({ ok: false, error: PUBLIC_ERRORS[code], code });
+    process.exitCode = 1;
+  }
+}
+
+await main(process.argv.slice(2));

package/src/extractor.mjs

@@ -0,0 +1,425 @@
+import { Defuddle } from 'defuddle/node';
+import { parseHTML } from 'linkedom';
+import { chromium } from 'playwright';
+
+import {
+  DEFAULT_TIMEOUT_MS,
+  PolicyError,
+  fetchResourceOnce,
+  fetchHtml as secureFetchHtml,
+  normalizeUrl,
+  resolveAndValidateUrl,
+} from './network-policy.mjs';
+
+export const MIN_MARKDOWN_CHARS = 200;
+export const MAX_BROWSER_REQUESTS = 80;
+export const BROWSER_TIMEOUT_MS = 30_000;
+export const MAX_BROWSER_RESOURCE_BYTES = 4 * 1024 * 1024;
+export const MAX_BROWSER_TOTAL_BYTES = 32 * 1024 * 1024;
+
+const GENERIC_TITLES = new Set([
+  'home',
+  'homepage',
+  'index',
+  'new tab',
+  'untitled',
+  'untitled page',
+]);
+
+export class ExtractorError extends Error {
+  constructor(code, message) {
+    super(message);
+    this.name = 'ExtractorError';
+    this.code = code;
+  }
+}
+
+function cleanString(value) {
+  const cleaned = typeof value === 'string' ? value.replace(/\s+/g, ' ').trim() : '';
+  return cleaned || null;
+}
+
+function countWords(markdown) {
+  const words = markdown
+    .replace(/```[\s\S]*?```/g, ' ')
+    .replace(/!\[[^\]]*\]\([^)]*\)/g, ' ')
+    .replace(/\[([^\]]+)\]\([^)]*\)/g, '$1')
+    .match(/[\p{L}\p{N}]+(?:['’_-][\p{L}\p{N}]+)*/gu);
+  return words?.length || 0;
+}
+
+function meaningfulTitle(title) {
+  const cleaned = cleanString(title);
+  if (!cleaned || cleaned.length < 3) return false;
+  if (GENERIC_TITLES.has(cleaned.toLowerCase())) return false;
+  return /[\p{L}\p{N}]/u.test(cleaned);
+}
+
+function collectKeywords(document) {
+  const selectors = [
+    'meta[name="keywords" i]',
+    'meta[name="news_keywords" i]',
+    'meta[property="article:tag" i]',
+  ];
+  const seen = new Set();
+  const keywords = [];
+  for (const selector of selectors) {
+    for (const node of document.querySelectorAll(selector)) {
+      const raw = node.getAttribute('content');
+      if (!raw) continue;
+      for (const part of raw.split(/[;,\n]+/)) {
+        const cleaned = cleanString(part);
+        if (!cleaned || !/[\p{L}\p{N}]/u.test(cleaned)) continue;
+        const key = cleaned.toLocaleLowerCase('en-US');
+        if (seen.has(key)) continue;
+        seen.add(key);
+        keywords.push(cleaned);
+        if (keywords.length >= 32) return keywords;
+      }
+    }
+  }
+  return keywords;
+}
+
+export function meetsQualityGate(result) {
+  return meaningfulTitle(result?.title) && typeof result?.markdown === 'string' && result.markdown.trim().length >= MIN_MARKDOWN_CHARS;
+}
+
+/** Extract the balanced JSON object starting at the given index ("" if unbalanced). */
+function extractBalancedObjectAt(text, startIndex) {
+  let depth = 0;
+  let inString = false;
+  let escaped = false;
+  for (let i = startIndex; i < text.length; i += 1) {
+    const ch = text[i];
+    if (inString) {
+      if (escaped) escaped = false;
+      else if (ch === '\\') escaped = true;
+      else if (ch === '"') inString = false;
+      continue;
+    }
+    if (ch === '"') {
+      inString = true;
+      continue;
+    }
+    if (ch === '{') depth += 1;
+    else if (ch === '}') {
+      depth -= 1;
+      if (depth === 0) return text.slice(startIndex, i + 1);
+    }
+  }
+  return '';
+}
+
+/**
+ * Netease mobile pages (c.m.163.com) expose author/published only inside the
+ * embedded `window.__INITIAL_STATE__` JSON — not in standard meta tags or
+ * JSON-LD. The full state blob may be followed by extra script payloads that
+ * make a wholesale JSON.parse unreliable, so we read the balanced `main`
+ * object with field-level regexes and ignore recommendation-list sources.
+ */
+function extractNeteaseMeta(html) {
+  const marker = 'window.__INITIAL_STATE__';
+  const markerIndex = html.indexOf(marker);
+  if (markerIndex < 0) return { author: '', published: '' };
+  const eqIndex = html.indexOf('=', markerIndex + marker.length);
+  if (eqIndex < 0) return { author: '', published: '' };
+
+  const stateBlock = extractBalancedObjectAt(html, eqIndex + 1);
+  if (!stateBlock) return { author: '', published: '' };
+
+  const mainKey = stateBlock.indexOf('"main"');
+  const mainStart = mainKey >= 0 ? stateBlock.indexOf('{', mainKey) : -1;
+  const mainBlock = mainStart >= 0 ? extractBalancedObjectAt(stateBlock, mainStart) : stateBlock;
+
+  let author = /"source"\s*:\s*"([^"]+)"/.exec(mainBlock)?.[1] ?? '';
+  if (!author) {
+    author = /"sourceinfo"\s*:\s*\{[\s\S]*?"tname"\s*:\s*"([^"]+)"/.exec(mainBlock)?.[1] ?? '';
+  }
+  const published = /"ptime"\s*:\s*"([^"]+)"/.exec(mainBlock)?.[1] ?? '';
+
+  return {
+    author: cleanString(author) || '',
+    published: cleanString(published) || '',
+  };
+}
+
+function safeCanonical(document, sourceUrl) {
+  const canonical = document.querySelector('link[rel~="canonical" i]');
+  const href = canonical?.getAttribute('href');
+  if (!href) return sourceUrl.href;
+  try {
+    const normalized = normalizeUrl(new URL(href, sourceUrl).href);
+    if (normalized.origin !== sourceUrl.origin) return sourceUrl.href;
+    canonical.setAttribute('href', normalized.href);
+    return normalized.href;
+  } catch {
+    canonical?.remove();
+    return sourceUrl.href;
+  }
+}
+
+function absolutizeLinks(document, sourceUrl) {
+  for (const anchor of document.querySelectorAll('a[href]')) {
+    const href = anchor.getAttribute('href')?.trim();
+    if (!href || href.startsWith('#')) continue;
+    try {
+      const absolute = new URL(href, sourceUrl);
+      if (absolute.protocol === 'http:' || absolute.protocol === 'https:' || absolute.protocol === 'mailto:') {
+        anchor.setAttribute('href', absolute.href);
+      } else {
+        anchor.removeAttribute('href');
+      }
+    } catch {
+      anchor.removeAttribute('href');
+    }
+  }
+  for (const element of document.querySelectorAll('img[src], source[src]')) {
+    if (element.tagName?.toLowerCase() === 'img') {
+      // 网易等站点用懒加载：src 是占位图，真实地址藏在 data-echo/data-src/data-original。
+      // 优先用真实地址覆盖占位 src，否则 Defuddle 只会抓到占位图。
+      const lazySrc = extractLazyImageSrc(element, sourceUrl);
+      if (lazySrc) element.setAttribute('src', lazySrc);
+    }
+    const src = element.getAttribute('src')?.trim();
+    if (!src) continue;
+    try {
+      const absolute = new URL(src, sourceUrl);
+      if (absolute.protocol === 'http:' || absolute.protocol === 'https:') element.setAttribute('src', absolute.href);
+      else element.removeAttribute('src');
+    } catch {
+      element.removeAttribute('src');
+    }
+  }
+}
+
+// 懒加载真实图片地址：优先尝试 data-echo / data-src / data-original，
+// 解析为同源可访问的 http(s) URL；失败时回退到原 src（可能已是占位图）。
+function extractLazyImageSrc(element, sourceUrl) {
+  for (const attr of ['data-echo', 'data-src', 'data-original', 'data-lazy-src']) {
+    const candidate = element.getAttribute(attr)?.trim();
+    if (!candidate) continue;
+    try {
+      const absolute = new URL(candidate, sourceUrl);
+      if (absolute.protocol === 'http:' || absolute.protocol === 'https:') return absolute.href;
+    } catch {
+      // 忽略无法解析的候选，继续尝试下一个属性
+    }
+  }
+  return null;
+}
+
+/** Parse already-fetched HTML without evaluating page scripts or allowing Defuddle network fallbacks. */
+export async function extractHtml(html, inputUrl) {
+  if (typeof html !== 'string') throw new ExtractorError('INVALID_HTML', 'The HTML input is invalid.');
+  const sourceUrl = normalizeUrl(inputUrl);
+  const { document } = parseHTML(html);
+  const canonicalUrl = safeCanonical(document, sourceUrl);
+  absolutizeLinks(document, sourceUrl);
+
+  let parsed;
+  const originalConsoleError = console.error;
+  const originalConsoleWarn = console.warn;
+  try {
+    console.error = () => {};
+    console.warn = () => {};
+    parsed = await Defuddle(document, sourceUrl.href, {
+      markdown: true,
+      useAsync: false,
+      debug: false,
+    });
+  } catch {
+    throw new ExtractorError('EXTRACTION_FAILED', 'Article extraction failed.');
+  } finally {
+    // Defuddle can log page-controlled malformed metadata URLs. Keep the CLI
+    // protocol and Hermes logs free of untrusted diagnostics and stack traces.
+    console.error = originalConsoleError;
+    console.warn = originalConsoleWarn;
+  }
+
+  const markdown = typeof parsed.content === 'string' ? parsed.content.trim() : '';
+  const embeddedMeta = sourceUrl.hostname.includes('163.com') ? extractNeteaseMeta(html) : null;
+  return {
+    title: cleanString(parsed.title),
+    author: cleanString(parsed.author) || embeddedMeta?.author || '',
+    published: cleanString(parsed.published) || embeddedMeta?.published || '',
+    description: cleanString(parsed.description) || '',
+    site: cleanString(parsed.site) || cleanString(parsed.domain) || sourceUrl.hostname,
+    canonicalUrl,
+    keywords: collectKeywords(document),
+    markdown,
+    wordCount: Number.isSafeInteger(parsed.wordCount) && parsed.wordCount >= 0
+      ? parsed.wordCount
+      : countWords(markdown),
+  };
+}
+
+function qualityError() {
+  return new ExtractorError('QUALITY_GATE', 'The page did not contain a substantial article.');
+}
+
+async function closeQuietly(target) {
+  try {
+    await target?.close();
+  } catch {
+    // Cleanup must not hide the extraction result or its original error.
+  }
+}
+
+function responseHeadersForBrowser(headers) {
+  const blocked = new Set([
+    'connection', 'content-encoding', 'content-length', 'keep-alive',
+    'proxy-authenticate', 'proxy-authorization', 'set-cookie', 'te',
+    'trailer', 'transfer-encoding', 'upgrade',
+  ]);
+  const output = {};
+  for (const [name, value] of Object.entries(headers || {})) {
+    const lower = name.toLowerCase();
+    if (blocked.has(lower) || value == null) continue;
+    output[lower] = Array.isArray(value) ? value.join(', ') : String(value);
+  }
+  return output;
+}
+
+/** Build a browser route that never lets Chromium perform HTTP(S) I/O. */
+export function buildSecureRouteHandler({
+  resolver,
+  allowNonDefaultPorts = false,
+  secureFetcher = fetchResourceOnce,
+  timeoutMs = BROWSER_TIMEOUT_MS,
+  maxRequests = MAX_BROWSER_REQUESTS,
+  maxResourceBytes = MAX_BROWSER_RESOURCE_BYTES,
+  maxTotalBytes = MAX_BROWSER_TOTAL_BYTES,
+} = {}) {
+  let requestCount = 0;
+  let totalBytes = 0;
+  return async route => {
+    requestCount += 1;
+    if (requestCount > maxRequests) {
+      await route.abort('blockedbyclient');
+      return;
+    }
+
+    const request = route.request();
+    let requestUrl;
+    try {
+      requestUrl = new URL(request.url());
+    } catch {
+      await route.abort('blockedbyclient');
+      return;
+    }
+
+    if (requestUrl.protocol === 'data:' || requestUrl.protocol === 'about:') {
+      await route.continue();
+      return;
+    }
+    if ((requestUrl.protocol !== 'http:' && requestUrl.protocol !== 'https:') || request.method() !== 'GET') {
+      await route.abort('blockedbyclient');
+      return;
+    }
+
+    const remaining = maxTotalBytes - totalBytes;
+    if (remaining <= 0) {
+      await route.abort('blockedbyclient');
+      return;
+    }
+    try {
+      const response = await secureFetcher(requestUrl.href, {
+        resolver,
+        allowNonDefaultPorts,
+        timeoutMs,
+        maxBytes: Math.min(maxResourceBytes, remaining),
+      });
+      totalBytes += response.body.length;
+      await route.fulfill({
+        status: response.statusCode,
+        headers: responseHeadersForBrowser(response.headers),
+        body: response.body,
+      });
+    } catch {
+      await route.abort('blockedbyclient');
+    }
+  };
+}
+
+async function extractWithPlaywright(inputUrl, {
+  resolver,
+  allowNonDefaultPorts,
+  browserLauncher = chromium,
+  timeoutMs = BROWSER_TIMEOUT_MS,
+  maxRequests = MAX_BROWSER_REQUESTS,
+} = {}) {
+  const approved = await resolveAndValidateUrl(inputUrl, { resolver, allowNonDefaultPorts });
+  let browser;
+  let context;
+  try {
+    browser = await browserLauncher.launch({
+      headless: true,
+      args: [
+        '--host-resolver-rules=MAP * ~NOTFOUND',
+        '--disable-webrtc',
+        '--force-webrtc-ip-handling-policy=disable_non_proxied_udp',
+      ],
+    });
+    context = await browser.newContext({
+      serviceWorkers: 'block',
+      acceptDownloads: false,
+    });
+    context.setDefaultTimeout(timeoutMs);
+    context.setDefaultNavigationTimeout(timeoutMs);
+
+    await context.route('**/*', buildSecureRouteHandler({
+      resolver,
+      allowNonDefaultPorts,
+      timeoutMs,
+      maxRequests,
+    }));
+    if (typeof context.routeWebSocket === 'function') {
+      await context.routeWebSocket('**/*', socket => socket.close({ code: 1008, reason: 'blocked' }));
+    }
+
+    const page = await context.newPage();
+    await page.goto(approved.url.href, { waitUntil: 'domcontentloaded', timeout: timeoutMs });
+    const finalUrl = page.url();
+    await resolveAndValidateUrl(finalUrl, { resolver, allowNonDefaultPorts });
+    const result = await extractHtml(await page.content(), finalUrl);
+    if (!meetsQualityGate(result)) throw qualityError();
+    return { ...result, url: normalizeUrl(finalUrl).href, method: 'playwright' };
+  } catch (cause) {
+    if (cause instanceof PolicyError || cause instanceof ExtractorError) throw cause;
+    throw new ExtractorError('BROWSER_FAILED', 'Browser extraction failed.');
+  } finally {
+    await closeQuietly(context);
+    await closeQuietly(browser);
+  }
+}
+
+/** Fetch and extract a URL statically, with an isolated Chromium fallback for low-quality pages. */
+export async function extractUrl(inputUrl, options = {}) {
+  const normalized = normalizeUrl(inputUrl, { allowNonDefaultPorts: options.allowNonDefaultPorts });
+  const fetcher = options.fetchHtml || secureFetchHtml;
+  const fetched = await fetcher(normalized.href, {
+    resolver: options.resolver,
+    maxRedirects: options.maxRedirects,
+    maxBytes: options.maxBytes,
+    timeoutMs: options.staticTimeoutMs || DEFAULT_TIMEOUT_MS,
+    allowNonDefaultPorts: options.allowNonDefaultPorts,
+  });
+  if (!fetched || typeof fetched.html !== 'string' || typeof fetched.finalUrl !== 'string') {
+    throw new ExtractorError('INVALID_RESPONSE', 'The static fetch returned an invalid response.');
+  }
+
+  const staticResult = await extractHtml(fetched.html, fetched.finalUrl);
+  if (meetsQualityGate(staticResult)) {
+    return { ...staticResult, url: normalizeUrl(fetched.finalUrl).href, method: 'static' };
+  }
+  if (options.allowBrowser === false || options.dynamic === false || options.browser === false) throw qualityError();
+
+  return extractWithPlaywright(normalized.href, {
+    resolver: options.resolver,
+    allowNonDefaultPorts: options.allowNonDefaultPorts,
+    browserLauncher: options.browserLauncher,
+    timeoutMs: options.browserTimeoutMs || BROWSER_TIMEOUT_MS,
+    maxRequests: options.maxBrowserRequests || MAX_BROWSER_REQUESTS,
+  });
+}

package/src/network-policy.mjs

@@ -0,0 +1,429 @@
+import { promises as dns } from 'node:dns';
+import http from 'node:http';
+import https from 'node:https';
+import net from 'node:net';
+import { Transform, Writable } from 'node:stream';
+import { pipeline } from 'node:stream/promises';
+import { createBrotliDecompress, createGunzip, createInflate } from 'node:zlib';
+
+export const MAX_BODY_BYTES = 8 * 1024 * 1024;
+export const DEFAULT_TIMEOUT_MS = 20_000;
+export const DEFAULT_MAX_REDIRECTS = 5;
+
+const USER_AGENT = 'Web-Clip-Extractor/0.1 (+local article extractor)';
+const HTML_CONTENT_TYPES = new Set(['text/html', 'application/xhtml+xml']);
+const REDIRECT_STATUSES = new Set([301, 302, 303, 307, 308]);
+
+export class PolicyError extends Error {
+  constructor(code, message) {
+    super(message);
+    this.name = 'PolicyError';
+    this.code = code;
+  }
+}
+
+function ipv4Number(address) {
+  if (!net.isIPv4(address)) return null;
+  return address.split('.').reduce((value, part) => value * 256 + Number(part), 0) >>> 0;
+}
+
+function ipv4InCidr(value, base, prefix) {
+  const shift = 32 - prefix;
+  return value >>> shift === base >>> shift;
+}
+
+const BLOCKED_IPV4_RANGES = [
+  ['0.0.0.0', 8],
+  ['10.0.0.0', 8],
+  ['100.64.0.0', 10],
+  ['127.0.0.0', 8],
+  ['169.254.0.0', 16],
+  ['172.16.0.0', 12],
+  ['192.0.0.0', 24],
+  ['192.0.2.0', 24],
+  ['192.88.99.0', 24],
+  ['192.168.0.0', 16],
+  ['198.18.0.0', 15],
+  ['198.51.100.0', 24],
+  ['203.0.113.0', 24],
+  ['224.0.0.0', 4],
+  ['240.0.0.0', 4],
+].map(([base, prefix]) => [ipv4Number(base), prefix]);
+
+function parseIpv6(address) {
+  if (typeof address !== 'string' || address.includes('%') || !net.isIPv6(address)) return null;
+
+  let source = address.toLowerCase();
+  const dottedIndex = source.lastIndexOf(':');
+  if (source.includes('.')) {
+    const dotted = source.slice(dottedIndex + 1);
+    const value = ipv4Number(dotted);
+    if (value === null) return null;
+    source = `${source.slice(0, dottedIndex)}:${(value >>> 16).toString(16)}:${(value & 0xffff).toString(16)}`;
+  }
+
+  const halves = source.split('::');
+  if (halves.length > 2) return null;
+  const left = halves[0] ? halves[0].split(':') : [];
+  const right = halves.length === 2 && halves[1] ? halves[1].split(':') : [];
+  const missing = 8 - left.length - right.length;
+  if ((halves.length === 1 && missing !== 0) || missing < 0) return null;
+
+  const words = [...left, ...Array(halves.length === 2 ? missing : 0).fill('0'), ...right].map(
+    (part) => Number.parseInt(part, 16),
+  );
+  if (
+    words.length !== 8 ||
+    words.some((word) => !Number.isInteger(word) || word < 0 || word > 0xffff)
+  )
+    return null;
+
+  return words.reduce((value, word) => (value << 16n) | BigInt(word), 0n);
+}
+
+function ipv6InCidr(value, base, prefix) {
+  const shift = 128n - BigInt(prefix);
+  return value >> shift === base >> shift;
+}
+
+const BLOCKED_IPV6_RANGES = [
+  ['::', 96],
+  ['::', 128],
+  ['::1', 128],
+  ['64:ff9b::', 96],
+  ['64:ff9b:1::', 48],
+  ['100::', 64],
+  ['2001::', 23],
+  ['2001:2::', 48],
+  ['2001:10::', 28],
+  ['2001:20::', 28],
+  ['2001:db8::', 32],
+  ['2002::', 16],
+  ['3fff::', 20],
+  ['5f00::', 16],
+  ['fc00::', 7],
+  ['fe80::', 10],
+  ['fec0::', 10],
+  ['ff00::', 8],
+].map(([base, prefix]) => [parseIpv6(base), prefix]);
+
+/** Return true for invalid, non-routable, private, or special-use IP addresses. */
+export function isBlockedIp(address) {
+  const ipv4 = ipv4Number(address);
+  if (ipv4 !== null) {
+    return BLOCKED_IPV4_RANGES.some(([base, prefix]) => ipv4InCidr(ipv4, base, prefix));
+  }
+
+  const ipv6 = parseIpv6(address);
+  if (ipv6 === null) return true;
+
+  // IPv4-mapped IPv6 (::ffff:0:0/96) is classified by its embedded IPv4 value.
+  if (ipv6 >> 32n === 0xffffn) {
+    const mapped = Number(ipv6 & 0xffffffffn);
+    return BLOCKED_IPV4_RANGES.some(([base, prefix]) => ipv4InCidr(mapped, base, prefix));
+  }
+
+  return BLOCKED_IPV6_RANGES.some(([base, prefix]) => ipv6InCidr(ipv6, base, prefix));
+}
+
+function error(code, message) {
+  return new PolicyError(code, message);
+}
+
+/** Parse and canonicalize a URL before any DNS or network operation. */
+export function normalizeUrl(input, { allowNonDefaultPorts = false } = {}) {
+  let url;
+  try {
+    url = new URL(input);
+  } catch {
+    throw error('INVALID_URL', 'The URL is malformed.');
+  }
+
+  if (url.protocol !== 'http:' && url.protocol !== 'https:') {
+    throw error('UNSUPPORTED_SCHEME', 'Only HTTP and HTTPS URLs are allowed.');
+  }
+  if (url.username || url.password) {
+    throw error('URL_CREDENTIALS', 'URLs containing credentials are not allowed.');
+  }
+  if (!url.hostname) throw error('INVALID_URL', 'The URL has no hostname.');
+  if (url.port && !allowNonDefaultPorts) {
+    throw error('NON_DEFAULT_PORT', 'Non-default ports are not allowed.');
+  }
+
+  url.hash = '';
+  const entries = [...url.searchParams.entries()]
+    .filter(
+      ([key]) =>
+        !/^utm_/i.test(key) && !['fbclid', 'gclid', 'share_token'].includes(key.toLowerCase()),
+    )
+    .map(([key, value], index) => ({ key, value, index }))
+    .sort((a, b) => {
+      if (a.key !== b.key) return a.key < b.key ? -1 : 1;
+      if (a.value !== b.value) return a.value < b.value ? -1 : 1;
+      return a.index - b.index;
+    });
+  url.search = '';
+  for (const { key, value } of entries) url.searchParams.append(key, value);
+  return url;
+}
+
+function hostnameForDns(url) {
+  return url.hostname.startsWith('[') && url.hostname.endsWith(']')
+    ? url.hostname.slice(1, -1)
+    : url.hostname;
+}
+
+/** Resolve a URL once, reject it if any answer is unsafe, and return a pinned answer. */
+export async function resolveAndValidateUrl(
+  input,
+  { resolver = dns.lookup, allowNonDefaultPorts = false } = {},
+) {
+  const url =
+    input instanceof URL
+      ? normalizeUrl(input.href, { allowNonDefaultPorts })
+      : normalizeUrl(input, { allowNonDefaultPorts });
+  let answers;
+  try {
+    answers = await resolver(hostnameForDns(url), { all: true, verbatim: true });
+  } catch {
+    throw error('DNS_FAILED', 'The hostname could not be resolved.');
+  }
+
+  if (!Array.isArray(answers) || answers.length === 0) {
+    throw error('DNS_FAILED', 'The hostname returned no addresses.');
+  }
+  for (const answer of answers) {
+    if (!answer || (answer.family !== 4 && answer.family !== 6) || isBlockedIp(answer.address)) {
+      throw error('BLOCKED_ADDRESS', 'The hostname resolves to a blocked address.');
+    }
+  }
+
+  return {
+    url,
+    address: answers[0].address,
+    family: answers[0].family,
+    addresses: answers.map((answer) => ({ address: answer.address, family: answer.family })),
+  };
+}
+
+function firstHeader(headers, name) {
+  const value = headers?.[name];
+  return Array.isArray(value) ? value[0] : value;
+}
+
+function contentType(headers) {
+  return String(firstHeader(headers, 'content-type') || '')
+    .split(';', 1)[0]
+    .trim()
+    .toLowerCase();
+}
+
+function byteLimit(maxBytes) {
+  let seen = 0;
+  return new Transform({
+    transform(chunk, encoding, callback) {
+      seen += chunk.length;
+      if (seen > maxBytes) callback(error('BODY_TOO_LARGE', 'The response body is too large.'));
+      else callback(null, chunk);
+    },
+  });
+}
+
+function decoderFor(headers) {
+  const encoding = String(firstHeader(headers, 'content-encoding') || 'identity')
+    .trim()
+    .toLowerCase();
+  if (!encoding || encoding === 'identity') return null;
+  if (encoding === 'gzip' || encoding === 'x-gzip') return createGunzip();
+  if (encoding === 'deflate') return createInflate();
+  if (encoding === 'br') return createBrotliDecompress();
+  throw error('UNSUPPORTED_ENCODING', 'The response uses an unsupported content encoding.');
+}
+
+async function collectBody(response, maxBytes) {
+  const chunks = [];
+  const collector = new Writable({
+    write(chunk, encoding, callback) {
+      chunks.push(Buffer.from(chunk));
+      callback();
+    },
+  });
+  const decoder = decoderFor(response.headers);
+  const streams = decoder
+    ? [response, byteLimit(maxBytes), decoder, byteLimit(maxBytes), collector]
+    : [response, byteLimit(maxBytes), collector];
+  try {
+    await pipeline(streams);
+  } catch (cause) {
+    if (cause instanceof PolicyError) throw cause;
+    throw error('INVALID_RESPONSE', 'The response body could not be decoded.');
+  }
+  return Buffer.concat(chunks);
+}
+
+/** Make one HTTP request while forcing Node to use the already-approved DNS answer. */
+export function performPinnedRequest({ url, address, family, timeoutMs, maxBytes, headers }) {
+  return new Promise((resolve, reject) => {
+    const transport = url.protocol === 'https:' ? https : http;
+    const dnsHostname = hostnameForDns(url);
+    const request = transport.request(
+      {
+        protocol: url.protocol,
+        hostname: dnsHostname,
+        port: url.port || undefined,
+        method: 'GET',
+        path: `${url.pathname}${url.search}`,
+        agent: false,
+        family,
+        servername: net.isIP(dnsHostname) ? undefined : dnsHostname,
+        headers: {
+          'User-Agent': USER_AGENT,
+          Accept: 'text/html, application/xhtml+xml;q=0.9',
+          'Accept-Encoding': 'gzip, deflate, br',
+          ...headers,
+          Host: url.host,
+        },
+        lookup(_hostname, lookupOptions, callback) {
+          if (lookupOptions?.all) callback(null, [{ address, family }]);
+          else callback(null, address, family);
+        },
+      },
+      async (response) => {
+        try {
+          const statusCode = response.statusCode || 0;
+          if (REDIRECT_STATUSES.has(statusCode)) {
+            response.resume();
+            resolve({ statusCode, headers: response.headers, body: Buffer.alloc(0) });
+            return;
+          }
+          const declaredSize = Number(firstHeader(response.headers, 'content-length'));
+          if (Number.isFinite(declaredSize) && declaredSize > maxBytes) {
+            response.destroy();
+            reject(error('BODY_TOO_LARGE', 'The response body is too large.'));
+            return;
+          }
+          const body = await collectBody(response, maxBytes);
+          resolve({ statusCode, headers: response.headers, body });
+        } catch (cause) {
+          reject(cause);
+        }
+      },
+    );
+
+    request.setTimeout(timeoutMs, () =>
+      request.destroy(error('TIMEOUT', 'The request timed out.')),
+    );
+    request.once('error', (cause) => {
+      if (cause instanceof PolicyError) reject(cause);
+      else reject(error('NETWORK_ERROR', 'The request failed.'));
+    });
+    request.end();
+  });
+}
+
+/**
+ * Fetch exactly one resource hop through an already-validated and pinned DNS
+ * answer. Redirects are returned to the caller so an interception layer can
+ * revalidate the browser's next request before any network I/O.
+ */
+export async function fetchResourceOnce(
+  input,
+  {
+    resolver = dns.lookup,
+    requestImpl = performPinnedRequest,
+    maxBytes = MAX_BODY_BYTES,
+    timeoutMs = DEFAULT_TIMEOUT_MS,
+    allowNonDefaultPorts = false,
+    headers = { Accept: '*/*' },
+  } = {},
+) {
+  if (!Number.isSafeInteger(maxBytes) || maxBytes < 1)
+    throw error('INVALID_OPTIONS', 'The body limit is invalid.');
+  if (!Number.isFinite(timeoutMs) || timeoutMs <= 0)
+    throw error('INVALID_OPTIONS', 'The timeout is invalid.');
+  const approved = await resolveAndValidateUrl(input, { resolver, allowNonDefaultPorts });
+  const response = await requestImpl({
+    url: approved.url,
+    address: approved.address,
+    family: approved.family,
+    timeoutMs,
+    maxBytes,
+    headers,
+  });
+  if (
+    !response ||
+    !Number.isInteger(Number(response.statusCode)) ||
+    !Buffer.isBuffer(response.body)
+  ) {
+    throw error('INVALID_RESPONSE', 'The response body is invalid.');
+  }
+  return {
+    statusCode: Number(response.statusCode),
+    headers: response.headers || {},
+    body: response.body,
+    url: approved.url.href,
+  };
+}
+
+/** Securely fetch an HTML document, validating and pinning every redirect hop. */
+export async function fetchHtml(
+  input,
+  {
+    resolver = dns.lookup,
+    requestImpl = performPinnedRequest,
+    maxRedirects = DEFAULT_MAX_REDIRECTS,
+    maxBytes = MAX_BODY_BYTES,
+    timeoutMs = DEFAULT_TIMEOUT_MS,
+    allowNonDefaultPorts = false,
+    headers,
+  } = {},
+) {
+  if (!Number.isInteger(maxRedirects) || maxRedirects < 0)
+    throw error('INVALID_OPTIONS', 'The redirect limit is invalid.');
+  if (!Number.isSafeInteger(maxBytes) || maxBytes < 1)
+    throw error('INVALID_OPTIONS', 'The body limit is invalid.');
+  if (!Number.isFinite(timeoutMs) || timeoutMs <= 0)
+    throw error('INVALID_OPTIONS', 'The timeout is invalid.');
+
+  let current = normalizeUrl(input, { allowNonDefaultPorts });
+  for (let redirects = 0; ; redirects += 1) {
+    const approved = await resolveAndValidateUrl(current, { resolver, allowNonDefaultPorts });
+    const response = await requestImpl({
+      url: approved.url,
+      address: approved.address,
+      family: approved.family,
+      timeoutMs,
+      maxBytes,
+      headers,
+    });
+    const statusCode = Number(response?.statusCode || 0);
+
+    if (REDIRECT_STATUSES.has(statusCode)) {
+      const location = firstHeader(response.headers, 'location');
+      if (!location) throw error('INVALID_REDIRECT', 'The redirect has no destination.');
+      if (redirects >= maxRedirects)
+        throw error('TOO_MANY_REDIRECTS', 'The response redirected too many times.');
+      let target;
+      try {
+        target = new URL(location, approved.url);
+      } catch {
+        throw error('INVALID_REDIRECT', 'The redirect destination is malformed.');
+      }
+      current = normalizeUrl(target.href, { allowNonDefaultPorts });
+      continue;
+    }
+
+    if (statusCode < 200 || statusCode >= 300) {
+      throw error('HTTP_STATUS', 'The server returned an unsuccessful status.');
+    }
+    if (!HTML_CONTENT_TYPES.has(contentType(response.headers))) {
+      throw error('UNSUPPORTED_CONTENT_TYPE', 'The response is not HTML.');
+    }
+    if (!Buffer.isBuffer(response.body))
+      throw error('INVALID_RESPONSE', 'The response body is invalid.');
+    if (response.body.length > maxBytes)
+      throw error('BODY_TOO_LARGE', 'The response body is too large.');
+
+    return { html: response.body.toString('utf8'), finalUrl: approved.url.href };
+  }
+}