webforai 2.1.1 → 4.0.0

package/README.md

@@ -10,8 +10,8 @@
 </p>
 
 <p align="center">
-  A esm-native library that converts HTML to Markdown & Useful Utilities with simple, lightweight and epic quality.
-<p>
+  Convert web pages and local HTML to clean, LLM-ready Markdown — with a TypeScript library, the <code>npx webforai</code> CLI, or a hosted API.
+</p>
 
 <p align="center">
   <a href="https://www.npmjs.com/package/webforai">
@@ -30,12 +30,30 @@
 
 ## Documentation
 
-[Head to the documentation](https://webforai.dev/) to read and learn more about Webforai.
+Everything is documented at **[webforai.dev](https://webforai.dev)**. Start from what you want to do:
+
+| I want to… | Use | Start here |
+| --- | --- | --- |
+| convert HTML in my own code (Node.js, browsers, Workers) | the `webforai` library | [Getting started](https://webforai.dev/getting-started) |
+| get Markdown for a URL from the shell or an AI agent | the CLI, `npx webforai <url>` | [CLI](https://webforai.dev/cli) |
+| call a hosted API that runs the browsers and proxies for me | webforai platform | [platform.webforai.dev](https://platform.webforai.dev) (sign-up, API keys, usage) · [platform docs](https://webforai.dev/platform) |
+| run my own copy of that platform | [`apps/platform`](https://github.com/inaridiy/webforai/tree/main/apps/platform) | [Deploying your own instance](https://github.com/inaridiy/webforai/tree/main/apps/platform#deploying-your-own-instance) |
+
+The library, the CLI and the typed platform client (`webforai/platform`) all ship in the one
+`webforai` npm package.
 
 ## Overview
 
 ```bash
-npx webforai@latest
+# URL or local HTML file → Markdown on stdout (add -o out.md to write a file)
+npx webforai@latest https://example.com/article
+
+# machine-readable output, hosted-platform fetching, agent installation
+npx webforai@latest https://example.com --json
+npx webforai@latest https://example.com --engine auto      # platform, renders JS when needed
+npx webforai@latest crawl https://docs.example.com -o docs --llms-txt   # site → .md files + llms.txt
+npx webforai@latest batch --file urls.txt -o pages         # many URLs → one .md per page
+npx webforai@latest skill --install                        # Agent Skill via `npx skills add`
 ```
 
 or
@@ -52,6 +70,70 @@ const html = await loadHtml(url);
 const markdown = htmlToMarkdown(html, { baseUrl: url });
 ```
 
+The CLI keeps Markdown/JSON on stdout and API failures on stderr, including the error code
+and `retryAfter` when supplied; `invalid_api_key`, `payment_required` and `spend_cap_reached`
+name the platform dashboard. `crawl` and `batch` run hosted jobs (API key required) and write
+one Markdown file per page. `--engine` and `--region` select the platform loader; combining
+them with an explicit local loader exits with usage error `2`. The interactive wizard applies
+the same validation and honors `WEBFORAI_PLATFORM_URL`.
+
+## What v3 changes
+
+**Better extraction.** The main-content extractor scores candidate containers the way Readability
+does — paragraph density, link density, class weighting, ancestor propagation — and adds signals
+the 2010 heuristics predate: semantic markup (`<article>`, `<main>`, `role="main"`, schema.org
+`articleBody`), the rendered geometry a browser loader can supply, and CJK-aware length
+thresholds. Across a 60-page corpus of real sites, the share of output lines that were bare links
+— the clearest sign navigation leaked in — dropped from 10.6% to 5.0%.
+
+**Site adapters.** Pages that generic scoring cannot read are handled directly, matched by
+hostname or by a platform fingerprint so self-hosted instances are covered too: GitHub, Stack
+Overflow, npm, MDN, Zenn, Qiita, Medium, Substack, note, Hatena, WordPress, Wikipedia, Reddit,
+YouTube, Hacker News, Docusaurus, VitePress, MkDocs, Read the Docs and GitBook. An adapter that
+produces too little is ignored in favour of generic extraction, so a site redesign degrades
+rather than breaks.
+
+```ts
+// Pass adapters: false for generic extraction only.
+import { createAutoExtractor } from "webforai";
+
+const markdown = htmlToMarkdown(html, {
+  url,
+  extractors: [createAutoExtractor({ adapters: false })],
+});
+```
+
+**Page metadata.**
+
+```ts
+import { htmlToMarkdownWithMetadata } from "webforai";
+
+const { markdown, metadata } = htmlToMarkdownWithMetadata(html, { url });
+// metadata: { title, author, published, siteName, canonicalUrl, lang, ... }
+
+// Or prepend it as YAML front matter:
+const withFrontmatter = htmlToMarkdown(html, { url, frontmatter: true });
+```
+
+**Markup fixes.** Lazily-loaded images (`data-src`, `srcset`) now resolve to a real URL instead
+of `![alt]()`; KaTeX and MathJax expressions are no longer emitted twice; `role="heading"`
+elements become real headings; superscripts and subscripts stay distinguishable; and definition
+lists keep their term/definition pairing.
+
+See the [changeset](/.changeset) for the full list, including the breaking changes.
+
+## Platform (hosted API)
+
+[`apps/platform`](https://github.com/inaridiy/webforai/tree/main/apps/platform) is an OSS,
+self-hostable SaaS built on this library: a metered crawl→Markdown HTTP API on Cloudflare
+(Workers + Workflows + Containers + Browser Run) with Better Auth accounts/API keys and
+Stripe usage-based billing. Use it through the typed `webforai/platform` client
+(`import { createPlatformClient } from "webforai/platform"`) or the CLI
+(`npx webforai <url> --engine auto`). Sign up and manage API keys at
+[platform.webforai.dev](https://platform.webforai.dev); the docs live at
+[webforai.dev/platform](https://webforai.dev/platform). See its README and the design ledger
+in `docs/specs/platform`.
+
 ## Support
 
 - [GitHub Sponsors](https://github.com/sponsors/inaridiy)

package/dist/adapters/index.js

@@ -0,0 +1,58 @@
+// src/adapters/index.ts
+import { stripNonContent } from "../extractors/lib/sanitize.js";
+import { MetricsCollector, cloneHast } from "../utils/hast-fast.js";
+import { buildPageSignature, resolveAdapterByFingerprint, resolveAdapterByHost } from "./registry.js";
+import { DECLARATIVE_ADAPTERS } from "./sites/declarative.js";
+import { hackernewsAdapter } from "./sites/hackernews.js";
+import { youtubeAdapter } from "./sites/youtube.js";
+import {
+  buildPageSignature as buildPageSignature2,
+  defineSelectorAdapter,
+  removeAll,
+  resolveAdapter,
+  selectFirst
+} from "./registry.js";
+import { DECLARATIVE_ADAPTERS as DECLARATIVE_ADAPTERS2 } from "./sites/declarative.js";
+import { hackernewsAdapter as hackernewsAdapter2 } from "./sites/hackernews.js";
+import { youtubeAdapter as youtubeAdapter2, youtubeCaptionTracks } from "./sites/youtube.js";
+var BUILTIN_ADAPTERS = [youtubeAdapter, hackernewsAdapter, ...DECLARATIVE_ADAPTERS];
+var adapterById = (id) => BUILTIN_ADAPTERS.find((adapter) => adapter.id === id);
+var MIN_ADAPTER_TEXT = 40;
+var runAdapters = (context, adapters = BUILTIN_ADAPTERS) => {
+  let matched = resolveAdapterByHost(context, adapters);
+  let withSignature = context;
+  if (!matched) {
+    withSignature = { ...context, signature: context.signature ?? buildPageSignature(context.hast) };
+    matched = resolveAdapterByFingerprint(withSignature, adapters);
+  }
+  const adapter = matched;
+  if (!adapter) {
+    return void 0;
+  }
+  const target = withSignature.owned ? withSignature : { ...withSignature, hast: cloneHast(withSignature.hast), owned: true };
+  let extracted;
+  try {
+    extracted = adapter.extract(target);
+  } catch {
+    return void 0;
+  }
+  if (!extracted) {
+    return void 0;
+  }
+  stripNonContent(extracted);
+  return new MetricsCollector().textLength(extracted) >= MIN_ADAPTER_TEXT ? extracted : void 0;
+};
+export {
+  BUILTIN_ADAPTERS,
+  DECLARATIVE_ADAPTERS2 as DECLARATIVE_ADAPTERS,
+  adapterById,
+  buildPageSignature2 as buildPageSignature,
+  defineSelectorAdapter,
+  hackernewsAdapter2 as hackernewsAdapter,
+  removeAll,
+  resolveAdapter,
+  runAdapters,
+  selectFirst,
+  youtubeAdapter2 as youtubeAdapter,
+  youtubeCaptionTracks
+};

package/dist/adapters/registry.js

@@ -0,0 +1,149 @@
+// src/adapters/registry.ts
+import { select, selectAll } from "hast-util-select";
+import { MetricsCollector, classList, isElement, pruneInPlace, stringProperty, walk } from "../utils/hast-fast.js";
+var EMPTY_SIGNATURE = {
+  classes: /* @__PURE__ */ new Set(),
+  ids: /* @__PURE__ */ new Set(),
+  tags: /* @__PURE__ */ new Set(),
+  generator: "",
+  meta: ""
+};
+var buildPageSignature = (root) => {
+  const classes = /* @__PURE__ */ new Set();
+  const ids = /* @__PURE__ */ new Set();
+  const tags = /* @__PURE__ */ new Set();
+  const generatorParts = [];
+  const metaParts = [];
+  walk(root, (node) => {
+    if (!isElement(node)) {
+      return;
+    }
+    tags.add(node.tagName);
+    for (const name of classList(node)) {
+      classes.add(name);
+    }
+    const id = stringProperty(node, "id");
+    if (id) {
+      ids.add(id);
+    }
+    if (node.tagName === "meta") {
+      const key = stringProperty(node, "name") ?? stringProperty(node, "property") ?? "";
+      const content = stringProperty(node, "content") ?? "";
+      metaParts.push(`${key}=${content}`);
+      if (key.toLowerCase() === "generator") {
+        generatorParts.push(content);
+      }
+    }
+  });
+  return {
+    classes,
+    ids,
+    tags,
+    generator: generatorParts.join(" ").toLowerCase(),
+    meta: metaParts.join("\n").toLowerCase()
+  };
+};
+var DEFAULT_MIN_LENGTH = 40;
+var hostOf = (url) => {
+  if (!url) {
+    return void 0;
+  }
+  try {
+    return new URL(url).hostname;
+  } catch {
+    return void 0;
+  }
+};
+var defineSelectorAdapter = (spec) => ({
+  id: spec.id,
+  matches: (context) => {
+    const host = hostOf(context.url);
+    return Boolean(host && spec.hosts?.test(host));
+  },
+  matchesFingerprint: (context) => {
+    const signature = context.signature ?? EMPTY_SIGNATURE;
+    return spec.fingerprint?.(signature) ?? false;
+  },
+  extract: (context) => {
+    for (const selector of spec.content) {
+      const content = tryContentSelector(selector, spec, context);
+      if (content) {
+        return content;
+      }
+    }
+    return void 0;
+  }
+});
+var tryContentSelector = (selector, spec, context) => {
+  const found = selectAll(selector, context.hast);
+  if (found.length === 0) {
+    return void 0;
+  }
+  for (const element of found) {
+    removeAll(element, spec.remove);
+  }
+  const total = found.reduce((sum, element) => sum + new MetricsCollector().textLength(element), 0);
+  if (total < (spec.minLength ?? DEFAULT_MIN_LENGTH)) {
+    return void 0;
+  }
+  const decorated = spec.decorate?.(found, context);
+  if (decorated) {
+    return decorated;
+  }
+  return found.length === 1 ? found[0] : { type: "root", children: found };
+};
+var removeAll = (root, selectors) => {
+  if (!selectors || selectors.length === 0) {
+    return;
+  }
+  const doomed = /* @__PURE__ */ new Set();
+  for (const selector of selectors) {
+    for (const element of selectAll(selector, root)) {
+      doomed.add(element);
+    }
+  }
+  if (doomed.size > 0) {
+    pruneInPlace(root, (node) => !(isElement(node) && doomed.has(node)));
+  }
+};
+var selectFirst = (root, selectors) => {
+  for (const selector of selectors) {
+    const found = select(selector, root);
+    if (found) {
+      return found;
+    }
+  }
+  return void 0;
+};
+var resolveAdapterByHost = (context, adapters) => {
+  for (const adapter of adapters) {
+    try {
+      if (adapter.matches(context)) {
+        return adapter;
+      }
+    } catch {
+    }
+  }
+  return void 0;
+};
+var resolveAdapterByFingerprint = (context, adapters) => {
+  for (const adapter of adapters) {
+    try {
+      if (adapter.matchesFingerprint?.(context)) {
+        return adapter;
+      }
+    } catch {
+    }
+  }
+  return void 0;
+};
+var resolveAdapter = (context, adapters) => resolveAdapterByHost(context, adapters) ?? resolveAdapterByFingerprint(context, adapters);
+export {
+  buildPageSignature,
+  defineSelectorAdapter,
+  removeAll,
+  resolveAdapter,
+  resolveAdapterByFingerprint,
+  resolveAdapterByHost,
+  selectFirst
+};

package/dist/adapters/sites/declarative.js

@@ -0,0 +1,178 @@
+// src/adapters/sites/declarative.ts
+import { defineSelectorAdapter } from "../registry.js";
+var COMMON_NOISE = [
+  ".sr-only",
+  "[aria-hidden='true'][class*='icon']",
+  "button",
+  ".social-share",
+  ".share-buttons",
+  ".newsletter-signup"
+];
+var SPECS = [
+  {
+    id: "github",
+    hosts: /(^|\.)github\.com$/,
+    // Issues and PRs expose a testid; repository landing pages render the README instead.
+    content: ["[data-testid='issue-body']", "article.markdown-body", ".markdown-body"],
+    remove: [".js-comment-edit-history", ".Details-content--hidden", "clipboard-copy", ...COMMON_NOISE],
+    minLength: 30
+  },
+  {
+    id: "stackoverflow",
+    hosts: /(^|\.)(stackoverflow\.com|stackexchange\.com|superuser\.com|serverfault\.com|askubuntu\.com)$/,
+    content: ["#mainbar", ".question, .answer"],
+    remove: [
+      ".js-post-menu",
+      ".post-menu",
+      ".user-info",
+      ".votecell",
+      ".comments-link",
+      ".js-add-link",
+      ...COMMON_NOISE
+    ],
+    minLength: 100
+  },
+  {
+    id: "npm",
+    hosts: /(^|\.)npmjs\.com$/,
+    content: ["#readme", "#package-tab-readme", "[class*='markdown']"],
+    remove: COMMON_NOISE,
+    minLength: 50
+  },
+  {
+    id: "zenn",
+    hosts: /(^|\.)zenn\.dev$/,
+    content: [".znc"],
+    remove: COMMON_NOISE,
+    minLength: 50
+  },
+  {
+    id: "qiita",
+    hosts: /(^|\.)qiita\.com$/,
+    content: [".it-MdContent", "[class*='mdContent']", "article"],
+    remove: [".code-copy", ...COMMON_NOISE],
+    minLength: 50
+  },
+  {
+    id: "medium",
+    hosts: /(^|\.)medium\.com$/,
+    // Medium also powers custom domains; the fingerprint catches those.
+    fingerprint: (signature) => signature.meta.includes("al:android:app_name=medium"),
+    content: ["article section", "article", "[data-testid='storyContent']"],
+    remove: ["[data-testid='audioPlayButton']", "[role='separator']", ...COMMON_NOISE],
+    minLength: 200
+  },
+  {
+    id: "substack",
+    hosts: /(^|\.)substack\.com$/,
+    // Substack's custom domains are far more common than *.substack.com.
+    fingerprint: (signature) => signature.classes.has("available-content"),
+    content: [".available-content", ".body.markup", ".post-content"],
+    remove: [".subscription-widget-wrap", ".button-wrapper", ".paywall", ...COMMON_NOISE],
+    minLength: 200
+  },
+  {
+    id: "note",
+    hosts: /(^|\.)note\.com$/,
+    content: [".note-common-styles__textnote-body", "[class*='textnote-body']"],
+    remove: COMMON_NOISE,
+    minLength: 50
+  },
+  {
+    id: "hatena",
+    hosts: /(^|\.)(hatenablog\.com|hatenadiary\.(com|jp)|hateblo\.jp|hatenastaff\.com)$/,
+    fingerprint: (signature) => signature.classes.has("hatenablog-entry") || signature.classes.has("hatena-body"),
+    content: [".entry-content"],
+    remove: [".entry-footer", ".social-buttons", ".hatena-asin-detail", ...COMMON_NOISE],
+    minLength: 100
+  },
+  {
+    id: "wordpress",
+    // Deliberately fingerprint-only: WordPress powers a large share of the web on its own
+    // domains, so keying off a hostname would cover almost none of it.
+    fingerprint: (signature) => signature.generator.includes("wordpress") || signature.classes.has("wp-block-post-content") || signature.classes.has("wp-singular"),
+    content: [".wp-block-post-content", ".entry-content", ".post-content", "article .content"],
+    remove: [".sharedaddy", ".jp-relatedposts", ".wp-block-comments", "#comments", ...COMMON_NOISE],
+    minLength: 150
+  },
+  {
+    id: "wikipedia",
+    hosts: /(^|\.)(wikipedia\.org|wikimedia\.org|wikibooks\.org|wiktionary\.org|wikisource\.org)$/,
+    fingerprint: (signature) => signature.classes.has("mw-parser-output"),
+    content: [".mw-parser-output", "#mw-content-text"],
+    remove: [
+      ".mw-editsection",
+      ".navbox",
+      ".navbox-inner",
+      ".vertical-navbox",
+      ".sidebar",
+      ".side-box",
+      ".metadata",
+      ".reflist",
+      ".refbegin",
+      ".sistersitebox",
+      ".ambox",
+      ".hatnote",
+      ".shortdescription",
+      ".mw-jump-link",
+      ".printfooter",
+      ".catlinks",
+      "#toc",
+      ".toc",
+      "table.mbox-small",
+      ...COMMON_NOISE
+    ],
+    minLength: 200
+  },
+  {
+    id: "reddit",
+    hosts: /(^|\.)reddit\.com$/,
+    content: ["shreddit-post [slot='text-body']", "[data-test-id='post-content']", ".Post"],
+    remove: ["faceplate-tracker[noun='comment_action_bar']", "shreddit-comment-action-row", ...COMMON_NOISE],
+    // Reddit renders most of a thread into shadow roots that never reach the serialised HTML,
+    // so a weak match here is worse than generic extraction. Decline unless a substantial
+    // post body is actually present in the document.
+    minLength: 600
+  },
+  {
+    id: "docusaurus",
+    fingerprint: (signature) => signature.classes.has("theme-doc-markdown") || signature.classes.has("docusaurus-mt-lg"),
+    content: [".theme-doc-markdown", "article .markdown"],
+    remove: [".theme-doc-toc-mobile", ".pagination-nav", ".theme-doc-version-banner", ...COMMON_NOISE],
+    minLength: 150
+  },
+  {
+    id: "vitepress",
+    fingerprint: (signature) => signature.classes.has("vp-doc") || signature.classes.has("VPDoc"),
+    content: [".vp-doc > div", ".vp-doc", ".VPDoc .content-container"],
+    remove: [".VPDocFooter", ".vp-doc-footer", ".header-anchor", ...COMMON_NOISE],
+    minLength: 150
+  },
+  {
+    id: "mkdocs",
+    fingerprint: (signature) => signature.classes.has("md-content__inner"),
+    content: [".md-content__inner", "[data-md-component='content']"],
+    remove: [".md-source-file", ".md-feedback", ".headerlink", ...COMMON_NOISE],
+    minLength: 150
+  },
+  {
+    id: "readthedocs",
+    fingerprint: (signature) => signature.classes.has("rst-content"),
+    content: [".rst-content", "[role='main']"],
+    remove: [".headerlink", ".rst-footer-buttons", ".wy-breadcrumbs", ...COMMON_NOISE],
+    minLength: 150
+  },
+  {
+    id: "gitbook",
+    fingerprint: (signature) => signature.classes.has("gitbook-root"),
+    content: ["[data-testid='page.contentEditor']", "main article"],
+    remove: COMMON_NOISE,
+    minLength: 150
+  }
+];
+var DECLARATIVE_ADAPTERS = SPECS.map(defineSelectorAdapter);
+var declarativeAdapterById = (id) => DECLARATIVE_ADAPTERS.find((adapter) => adapter.id === id);
+export {
+  DECLARATIVE_ADAPTERS,
+  declarativeAdapterById
+};

package/dist/adapters/sites/hackernews.js

@@ -0,0 +1,94 @@
+// src/adapters/sites/hackernews.ts
+import { select, selectAll } from "hast-util-select";
+import { numericProperty, stringProperty } from "../../utils/hast-fast.js";
+var MAX_QUOTE_DEPTH = 6;
+var text = (value) => ({ type: "text", value });
+var element = (tagName, children) => ({
+  type: "element",
+  tagName,
+  properties: {},
+  children
+});
+var commentDepth = (row) => {
+  const cell = select("td.ind", row);
+  if (!cell) {
+    return 0;
+  }
+  const indent = numericProperty(cell, "indent");
+  if (indent !== void 0) {
+    return indent;
+  }
+  const spacer = select("img", cell);
+  const width = spacer ? numericProperty(spacer, "width") : void 0;
+  return width === void 0 ? 0 : Math.round(width / 40);
+};
+var nest = (content, depth) => {
+  let current = content;
+  for (let level = 0; level < Math.min(depth, MAX_QUOTE_DEPTH); level++) {
+    current = [element("blockquote", current)];
+  }
+  return current[0];
+};
+var storyNodes = (root) => {
+  const nodes = [];
+  const titleLink = select(".titleline > a", root);
+  if (titleLink) {
+    nodes.push(element("h1", [...titleLink.children]));
+    const href = stringProperty(titleLink, "href");
+    if (href && /^https?:/.test(href)) {
+      nodes.push(element("p", [{ ...titleLink, children: [text(href)] }]));
+    }
+  }
+  const topText = select(".toptext", root);
+  if (topText) {
+    nodes.push(element("div", [...topText.children]));
+  }
+  return nodes;
+};
+var commentNodes = (root) => {
+  const nodes = [];
+  for (const row of selectAll("tr.comtr, tr.athing.comtr", root)) {
+    const body = select(".commtext", row);
+    if (!body) {
+      continue;
+    }
+    const author = select(".hnuser", row);
+    const header = author ? [element("strong", [...author.children])] : [];
+    const content = [
+      ...header.length > 0 ? [element("p", header)] : [],
+      element("div", [...body.children])
+    ];
+    nodes.push(nest(content, commentDepth(row)));
+  }
+  return nodes;
+};
+var hackernewsAdapter = {
+  id: "hackernews",
+  matches: ({ url }) => {
+    if (!url) {
+      return false;
+    }
+    try {
+      return /(^|\.)ycombinator\.com$/.test(new URL(url).hostname);
+    } catch {
+      return false;
+    }
+  },
+  matchesFingerprint: ({ signature }) => Boolean(signature?.ids.has("hnmain") || signature?.classes.has("fatitem")),
+  extract: ({ hast }) => {
+    const story = storyNodes(hast);
+    const comments = commentNodes(hast);
+    if (story.length === 0 && comments.length === 0) {
+      return void 0;
+    }
+    const children = [...story];
+    if (comments.length > 0) {
+      children.push(element("h2", [text("Comments")]));
+      children.push(...comments);
+    }
+    return { type: "root", children };
+  }
+};
+export {
+  hackernewsAdapter
+};